crypto: crypto API library interfaces for WireGuard
Subsystems (11)
Ard Biesheuvel
maintainerv5 cover crypto: crypto API library interfaces for WireGuard
This series implements the crypto library abstractions that are needed to incorporate WireGuard into the mainline kernel.
Changes since v4:
- Address most review feedback from Eric, with the exception of the remark about libraries being selectable by the user - this is something we need to revisit in the context of moving to weak references or static calls to make accelerated versions of libraries loadable at any time. (Currently, loading an accelerated version at runtime will not supersede calls to the generic routines in the kernel proper, which is counterintuitive, and this is currently being addressed by making the generic library versions only selectable as modules if the accelerated ones are selected as modules as well)
- Align the generic blake2s Kconfig symbols, filenames etc with the recently added blake2b driver.
- Rewrote the blake2s selftest for better coverage of key length and input length combinations, and added a HMAC selftest as well.
- Rename blake2s_hmac() to blake2s256_hmac(), and drop the digest length argument, which was not implemented correctly, and never deviates from the full length in practice anyway.
- Update to more recent version of the blake2s x86 Zinc code
Changes since v3:
- Unify the way the generic vs arch libraries are organized between ChaCha20 and Poly1305 on the one hand and Curve25519 and Blake2s on the other. All are now made up of a generic library, a generic crypto API driver (skcipher for [X]ChaCha, shash for Poly1305 and Blake2s and kpp for Curve25519) and optional per-arch versions providing both the library and the crypto API interfaces while potentially relying on the generic *library* only as a fallback (and not on the generic crypto API driver). Implementations of the libary interface that don't require the fallback don't pull in the generic code at all, but the generic crypto API drivers are tied to the generic implementations directly (this is necessary since we fuzz test the accelerated implementations against the generic implementations)
- Provide testmgr test vectors for the Curve25519 and Blake2s crypto API drivers that were added in this revision. This also required some changes to the KPP test routines so we can test for failures as well.
- Update to the latest version of Andy Polyakov's Poly1305 implementation for MIPS that incorporates Rene's improvements for 32r2
- Remove logic in the x86 and ARM implementations of ChaCha and Poly1305 to prefer the non-SIMD path for short inputs. This is no longer necessary, and even undesirable since it forced ChaCha20Poly1305's ChaCha pass generating the Poly1305 nonce to always take the slower scalar path.
Changes since v2:
- Reduce the cc: audience a bit, since I assumed that not everyone is interested in discussing the details of this.
- Incorporate scalar ARM code for ChaCha, and the 64-bit MIPS code for Poly1305. NOTE: the Cryptogams MIPS code now supports 32-bit MIPS as well, and not just 32r2, so I omitted Rene's Poly1305 implementation for now, and used Andy's code for everything.
- Incorporate NEON opt-out for Cortex-A5/A7. Note that the code is still exposed via the crypto API, but with a low prioririty, so it is still available and still gets test coverage, but is not used by default.
- Use static keys (*not* static calls) in the SIMD and bmi2/adx drivers to keep track of which implementation is being used, to avoid the memory load on each call.
- Defer using weak references or static calls until the dust around this has settled. Instead, rely on Kconfig constraints and symbol dependencies to ensure that the arch code is always used when it is loaded. This means you can only opt out of using the arch code if you disable it in Kconfig but this is something I can live with for now.
- Refactor the Curve25519 glue code slightly so that the call sites branch to the arch or generic code directly.
- Split up the Poly1305 refactoring patches so they can be reviewed more easily.
Changes since RFC/v1:
- dropped the WireGuard patch itself, and the followup patches - since the purpose was to illustrate the extent of the required changes, there is no reason to keep including them.
- import the MIPS 32r2 versions of ChaCha and Poly1305, but expose both the crypto API and library interfaces so that not only WireGuard but also IPsec and Adiantum can benefit immediately. (The latter required adding support for the reduced round version of ChaCha to the MIPS asm code)
- fix up various minor kconfig/build issues found in randconfig testing (thanks Arnd!)
Patches can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=wireguard-crypto-library-api-v5
Ard Biesheuvel (27):
crypto: tidy up lib/crypto Kconfig and Makefile
crypto: chacha - move existing library code into lib/crypto
crypto: x86/chacha - depend on generic chacha library instead of
crypto driver
crypto: x86/chacha - expose SIMD ChaCha routine as library function
crypto: arm64/chacha - depend on generic chacha library instead of
crypto driver
crypto: arm64/chacha - expose arm64 ChaCha routine as library function
crypto: arm/chacha - import Eric Biggers's scalar accelerated ChaCha
code
crypto: arm/chacha - remove dependency on generic ChaCha driver
crypto: arm/chacha - expose ARM ChaCha routine as library function
crypto: mips/chacha - wire up accelerated 32r2 code from Zinc
crypto: chacha - unexport chacha_generic routines
crypto: poly1305 - move core routines into a separate library
crypto: x86/poly1305 - unify Poly1305 state struct with generic code
crypto: poly1305 - expose init/update/final library interface
crypto: x86/poly1305 - depend on generic library not generic shash
crypto: x86/poly1305 - expose existing driver as poly1305 library
crypto: arm64/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON
implementation
crypto: arm/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON
implementation
crypto: mips/poly1305 - incorporate OpenSSL/CRYPTOGAMS optimized
implementation
int128: move __uint128_t compiler test to Kconfig
crypto: testmgr - add test cases for Blake2s
crypto: blake2s - implement generic shash driver
crypto: curve25519 - add kpp selftest
crypto: curve25519 - implement generic KPP driver
crypto: lib/curve25519 - work around Clang stack spilling issue
crypto: chacha20poly1305 - import construction and selftest from Zinc
crypto: lib/chacha20poly1305 - reimplement crypt_from_sg() routine
Jason A. Donenfeld (7):
crypto: mips/chacha - import 32r2 ChaCha code from Zinc
crypto: BLAKE2s - generic C library implementation and selftest
crypto: BLAKE2s - x86_64 SIMD implementation
crypto: Curve25519 - generic C library implementations
crypto: Curve25519 - x86_64 library and KPP implementations
crypto: arm - import Bernstein and Schwabe's Curve25519 ARM
implementation
crypto: arm/Curve25519 - wire up NEON implementation
arch/arm/crypto/Kconfig | 16 +-
arch/arm/crypto/Makefile | 17 +-
arch/arm/crypto/chacha-glue.c | 343 +
arch/arm/crypto/chacha-neon-glue.c | 202 -
arch/arm/crypto/chacha-scalar-core.S | 460 ++
arch/arm/crypto/curve25519-core.S | 2062 ++++++
arch/arm/crypto/curve25519-glue.c | 127 +
arch/arm/crypto/poly1305-armv4.pl | 1236 ++++
arch/arm/crypto/poly1305-core.S_shipped | 1158 +++
arch/arm/crypto/poly1305-glue.c | 276 +
arch/arm64/Kconfig | 2 +-
arch/arm64/crypto/Kconfig | 9 +-
arch/arm64/crypto/Makefile | 10 +-
arch/arm64/crypto/chacha-neon-glue.c | 81 +-
arch/arm64/crypto/poly1305-armv8.pl | 913 +++
arch/arm64/crypto/poly1305-core.S_shipped | 835 +++
arch/arm64/crypto/poly1305-glue.c | 237 +
arch/mips/Makefile | 2 +-
arch/mips/crypto/Makefile | 18 +
arch/mips/crypto/chacha-core.S | 497 ++
arch/mips/crypto/chacha-glue.c | 150 +
arch/mips/crypto/poly1305-glue.c | 203 +
arch/mips/crypto/poly1305-mips.pl | 1273 ++++
arch/riscv/Kconfig | 2 +-
arch/x86/Kconfig | 2 +-
arch/x86/crypto/Makefile | 3 +
arch/x86/crypto/blake2s-core.S | 258 +
arch/x86/crypto/blake2s-glue.c | 233 +
arch/x86/crypto/chacha_glue.c | 181 +-
arch/x86/crypto/curve25519-x86_64.c | 2475 +++++++
arch/x86/crypto/poly1305_glue.c | 199 +-
crypto/Kconfig | 71 +-
crypto/Makefile | 2 +
crypto/adiantum.c | 5 +-
crypto/blake2s_generic.c | 171 +
crypto/chacha_generic.c | 84 +-
crypto/curve25519-generic.c | 90 +
crypto/ecc.c | 2 +-
crypto/nhpoly1305.c | 3 +-
crypto/poly1305_generic.c | 228 +-
crypto/testmgr.c | 30 +
crypto/testmgr.h | 1520 +++-
include/crypto/blake2s.h | 106 +
include/crypto/chacha.h | 83 +-
include/crypto/chacha20poly1305.h | 48 +
include/crypto/curve25519.h | 71 +
include/crypto/internal/blake2s.h | 24 +
include/crypto/internal/chacha.h | 43 +
include/crypto/internal/poly1305.h | 58 +
include/crypto/poly1305.h | 69 +-
init/Kconfig | 4 +
lib/Makefile | 3 +-
lib/crypto/Kconfig | 130 +
lib/crypto/Makefile | 42 +-
lib/crypto/blake2s-generic.c | 111 +
lib/crypto/blake2s-selftest.c | 622 ++
lib/crypto/blake2s.c | 126 +
lib/{ => crypto}/chacha.c | 20 +-
lib/crypto/chacha20poly1305-selftest.c | 7393 ++++++++++++++++++++
lib/crypto/chacha20poly1305.c | 369 +
lib/crypto/curve25519-fiat32.c | 864 +++
lib/crypto/curve25519-hacl64.c | 788 +++
lib/crypto/curve25519.c | 25 +
lib/crypto/libchacha.c | 35 +
lib/crypto/poly1305.c | 232 +
lib/ubsan.c | 2 +-
lib/ubsan.h | 2 +-
67 files changed, 26148 insertions(+), 808 deletions(-)
create mode 100644 arch/arm/crypto/chacha-glue.c
delete mode 100644 arch/arm/crypto/chacha-neon-glue.c
create mode 100644 arch/arm/crypto/chacha-scalar-core.S
create mode 100644 arch/arm/crypto/curve25519-core.S
create mode 100644 arch/arm/crypto/curve25519-glue.c
create mode 100644 arch/arm/crypto/poly1305-armv4.pl
create mode 100644 arch/arm/crypto/poly1305-core.S_shipped
create mode 100644 arch/arm/crypto/poly1305-glue.c
create mode 100644 arch/arm64/crypto/poly1305-armv8.pl
create mode 100644 arch/arm64/crypto/poly1305-core.S_shipped
create mode 100644 arch/arm64/crypto/poly1305-glue.c
create mode 100644 arch/mips/crypto/chacha-core.S
create mode 100644 arch/mips/crypto/chacha-glue.c
create mode 100644 arch/mips/crypto/poly1305-glue.c
create mode 100644 arch/mips/crypto/poly1305-mips.pl
create mode 100644 arch/x86/crypto/blake2s-core.S
create mode 100644 arch/x86/crypto/blake2s-glue.c
create mode 100644 arch/x86/crypto/curve25519-x86_64.c
create mode 100644 crypto/blake2s_generic.c
create mode 100644 crypto/curve25519-generic.c
create mode 100644 include/crypto/blake2s.h
create mode 100644 include/crypto/chacha20poly1305.h
create mode 100644 include/crypto/curve25519.h
create mode 100644 include/crypto/internal/blake2s.h
create mode 100644 include/crypto/internal/chacha.h
create mode 100644 include/crypto/internal/poly1305.h
create mode 100644 lib/crypto/Kconfig
create mode 100644 lib/crypto/blake2s-generic.c
create mode 100644 lib/crypto/blake2s-selftest.c
create mode 100644 lib/crypto/blake2s.c
rename lib/{ => crypto}/chacha.c (88%)
create mode 100644 lib/crypto/chacha20poly1305-selftest.c
create mode 100644 lib/crypto/chacha20poly1305.c
create mode 100644 lib/crypto/curve25519-fiat32.c
create mode 100644 lib/crypto/curve25519-hacl64.c
create mode 100644 lib/crypto/curve25519.c
create mode 100644 lib/crypto/libchacha.c
create mode 100644 lib/crypto/poly1305.cSignature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 1/34 crypto: tidy up lib/crypto Kconfig and Makefile
In preparation of introducing a set of crypto library interfaces, tidy up the Makefile and split off the Kconfig symbols into a separate file.
--- crypto/Kconfig | 13 +------------ lib/crypto/Kconfig | 15 +++++++++++++++ lib/crypto/Makefile | 16 ++++++++-------- 3 files changed, 24 insertions(+), 20 deletions(-)
crypto/Kconfig +1 -12
@@ -895,9 +895,6 @@ config CRYPTO_SHA1_PPC_SPE SHA-1 secure hash standard (DFIPS 180-4) implemented using powerpc SPE SIMD instruction set. -config CRYPTO_LIB_SHA256 - tristate - config CRYPTO_SHA256 tristate "SHA224 and SHA256 digest algorithm" select CRYPTO_HASH
@@ -1036,9 +1033,6 @@ config CRYPTO_GHASH_CLMUL_NI_INTEL comment "Ciphers" -config CRYPTO_LIB_AES - tristate - config CRYPTO_AES tristate "AES cipher algorithms" select CRYPTO_ALGAPI
@@ -1167,9 +1161,6 @@ config CRYPTO_ANUBIS <https://www.cosic.esat.kuleuven.be/nessie/reports/> <http://www.larc.usp.br/~pbarreto/AnubisPage.html> -config CRYPTO_LIB_ARC4 - tristate - config CRYPTO_ARC4 tristate "ARC4 cipher algorithm" select CRYPTO_SKCIPHER
@@ -1357,9 +1348,6 @@ config CRYPTO_CAST6_AVX_X86_64 This module provides the Cast6 cipher algorithm that processes eight blocks parallel using the AVX instruction set. -config CRYPTO_LIB_DES - tristate - config CRYPTO_DES tristate "DES and Triple DES EDE cipher algorithms" select CRYPTO_ALGAPI
@@ -1864,6 +1852,7 @@ config CRYPTO_STATS config CRYPTO_HASH_INFO bool +source "lib/crypto/Kconfig" source "drivers/crypto/Kconfig" source "crypto/asymmetric_keys/Kconfig" source "certs/Kconfig"
lib/crypto/Kconfig new file +15 -0
@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0 + +comment "Crypto library routines" + +config CRYPTO_LIB_AES + tristate + +config CRYPTO_LIB_ARC4 + tristate + +config CRYPTO_LIB_DES + tristate + +config CRYPTO_LIB_SHA256 + tristate
lib/crypto/Makefile +8 -8
@@ -1,13 +1,13 @@ # SPDX-License-Identifier: GPL-2.0 -obj-$(CONFIG_CRYPTO_LIB_AES) += libaes.o -libaes-y := aes.o +obj-$(CONFIG_CRYPTO_LIB_AES) += libaes.o +libaes-y := aes.o -obj-$(CONFIG_CRYPTO_LIB_ARC4) += libarc4.o -libarc4-y := arc4.o +obj-$(CONFIG_CRYPTO_LIB_ARC4) += libarc4.o +libarc4-y := arc4.o -obj-$(CONFIG_CRYPTO_LIB_DES) += libdes.o -libdes-y := des.o +obj-$(CONFIG_CRYPTO_LIB_DES) += libdes.o +libdes-y := des.o -obj-$(CONFIG_CRYPTO_LIB_SHA256) += libsha256.o -libsha256-y := sha256.o +obj-$(CONFIG_CRYPTO_LIB_SHA256) += libsha256.o +libsha256-y := sha256.o
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 2/34 crypto: chacha - move existing library code into lib/crypto
In reply to Ard BiesheuvelCurrently, our generic ChaCha implementation consists of a permute function in lib/chacha.c that operates on the 64-byte ChaCha state directly [and which is always included into the core kernel since it is used by the /dev/random driver], and the crypto API plumbing to expose it as a skcipher.
In order to support in-kernel users that need the ChaCha streamcipher but have no need [or tolerance] for going through the abstractions of the crypto API, let's expose the streamcipher bits via a library API as well, in a way that permits the implementation to be superseded by an architecture specific one if provided.
So move the streamcipher code into a separate module in lib/crypto, and expose the init() and crypt() routines to users of the library.
---
arch/arm/crypto/chacha-neon-glue.c | 2 +-
arch/arm64/crypto/chacha-neon-glue.c | 2 +-
arch/x86/crypto/chacha_glue.c | 2 +-
crypto/Kconfig | 1 +
crypto/chacha_generic.c | 60 ++-------------
include/crypto/chacha.h | 77 +++++++++++++++-----
include/crypto/internal/chacha.h | 53 ++++++++++++++
lib/Makefile | 3 +-
lib/crypto/Kconfig | 26 +++++++
lib/crypto/Makefile | 4 +
lib/{ => crypto}/chacha.c | 20 ++---
lib/crypto/libchacha.c | 35 +++++++++
12 files changed, 199 insertions(+), 86 deletions(-)arch/arm/crypto/chacha-neon-glue.c +1 -1
@@ -20,7 +20,7 @@*/ #include <crypto/algapi.h> -#include <crypto/chacha.h> +#include <crypto/internal/chacha.h> #include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <linux/kernel.h>
arch/arm64/crypto/chacha-neon-glue.c +1 -1
@@ -20,7 +20,7 @@*/ #include <crypto/algapi.h> -#include <crypto/chacha.h> +#include <crypto/internal/chacha.h> #include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <linux/kernel.h>
arch/x86/crypto/chacha_glue.c +1 -1
@@ -7,7 +7,7 @@*/ #include <crypto/algapi.h> -#include <crypto/chacha.h> +#include <crypto/internal/chacha.h> #include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <linux/kernel.h>
crypto/Kconfig +1 -0
@@ -1412,6 +1412,7 @@ config CRYPTO_SALSA20 config CRYPTO_CHACHA20 tristate "ChaCha stream cipher algorithms" + select CRYPTO_LIB_CHACHA_GENERIC select CRYPTO_SKCIPHER help The ChaCha20, XChaCha20, and XChaCha12 stream cipher algorithms.
crypto/chacha_generic.c +5 -55
90 lines not on this page · raw message
include/crypto/chacha.h +60 -17
95 lines not on this page · raw message
include/crypto/internal/chacha.h new file +53 -0
53 lines not on this page · raw message
lib/Makefile +1 -2
@@ -26,8 +26,7 @@ endif lib-y := ctype.o string.o vsprintf.o cmdline.o \ rbtree.o radix-tree.o timerqueue.o xarray.o \ - idr.o extable.o \ - sha1.o chacha.o irq_regs.o argv_split.o \ + idr.o extable.o sha1.o irq_regs.o argv_split.o \ flex_proportions.o ratelimit.o show_mem.o \ is_single_threaded.o plist.o decompress.o kobject_uevent.o \ earlycpio.o seq_buf.o siphash.o dec_and_lock.o \
lib/crypto/Kconfig +26 -0
@@ -8,6 +8,32 @@ config CRYPTO_LIB_AES config CRYPTO_LIB_ARC4 tristate +config CRYPTO_ARCH_HAVE_LIB_CHACHA + tristate + help + Declares whether the architecture provides an arch-specific + accelerated implementation of the ChaCha library interface, + either builtin or as a module. + +config CRYPTO_LIB_CHACHA_GENERIC + tristate + select CRYPTO_ALGAPI + help + This symbol can be depended upon by arch implementations of the + ChaCha library interface that require the generic code as a + fallback, e.g., for SIMD implementations. If no arch specific + implementation is enabled, this implementation serves the users + of CRYPTO_LIB_CHACHA. + +config CRYPTO_LIB_CHACHA + tristate "ChaCha library interface" + depends on CRYPTO_ARCH_HAVE_LIB_CHACHA || !CRYPTO_ARCH_HAVE_LIB_CHACHA + select CRYPTO_LIB_CHACHA_GENERIC if CRYPTO_ARCH_HAVE_LIB_CHACHA=n + help + Enable the ChaCha library interface. This interface may be fulfilled + by either the generic implementation or an arch-specific one, if one + is available and enabled. + config CRYPTO_LIB_DES tristate
lib/crypto/Makefile +4 -0
@@ -1,5 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 +# chacha is used by the /dev/random driver which is always builtin +obj-y += chacha.o +obj-$(CONFIG_CRYPTO_LIB_CHACHA_GENERIC) += libchacha.o + obj-$(CONFIG_CRYPTO_LIB_AES) += libaes.o libaes-y := aes.o
lib/crypto/chacha.c renamed from lib/chacha.c +11 -9
@@ -5,9 +5,11 @@* Copyright (C) 2015 Martin Willi */ +#include <linux/bug.h> #include <linux/kernel.h> #include <linux/export.h> #include <linux/bitops.h> +#include <linux/string.h> #include <linux/cryptohash.h> #include <asm/unaligned.h> #include <crypto/chacha.h>
@@ -72,7 +74,7 @@ static void chacha_permute(u32 *x, int nrounds)* The caller has already converted the endianness of the input. This function * also handles incrementing the block counter in the input matrix. */ -void chacha_block (u32 *state,u8 *stream,int nrounds) +void chacha_block_generic (u32 *state,u8 *stream,int nrounds) {u32 x[16 ];int i;
@@ -86,11 +88,11 @@ void chacha_block(u32 *state, u8 *stream, int nrounds) state[12 ]++; } -EXPORT_SYMBOL (chacha_block); +EXPORT_SYMBOL (chacha_block_generic);/** -* hchacha_block - abbreviated ChaCha core, for XChaCha -* @in: input state matrix (16 32-bit words) +* hchacha_block_generic - abbreviated ChaCha core, for XChaCha +* @state: input state matrix (16 32-bit words) * @out: output (8 32-bit words) * @nrounds: number of rounds (20 or 12; 20 is recommended) *
@@ -99,15 +101,15 @@ EXPORT_SYMBOL(chacha_block);* skips the final addition of the initial state, and outputs only certain words * of the state. It should not be used for streaming directly. */ -void hchacha_block (const u32 *in,u32 *out,int nrounds) +void hchacha_block_generic (const u32 *state,u32 *stream,int nrounds) {u32 x[16 ]; -memcpy (x, in,64 ); +memcpy (x, state,64 );chacha_permute (x, nrounds); -memcpy (&out[0 ], &x[0 ],16 ); -memcpy (&out[4 ], &x[12 ],16 ); +memcpy (&stream[0 ], &x[0 ],16 ); +memcpy (&stream[4 ], &x[12 ],16 ); } -EXPORT_SYMBOL (hchacha_block); +EXPORT_SYMBOL (hchacha_block_generic);
lib/crypto/libchacha.c new file +35 -0
@@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* The ChaCha stream cipher (RFC7539) +* +* Copyright (C) 2015 Martin Willi +*/ + +#include <linux/kernel.h> +#include <linux/export.h> +#include <linux/module.h> + +#include <crypto/algapi.h> // for crypto_xor_cpy +#include <crypto/chacha.h> + +void chacha_crypt_generic (u32 *state,u8 *dst,const u8 *src, +unsigned int bytes,int nrounds) +{ +/* aligned to potentially speed up crypto_xor() */ +u8 stream[CHACHA_BLOCK_SIZE ]__aligned(sizeof(long)) ; + +while (bytes >=CHACHA_BLOCK_SIZE ) { +chacha_block_generic (state, stream, nrounds); +crypto_xor_cpy (dst, src, stream,CHACHA_BLOCK_SIZE ); + bytes -=CHACHA_BLOCK_SIZE ; + dst +=CHACHA_BLOCK_SIZE ; + src +=CHACHA_BLOCK_SIZE ; + } +if (bytes) { +chacha_block_generic (state, stream, nrounds); +crypto_xor_cpy (dst, src, stream, bytes); + } +} +EXPORT_SYMBOL (chacha_crypt_generic); + +MODULE_LICENSE ("GPL" );
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 3/34 crypto: x86/chacha - depend on generic chacha library instead of crypto driver
In reply to Ard BiesheuvelIn preparation of extending the x86 ChaCha driver to also expose the ChaCha library interface, drop the dependency on the chacha_generic crypto driver as a non-SIMD fallback, and depend on the generic ChaCha library directly. This way, we only pull in the code we actually need, without registering a set of ChaCha skciphers that we will never use.
Since turning the FPU on and off is cheap these days, simplify the SIMD routine by dropping the per-page yield, which makes for a cleaner switch to the library API as well. This also allows use to invoke the skcipher walk routines in non-atomic mode.
--- arch/x86/crypto/chacha_glue.c | 90 ++++++++------------ crypto/Kconfig | 2 +- 2 files changed, 36 insertions(+), 56 deletions(-)
arch/x86/crypto/chacha_glue.c +35 -55
@@ -123,37 +123,38 @@ static void chacha_dosimd(u32 *state, u8 *dst, const u8 *src, } } -static int chacha_simd_stream_xor (struct skcipher_walk *walk, +static int chacha_simd_stream_xor (struct skcipher_request *req,const struct chacha_ctx *ctx,const u8 *iv) {u32 *state, state_buf[16 +2 ]__aligned(8) ; -int next_yield =4096 ;/* bytes until next FPU yield */ -int err =0 ; +struct skcipher_walk walk; +int err; + + err =skcipher_walk_virt (&walk, req, false);BUILD_BUG_ON (CHACHA_STATE_ALIGN !=16 ); state =PTR_ALIGN (state_buf +0 ,CHACHA_STATE_ALIGN ); -crypto_chacha_init (state, ctx, iv); +chacha_init_generic (state, ctx->key, iv); -while (walk->nbytes >0 ) { -unsigned int nbytes = walk->nbytes; +while (walk.nbytes >0 ) { +unsigned int nbytes = walk.nbytes; -if (nbytes < walk->total) { - nbytes =round_down (nbytes, walk->stride); - next_yield -= nbytes; - } +if (nbytes < walk.total) + nbytes =round_down (nbytes, walk.stride); -chacha_dosimd (state, walk->dst.virt.addr, walk->src.virt.addr, - nbytes, ctx->nrounds); - -if (next_yield <=0 ) { -/* temporarily allow preemption */ -kernel_fpu_end (); +if (!crypto_simd_usable ()) { +chacha_crypt_generic (state, walk.dst.virt.addr, + walk.src.virt.addr, nbytes, + ctx->nrounds); + }else {kernel_fpu_begin (); - next_yield =4096 ; +chacha_dosimd (state, walk.dst.virt.addr, + walk.src.virt.addr, nbytes, + ctx->nrounds); +kernel_fpu_end (); } - - err =skcipher_walk_done (walk, walk->nbytes - nbytes); + err =skcipher_walk_done (&walk, walk.nbytes - nbytes); }return err;
@@ -163,55 +164,34 @@ static int chacha_simd(struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm); -struct skcipher_walk walk; -int err; -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ()) -return crypto_chacha_crypt (req); - - err =skcipher_walk_virt (&walk, req, true); -if (err) -return err; - -kernel_fpu_begin (); - err =chacha_simd_stream_xor (&walk, ctx, req->iv); -kernel_fpu_end (); -return err; +return chacha_simd_stream_xor (req, ctx, req->iv); }static int xchacha_simd (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm); -struct skcipher_walk walk; -struct chacha_ctx subctx;u32 *state, state_buf[16 +2 ]__aligned(8) ; +struct chacha_ctx subctx;u8 real_iv[16 ]; -int err; - -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ()) -return crypto_xchacha_crypt (req); - - err =skcipher_walk_virt (&walk, req, true); -if (err) -return err;BUILD_BUG_ON (CHACHA_STATE_ALIGN !=16 ); state =PTR_ALIGN (state_buf +0 ,CHACHA_STATE_ALIGN ); -crypto_chacha_init (state, ctx, req->iv); - -kernel_fpu_begin (); - -hchacha_block_ssse3 (state, subctx.key, ctx->nrounds); +chacha_init_generic (state, ctx->key, req->iv); + +if (req->cryptlen >CHACHA_BLOCK_SIZE &&crypto_simd_usable ()) { +kernel_fpu_begin (); +hchacha_block_ssse3 (state, subctx.key, ctx->nrounds); +kernel_fpu_end (); + }else { +hchacha_block_generic (state, subctx.key, ctx->nrounds); + } subctx.nrounds = ctx->nrounds;memcpy (&real_iv[0 ], req->iv +24 ,8 );memcpy (&real_iv[8 ], req->iv +16 ,8 ); - err =chacha_simd_stream_xor (&walk, &subctx, real_iv); - -kernel_fpu_end (); - -return err; +return chacha_simd_stream_xor (req, &subctx, real_iv); }static struct skcipher_alg algs[] = {
@@ -227,7 +207,7 @@ static struct skcipher_alg algs[] = { .max_keysize = CHACHA_KEY_SIZE, .ivsize = CHACHA_IV_SIZE, .chunksize = CHACHA_BLOCK_SIZE, - .setkey = crypto_chacha20_setkey, + .setkey = chacha20_setkey, .encrypt = chacha_simd, .decrypt = chacha_simd, }, {
@@ -242,7 +222,7 @@ static struct skcipher_alg algs[] = { .max_keysize = CHACHA_KEY_SIZE, .ivsize = XCHACHA_IV_SIZE, .chunksize = CHACHA_BLOCK_SIZE, - .setkey = crypto_chacha20_setkey, + .setkey = chacha20_setkey, .encrypt = xchacha_simd, .decrypt = xchacha_simd, }, {
@@ -257,7 +237,7 @@ static struct skcipher_alg algs[] = { .max_keysize = CHACHA_KEY_SIZE, .ivsize = XCHACHA_IV_SIZE, .chunksize = CHACHA_BLOCK_SIZE, - .setkey = crypto_chacha12_setkey, + .setkey = chacha12_setkey, .encrypt = xchacha_simd, .decrypt = xchacha_simd, },
crypto/Kconfig +1 -1
@@ -1436,7 +1436,7 @@ config CRYPTO_CHACHA20_X86_64 tristate "ChaCha stream cipher algorithms (x86_64/SSSE3/AVX2/AVX-512VL)" depends on X86 && 64BIT select CRYPTO_SKCIPHER - select CRYPTO_CHACHA20 + select CRYPTO_LIB_CHACHA_GENERIC help SSSE3, AVX2, and AVX-512VL optimized implementations of the ChaCha20, XChaCha20, and XChaCha12 stream ciphers.
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 4/34 crypto: x86/chacha - expose SIMD ChaCha routine as library function
In reply to Ard BiesheuvelWire the existing x86 SIMD ChaCha code into the new ChaCha library interface, so that users of the library interface will get the accelerated version when available.
Given that calls into the library API will always go through the routines in this module if it is enabled, switch to static keys to select the optimal implementation available (which may be none at all, in which case we defer to the generic implementation for all invocations).
--- arch/x86/crypto/chacha_glue.c | 91 ++++++++++++++------ crypto/Kconfig | 1 + include/crypto/chacha.h | 6 ++ 3 files changed, 73 insertions(+), 25 deletions(-)
arch/x86/crypto/chacha_glue.c +66 -25
@@ -21,24 +21,24 @@ asmlinkage void chacha_block_xor_ssse3(u32 *state, u8 *dst, const u8 *src, asmlinkage void chacha_4block_xor_ssse3(u32 *state, u8 *dst, const u8 *src,unsigned int len,int nrounds); asmlinkage void hchacha_block_ssse3(const u32 *state, u32 *out, int nrounds); -#ifdef CONFIG_AS_AVX2 + asmlinkage void chacha_2block_xor_avx2(u32 *state, u8 *dst, const u8 *src,unsigned int len,int nrounds); asmlinkage void chacha_4block_xor_avx2(u32 *state, u8 *dst, const u8 *src,unsigned int len,int nrounds); asmlinkage void chacha_8block_xor_avx2(u32 *state, u8 *dst, const u8 *src,unsigned int len,int nrounds); -static bool chacha_use_avx2; -#ifdef CONFIG_AS_AVX512 + asmlinkage void chacha_2block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src,unsigned int len,int nrounds); asmlinkage void chacha_4block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src,unsigned int len,int nrounds); asmlinkage void chacha_8block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src,unsigned int len,int nrounds); -static bool chacha_use_avx512vl; -#endif -#endif + +static __ro_after_init DEFINE_STATIC_KEY_FALSE (chacha_use_simd); +static __ro_after_init DEFINE_STATIC_KEY_FALSE (chacha_use_avx2); +static __ro_after_init DEFINE_STATIC_KEY_FALSE (chacha_use_avx512vl);static unsigned int chacha_advance (unsigned int len,unsigned int maxblocks) {
@@ -49,9 +49,8 @@ static unsigned int chacha_advance(unsigned int len, unsigned int maxblocks)static void chacha_dosimd (u32 *state,u8 *dst,const u8 *src,unsigned int bytes,int nrounds) { -#ifdef CONFIG_AS_AVX2 -#ifdef CONFIG_AS_AVX512 - if (chacha_use_avx512vl) { +if (IS_ENABLED (CONFIG_AS_AVX512 ) && +static_branch_likely (&chacha_use_avx512vl)) {while (bytes >=CHACHA_BLOCK_SIZE *8 ) {chacha_8block_xor_avx512vl (state, dst, src, bytes, nrounds);
@@ -79,8 +78,9 @@ static void chacha_dosimd(u32 *state, u8 *dst, const u8 *src,return ; } } -#endif -if (chacha_use_avx2) { + +if (IS_ENABLED (CONFIG_AS_AVX2 ) && +static_branch_likely (&chacha_use_avx2)) {while (bytes >=CHACHA_BLOCK_SIZE *8 ) {chacha_8block_xor_avx2 (state, dst, src, bytes, nrounds); bytes -=CHACHA_BLOCK_SIZE *8 ;
@@ -104,7 +104,7 @@ static void chacha_dosimd(u32 *state, u8 *dst, const u8 *src,return ; } } -#endif +while (bytes >=CHACHA_BLOCK_SIZE *4 ) {chacha_4block_xor_ssse3 (state, dst, src, bytes, nrounds); bytes -=CHACHA_BLOCK_SIZE *4 ;
@@ -123,6 +123,43 @@ static void chacha_dosimd(u32 *state, u8 *dst, const u8 *src, } } +void hchacha_block_arch (const u32 *state,u32 *stream,int nrounds) +{ + state =PTR_ALIGN (state,CHACHA_STATE_ALIGN ); + +if (!static_branch_likely (&chacha_use_simd) || !crypto_simd_usable ()) { +hchacha_block_generic (state, stream, nrounds); + }else { +kernel_fpu_begin (); +hchacha_block_ssse3 (state, stream, nrounds); +kernel_fpu_end (); + } +} +EXPORT_SYMBOL (hchacha_block_arch); + +void chacha_init_arch (u32 *state,const u32 *key,const u8 *iv) +{ + state =PTR_ALIGN (state,CHACHA_STATE_ALIGN ); + +chacha_init_generic (state, key, iv); +} +EXPORT_SYMBOL (chacha_init_arch); + +void chacha_crypt_arch (u32 *state,u8 *dst,const u8 *src,unsigned int bytes, +int nrounds) +{ + state =PTR_ALIGN (state,CHACHA_STATE_ALIGN ); + +if (!static_branch_likely (&chacha_use_simd) || !crypto_simd_usable () || + bytes <=CHACHA_BLOCK_SIZE ) +return chacha_crypt_generic (state, dst, src, bytes, nrounds); + +kernel_fpu_begin (); +chacha_dosimd (state, dst, src, bytes, nrounds); +kernel_fpu_end (); +} +EXPORT_SYMBOL (chacha_crypt_arch); +static int chacha_simd_stream_xor (struct skcipher_request *req,const struct chacha_ctx *ctx,const u8 *iv) {
@@ -143,7 +180,8 @@ static int chacha_simd_stream_xor(struct skcipher_request *req,if (nbytes < walk.total) nbytes =round_down (nbytes, walk.stride); -if (!crypto_simd_usable ()) { +if (!static_branch_likely (&chacha_use_simd) || + !crypto_simd_usable ()) {chacha_crypt_generic (state, walk.dst.virt.addr, walk.src.virt.addr, nbytes, ctx->nrounds);
@@ -246,18 +284,21 @@ static struct skcipher_alg algs[] = {static int __init chacha_simd_mod_init (void ) {if (!boot_cpu_has (X86_FEATURE_SSSE3 )) -return -ENODEV ; - -#ifdef CONFIG_AS_AVX2 - chacha_use_avx2 =boot_cpu_has (X86_FEATURE_AVX ) && -boot_cpu_has (X86_FEATURE_AVX2 ) && -cpu_has_xfeatures (XFEATURE_MASK_SSE |XFEATURE_MASK_YMM ,NULL ); -#ifdef CONFIG_AS_AVX512 - chacha_use_avx512vl = chacha_use_avx2 && -boot_cpu_has (X86_FEATURE_AVX512VL ) && -boot_cpu_has (X86_FEATURE_AVX512BW );/* kmovq */ -#endif -#endif +return 0 ; + +static_branch_enable (&chacha_use_simd); + +if (IS_ENABLED (CONFIG_AS_AVX2 ) && +boot_cpu_has (X86_FEATURE_AVX ) && +boot_cpu_has (X86_FEATURE_AVX2 ) && +cpu_has_xfeatures (XFEATURE_MASK_SSE |XFEATURE_MASK_YMM ,NULL )) { +static_branch_enable (&chacha_use_avx2); + +if (IS_ENABLED (CONFIG_AS_AVX512 ) && +boot_cpu_has (X86_FEATURE_AVX512VL ) && +boot_cpu_has (X86_FEATURE_AVX512BW ))/* kmovq */ +static_branch_enable (&chacha_use_avx512vl); + }return crypto_register_skciphers (algs,ARRAY_SIZE (algs)); }
crypto/Kconfig +1 -0
@@ -1437,6 +1437,7 @@ config CRYPTO_CHACHA20_X86_64 depends on X86 && 64BIT select CRYPTO_SKCIPHER select CRYPTO_LIB_CHACHA_GENERIC + select CRYPTO_ARCH_HAVE_LIB_CHACHA help SSSE3, AVX2, and AVX-512VL optimized implementations of the ChaCha20, XChaCha20, and XChaCha12 stream ciphers.
include/crypto/chacha.h +6 -0
@@ -25,6 +25,12 @@#define CHACHA_BLOCK_SIZE 64#define CHACHAPOLY_IV_SIZE 12 +#ifdef CONFIG_X86_64 +#define CHACHA_STATE_WORDS ((CHACHA_BLOCK_SIZE + 12) / sizeof(u32)) +#else +#define CHACHA_STATE_WORDS (CHACHA_BLOCK_SIZE / sizeof(u32)) +#endif +/* 192-bit nonce, then 64-bit stream position */ #define XCHACHA_IV_SIZE 32
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 5/34 crypto: arm64/chacha - depend on generic chacha library instead of crypto driver
In reply to Ard BiesheuvelDepend on the generic ChaCha library routines instead of pulling in the generic ChaCha skcipher driver, which is more than we need, and makes managing the dependencies between the generic library, generic driver, accelerated library and driver more complicated.
While at it, drop the logic to prefer the scalar code on short inputs. Turning the NEON on and off is cheap these days, and one major use case for ChaCha20 is ChaCha20-Poly1305, which is guaranteed to hit the scalar path upon every invocation (when doing the Poly1305 nonce generation)
--- arch/arm64/crypto/Kconfig | 2 +- arch/arm64/crypto/chacha-neon-glue.c | 40 +++++++++++--------- 2 files changed, 23 insertions(+), 19 deletions(-)
arch/arm64/crypto/Kconfig +1 -1
@@ -103,7 +103,7 @@ config CRYPTO_CHACHA20_NEON tristate "ChaCha20, XChaCha20, and XChaCha12 stream ciphers using NEON instructions" depends on KERNEL_MODE_NEON select CRYPTO_SKCIPHER - select CRYPTO_CHACHA20 + select CRYPTO_LIB_CHACHA_GENERIC config CRYPTO_NHPOLY1305_NEON tristate "NHPoly1305 hash function using NEON instructions (for Adiantum)"
arch/arm64/crypto/chacha-neon-glue.c +22 -18
@@ -68,7 +68,7 @@ static int chacha_neon_stream_xor(struct skcipher_request *req, err =skcipher_walk_virt (&walk, req, false); -crypto_chacha_init (state, ctx, iv); +chacha_init_generic (state, ctx->key, iv);while (walk.nbytes >0 ) {unsigned int nbytes = walk.nbytes;
@@ -76,10 +76,16 @@ static int chacha_neon_stream_xor(struct skcipher_request *req,if (nbytes < walk.total) nbytes =rounddown (nbytes, walk.stride); -kernel_neon_begin (); -chacha_doneon (state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes, ctx->nrounds); -kernel_neon_end (); +if (!crypto_simd_usable ()) { +chacha_crypt_generic (state, walk.dst.virt.addr, + walk.src.virt.addr, nbytes, + ctx->nrounds); + }else { +kernel_neon_begin (); +chacha_doneon (state, walk.dst.virt.addr, + walk.src.virt.addr, nbytes, ctx->nrounds); +kernel_neon_end (); + } err =skcipher_walk_done (&walk, walk.nbytes - nbytes); }
@@ -91,9 +97,6 @@ static int chacha_neon(struct skcipher_request *req)struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm); -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ()) -return crypto_chacha_crypt (req); -return chacha_neon_stream_xor (req, ctx, req->iv); }
@@ -105,14 +108,15 @@ static int xchacha_neon(struct skcipher_request *req)u32 state[16 ];u8 real_iv[16 ]; -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ()) -return crypto_xchacha_crypt (req); +chacha_init_generic (state, ctx->key, req->iv); -crypto_chacha_init (state, ctx, req->iv); - -kernel_neon_begin (); -hchacha_block_neon (state, subctx.key, ctx->nrounds); -kernel_neon_end (); +if (crypto_simd_usable ()) { +kernel_neon_begin (); +hchacha_block_neon (state, subctx.key, ctx->nrounds); +kernel_neon_end (); + }else { +hchacha_block_generic (state, subctx.key, ctx->nrounds); + } subctx.nrounds = ctx->nrounds;memcpy (&real_iv[0 ], req->iv +24 ,8 );
@@ -134,7 +138,7 @@ static struct skcipher_alg algs[] = { .ivsize = CHACHA_IV_SIZE, .chunksize = CHACHA_BLOCK_SIZE, .walksize = 5 * CHACHA_BLOCK_SIZE, - .setkey = crypto_chacha20_setkey, + .setkey = chacha20_setkey, .encrypt = chacha_neon, .decrypt = chacha_neon, }, {
@@ -150,7 +154,7 @@ static struct skcipher_alg algs[] = { .ivsize = XCHACHA_IV_SIZE, .chunksize = CHACHA_BLOCK_SIZE, .walksize = 5 * CHACHA_BLOCK_SIZE, - .setkey = crypto_chacha20_setkey, + .setkey = chacha20_setkey, .encrypt = xchacha_neon, .decrypt = xchacha_neon, }, {
@@ -166,7 +170,7 @@ static struct skcipher_alg algs[] = { .ivsize = XCHACHA_IV_SIZE, .chunksize = CHACHA_BLOCK_SIZE, .walksize = 5 * CHACHA_BLOCK_SIZE, - .setkey = crypto_chacha12_setkey, + .setkey = chacha12_setkey, .encrypt = xchacha_neon, .decrypt = xchacha_neon, }
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 6/34 crypto: arm64/chacha - expose arm64 ChaCha routine as library function
In reply to Ard BiesheuvelExpose the accelerated NEON ChaCha routine directly as a symbol export so that users of the ChaCha library API can use it directly.
Given that calls into the library API will always go through the routines in this module if it is enabled, switch to static keys to select the optimal implementation available (which may be none at all, in which case we defer to the generic implementation for all invocations).
--- arch/arm64/crypto/Kconfig | 1 + arch/arm64/crypto/chacha-neon-glue.c | 53 ++++++++++++++++---- 2 files changed, 43 insertions(+), 11 deletions(-)
arch/arm64/crypto/Kconfig +1 -0
@@ -104,6 +104,7 @@ config CRYPTO_CHACHA20_NEON depends on KERNEL_MODE_NEON select CRYPTO_SKCIPHER select CRYPTO_LIB_CHACHA_GENERIC + select CRYPTO_ARCH_HAVE_LIB_CHACHA config CRYPTO_NHPOLY1305_NEON tristate "NHPoly1305 hash function using NEON instructions (for Adiantum)"
arch/arm64/crypto/chacha-neon-glue.c +42 -11
95 lines not on this page · raw message
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 7/34 crypto: arm/chacha - import Eric Biggers's scalar accelerated ChaCha code
In reply to Ard Biesheuvel--- arch/arm/crypto/chacha-scalar-core.S | 461 ++++++++++++++++++++ 1 file changed, 461 insertions(+)
arch/arm/crypto/chacha-scalar-core.S new file +461 -0
@@ -0,0 +1,461 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2018 Google, Inc. + */ + +#include <linux/linkage.h> +#include <asm/assembler.h> + +/* + * Design notes: + * + * 16 registers would be needed to hold the state matrix, but only 14 are + * available because 'sp' and 'pc' cannot be used. So we spill the elements + * (x8, x9) to the stack and swap them out with (x10, x11). This adds one + * 'ldrd' and one 'strd' instruction per round. + * + * All rotates are performed using the implicit rotate operand accepted by the + * 'add' and 'eor' instructions. This is faster than using explicit rotate + * instructions. To make this work, we allow the values in the second and last + * rows of the ChaCha state matrix (rows 'b' and 'd') to temporarily have the + * wrong rotation amount. The rotation amount is then fixed up just in time + * when the values are used. 'brot' is the number of bits the values in row 'b' + * need to be rotated right to arrive at the correct values, and 'drot' + * similarly for row 'd'. (brot, drot) start out as (0, 0) but we make it such + * that they end up as (25, 24) after every round. + */ + + // ChaCha state registers + X0 .req r0 + X1 .req r1 + X2 .req r2 + X3 .req r3 + X4 .req r4 + X5 .req r5 + X6 .req r6 + X7 .req r7 + X8_X10 .req r8 // shared by x8 and x10 + X9_X11 .req r9 // shared by x9 and x11 + X12 .req r10 + X13 .req r11 + X14 .req r12 + X15 .req r14 + +.Lexpand_32byte_k: + // "expand 32-byte k" + .word 0x61707865, 0x3320646e, 0x79622d32, 0x6b206574 + +#ifdef __thumb2__ +# define adrl adr +#endif + +.macro __rev out, in, t0, t1, t2 +.if __LINUX_ARM_ARCH__ >= 6 + rev \out, \in +.else + lsl \t0, \in, #24 + and \t1, \in, #0xff00 + and \t2, \in, #0xff0000 + orr \out, \t0, \in, lsr #24 + orr \out, \out, \t1, lsl #8 + orr \out, \out, \t2, lsr #8 +.endif +.endm + +.macro _le32_bswap x, t0, t1, t2 +#ifdef __ARMEB__ + __rev \x, \x, \t0, \t1, \t2 +#endif +.endm + +.macro _le32_bswap_4x a, b, c, d, t0, t1, t2 + _le32_bswap \a, \t0, \t1, \t2 + _le32_bswap \b, \t0, \t1, \t2 + _le32_bswap \c, \t0, \t1, \t2 + _le32_bswap \d, \t0, \t1, \t2 +.endm + +.macro __ldrd a, b, src, offset +#if __LINUX_ARM_ARCH__ >= 6 + ldrd \a, \b, [\src, #\offset] +#else + ldr \a, [\src, #\offset] + ldr \b, [\src, #\offset + 4] +#endif +.endm + +.macro __strd a, b, dst, offset +#if __LINUX_ARM_ARCH__ >= 6 + strd \a, \b, [\dst, #\offset] +#else + str \a, [\dst, #\offset] + str \b, [\dst, #\offset + 4] +#endif +.endm + +.macro _halfround a1, b1, c1, d1, a2, b2, c2, d2 + + // a += b; d ^= a; d = rol(d, 16); + add \a1, \a1, \b1, ror #brot + add \a2, \a2, \b2, ror #brot + eor \d1, \a1, \d1, ror #drot + eor \d2, \a2, \d2, ror #drot + // drot == 32 - 16 == 16 + + // c += d; b ^= c; b = rol(b, 12); + add \c1, \c1, \d1, ror #16 + add \c2, \c2, \d2, ror #16 + eor \b1, \c1, \b1, ror #brot + eor \b2, \c2, \b2, ror #brot + // brot == 32 - 12 == 20 + + // a += b; d ^= a; d = rol(d, 8); + add \a1, \a1, \b1, ror #20 + add \a2, \a2, \b2, ror #20 + eor \d1, \a1, \d1, ror #16 + eor \d2, \a2, \d2, ror #16 + // drot == 32 - 8 == 24 + + // c += d; b ^= c; b = rol(b, 7); + add \c1, \c1, \d1, ror #24 + add \c2, \c2, \d2, ror #24 + eor \b1, \c1, \b1, ror #20 + eor \b2, \c2, \b2, ror #20 + // brot == 32 - 7 == 25 +.endm + +.macro _doubleround + + // column round + + // quarterrounds: (x0, x4, x8, x12) and (x1, x5, x9, x13) + _halfround X0, X4, X8_X10, X12, X1, X5, X9_X11, X13 + + // save (x8, x9); restore (x10, x11) + __strd X8_X10, X9_X11, sp, 0 + __ldrd X8_X10, X9_X11, sp, 8 + + // quarterrounds: (x2, x6, x10, x14) and (x3, x7, x11, x15) + _halfround X2, X6, X8_X10, X14, X3, X7, X9_X11, X15 + + .set brot, 25 + .set drot, 24 + + // diagonal round + + // quarterrounds: (x0, x5, x10, x15) and (x1, x6, x11, x12) + _halfround X0, X5, X8_X10, X15, X1, X6, X9_X11, X12 + + // save (x10, x11); restore (x8, x9) + __strd X8_X10, X9_X11, sp, 8 + __ldrd X8_X10, X9_X11, sp, 0 + + // quarterrounds: (x2, x7, x8, x13) and (x3, x4, x9, x14) + _halfround X2, X7, X8_X10, X13, X3, X4, X9_X11, X14 +.endm + +.macro _chacha_permute nrounds + .set brot, 0 + .set drot, 0 + .rept \nrounds / 2 + _doubleround + .endr +.endm + +.macro _chacha nrounds + +.Lnext_block\@: + // Stack: unused0-unused1 x10-x11 x0-x15 OUT IN LEN + // Registers contain x0-x9,x12-x15. + + // Do the core ChaCha permutation to update x0-x15. + _chacha_permute \nrounds + + add sp, #8 + // Stack: x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers contain x0-x9,x12-x15. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // Free up some registers (r8-r12,r14) by pushing (x8-x9,x12-x15). + push {X8_X10, X9_X11, X12, X13, X14, X15} + + // Load (OUT, IN, LEN). + ldr r14, [sp, #96] + ldr r12, [sp, #100] + ldr r11, [sp, #104] + + orr r10, r14, r12 + + // Use slow path if fewer than 64 bytes remain. + cmp r11, #64 + blt .Lxor_slowpath\@ + + // Use slow path if IN and/or OUT isn't 4-byte aligned. Needed even on + // ARMv6+, since ldmia and stmia (used below) still require alignment. + tst r10, #3 + bne .Lxor_slowpath\@ + + // Fast path: XOR 64 bytes of aligned data. + + // Stack: x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is OUT. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // x0-x3 + __ldrd r8, r9, sp, 32 + __ldrd r10, r11, sp, 40 + add X0, X0, r8 + add X1, X1, r9 + add X2, X2, r10 + add X3, X3, r11 + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10 + ldmia r12!, {r8-r11} + eor X0, X0, r8 + eor X1, X1, r9 + eor X2, X2, r10 + eor X3, X3, r11 + stmia r14!, {X0-X3} + + // x4-x7 + __ldrd r8, r9, sp, 48 + __ldrd r10, r11, sp, 56 + add X4, r8, X4, ror #brot + add X5, r9, X5, ror #brot + ldmia r12!, {X0-X3} + add X6, r10, X6, ror #brot + add X7, r11, X7, ror #brot + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10 + eor X4, X4, X0 + eor X5, X5, X1 + eor X6, X6, X2 + eor X7, X7, X3 + stmia r14!, {X4-X7} + + // x8-x15 + pop {r0-r7} // (x8-x9,x12-x15,x10-x11) + __ldrd r8, r9, sp, 32 + __ldrd r10, r11, sp, 40 + add r0, r0, r8 // x8 + add r1, r1, r9 // x9 + add r6, r6, r10 // x10 + add r7, r7, r11 // x11 + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10 + ldmia r12!, {r8-r11} + eor r0, r0, r8 // x8 + eor r1, r1, r9 // x9 + eor r6, r6, r10 // x10 + eor r7, r7, r11 // x11 + stmia r14!, {r0,r1,r6,r7} + ldmia r12!, {r0,r1,r6,r7} + __ldrd r8, r9, sp, 48 + __ldrd r10, r11, sp, 56 + add r2, r8, r2, ror #drot // x12 + add r3, r9, r3, ror #drot // x13 + add r4, r10, r4, ror #drot // x14 + add r5, r11, r5, ror #drot // x15 + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11 + ldr r9, [sp, #72] // load LEN + eor r2, r2, r0 // x12 + eor r3, r3, r1 // x13 + eor r4, r4, r6 // x14 + eor r5, r5, r7 // x15 + subs r9, #64 // decrement and check LEN + stmia r14!, {r2-r5} + + beq .Ldone\@ + +.Lprepare_for_next_block\@: + + // Stack: x0-x15 OUT IN LEN + + // Increment block counter (x12) + add r8, #1 + + // Store updated (OUT, IN, LEN) + str r14, [sp, #64] + str r12, [sp, #68] + str r9, [sp, #72] + + mov r14, sp + + // Store updated block counter (x12) + str r8, [sp, #48] + + sub sp, #16 + + // Reload state and do next block + ldmia r14!, {r0-r11} // load x0-x11 + __strd r10, r11, sp, 8 // store x10-x11 before state + ldmia r14, {r10-r12,r14} // load x12-x15 + b .Lnext_block\@ + +.Lxor_slowpath\@: + // Slow path: < 64 bytes remaining, or unaligned input or output buffer. + // We handle it by storing the 64 bytes of keystream to the stack, then + // XOR-ing the needed portion with the data. + + // Allocate keystream buffer + sub sp, #64 + mov r14, sp + + // Stack: ks0-ks15 x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is &ks0. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // Save keystream for x0-x3 + __ldrd r8, r9, sp, 96 + __ldrd r10, r11, sp, 104 + add X0, X0, r8 + add X1, X1, r9 + add X2, X2, r10 + add X3, X3, r11 + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10 + stmia r14!, {X0-X3} + + // Save keystream for x4-x7 + __ldrd r8, r9, sp, 112 + __ldrd r10, r11, sp, 120 + add X4, r8, X4, ror #brot + add X5, r9, X5, ror #brot + add X6, r10, X6, ror #brot + add X7, r11, X7, ror #brot + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10 + add r8, sp, #64 + stmia r14!, {X4-X7} + + // Save keystream for x8-x15 + ldm r8, {r0-r7} // (x8-x9,x12-x15,x10-x11) + __ldrd r8, r9, sp, 128 + __ldrd r10, r11, sp, 136 + add r0, r0, r8 // x8 + add r1, r1, r9 // x9 + add r6, r6, r10 // x10 + add r7, r7, r11 // x11 + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10 + stmia r14!, {r0,r1,r6,r7} + __ldrd r8, r9, sp, 144 + __ldrd r10, r11, sp, 152 + add r2, r8, r2, ror #drot // x12 + add r3, r9, r3, ror #drot // x13 + add r4, r10, r4, ror #drot // x14 + add r5, r11, r5, ror #drot // x15 + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11 + stmia r14, {r2-r5} + + // Stack: ks0-ks15 unused0-unused7 x0-x15 OUT IN LEN + // Registers: r8 is block counter, r12 is IN. + + ldr r9, [sp, #168] // LEN + ldr r14, [sp, #160] // OUT + cmp r9, #64 + mov r0, sp + movle r1, r9 + movgt r1, #64 + // r1 is number of bytes to XOR, in range [1, 64] + +.if __LINUX_ARM_ARCH__ < 6 + orr r2, r12, r14 + tst r2, #3 // IN or OUT misaligned? + bne .Lxor_next_byte\@ +.endif + + // XOR a word at a time +.rept 16 + subs r1, #4 + blt .Lxor_words_done\@ + ldr r2, [r12], #4 + ldr r3, [r0], #4 + eor r2, r2, r3 + str r2, [r14], #4 +.endr + b .Lxor_slowpath_done\@ +.Lxor_words_done\@: + ands r1, r1, #3 + beq .Lxor_slowpath_done\@ + + // XOR a byte at a time +.Lxor_next_byte\@: + ldrb r2, [r12], #1 + ldrb r3, [r0], #1 + eor r2, r2, r3 + strb r2, [r14], #1 + subs r1, #1 + bne .Lxor_next_byte\@ + +.Lxor_slowpath_done\@: + subs r9, #64 + add sp, #96 + bgt .Lprepare_for_next_block\@ + +.Ldone\@: +.endm // _chacha + +/* + * void chacha20_arm(u8 *out, const u8 *in, size_t len, const u32 key[8], + * const u32 iv[4]); + */ +ENTRY(chacha20_arm) + cmp r2, #0 // len == 0? + reteq lr + + push {r0-r2,r4-r11,lr} + + // Push state x0-x15 onto stack. + // Also store an extra copy of x10-x11 just before the state. + + ldr r4, [sp, #48] // iv + mov r0, sp + sub sp, #80 + + // iv: x12-x15 + ldm r4, {X12,X13,X14,X15} + stmdb r0!, {X12,X13,X14,X15} + + // key: x4-x11 + __ldrd X8_X10, X9_X11, r3, 24 + __strd X8_X10, X9_X11, sp, 8 + stmdb r0!, {X8_X10, X9_X11} + ldm r3, {X4-X9_X11} + stmdb r0!, {X4-X9_X11} + + // constants: x0-x3 + adrl X3, .Lexpand_32byte_k + ldm X3, {X0-X3} + __strd X0, X1, sp, 16 + __strd X2, X3, sp, 24 + + _chacha 20 + + add sp, #76 + pop {r4-r11, pc} +ENDPROC(chacha20_arm) + +/* + * void hchacha20_arm(const u32 state[16], u32 out[8]); + */ +ENTRY(hchacha20_arm) + push {r1,r4-r11,lr} + + mov r14, r0 + ldmia r14!, {r0-r11} // load x0-x11 + push {r10-r11} // store x10-x11 to stack + ldm r14, {r10-r12,r14} // load x12-x15 + sub sp, #8 + + _chacha_permute 20 + + // Skip over (unused0-unused1, x10-x11) + add sp, #16 + + // Fix up rotations of x12-x15 + ror X12, X12, #drot + ror X13, X13, #drot + pop {r4} // load 'out' + ror X14, X14, #drot + ror X15, X15, #drot + + // Store (x0-x3,x12-x15) to 'out' + stm r4, {X0,X1,X2,X3,X12,X13,X14,X15} + + pop {r4-r11,pc} +ENDPROC(hchacha20_arm)
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 8/34 crypto: arm/chacha - remove dependency on generic ChaCha driver
In reply to Ard BiesheuvelInstead of falling back to the generic ChaCha skcipher driver for non-SIMD cases, use a fast scalar implementation for ARM authored by Eric Biggers. This removes the module dependency on chacha-generic altogether, which also simplifies things when we expose the ChaCha library interface from this module.
--- arch/arm/crypto/Kconfig | 4 +- arch/arm/crypto/Makefile | 3 +- arch/arm/crypto/chacha-glue.c | 304 ++++++++++++++++++++ arch/arm/crypto/chacha-neon-glue.c | 202 ------------- arch/arm/crypto/chacha-scalar-core.S | 65 +++-- arch/arm64/crypto/chacha-neon-glue.c | 2 +- 6 files changed, 340 insertions(+), 240 deletions(-)
arch/arm/crypto/Kconfig +1 -3
@@ -128,10 +128,8 @@ config CRYPTO_CRC32_ARM_CE select CRYPTO_HASH config CRYPTO_CHACHA20_NEON - tristate "NEON accelerated ChaCha stream cipher algorithms" - depends on KERNEL_MODE_NEON + tristate "NEON and scalar accelerated ChaCha stream cipher algorithms" select CRYPTO_SKCIPHER - select CRYPTO_CHACHA20 config CRYPTO_NHPOLY1305_NEON tristate "NEON accelerated NHPoly1305 hash function (for Adiantum)"
arch/arm/crypto/Makefile +2 -1
@@ -33,7 +33,8 @@ aes-arm-ce-y := aes-ce-core.o aes-ce-glue.o ghash-arm-ce-y := ghash-ce-core.o ghash-ce-glue.o crct10dif-arm-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o -chacha-neon-y := chacha-neon-core.o chacha-neon-glue.o +chacha-neon-y := chacha-scalar-core.o chacha-glue.o +chacha-neon-$(CONFIG_KERNEL_MODE_NEON) += chacha-neon-core.o nhpoly1305-neon-y := nh-neon-core.o nhpoly1305-neon-glue.o ifdef REGENERATE_ARM_CRYPTO
arch/arm/crypto/chacha-glue.c new file +304 -0
304 lines not on this page · raw message
arch/arm/crypto/chacha-neon-glue.c deleted +0 -202
202 lines not on this page · raw message
arch/arm/crypto/chacha-scalar-core.S +32 -33
111 lines not on this page · raw message
arch/arm64/crypto/chacha-neon-glue.c +1 -1
@@ -1,5 +1,5 @@/* -* ARM NEON accelerated ChaCha and XChaCha stream ciphers, +* ARM NEON and scalar accelerated ChaCha and XChaCha stream ciphers, * including ChaCha20 (RFC7539) * * Copyright (C) 2016 - 2017 Linaro, Ltd. <[email protected]>
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 9/34 crypto: arm/chacha - expose ARM ChaCha routine as library function
In reply to Ard BiesheuvelExpose the accelerated NEON ChaCha routine directly as a symbol export so that users of the ChaCha library API can use it directly.
Given that calls into the library API will always go through the routines in this module if it is enabled, switch to static keys to select the optimal implementation available (which may be none at all, in which case we defer to the generic implementation for all invocations).
--- arch/arm/crypto/Kconfig | 1 + arch/arm/crypto/chacha-glue.c | 41 +++++++++++++++++++- 2 files changed, 41 insertions(+), 1 deletion(-)
arch/arm/crypto/Kconfig +1 -0
@@ -130,6 +130,7 @@ config CRYPTO_CRC32_ARM_CE config CRYPTO_CHACHA20_NEON tristate "NEON and scalar accelerated ChaCha stream cipher algorithms" select CRYPTO_SKCIPHER + select CRYPTO_ARCH_HAVE_LIB_CHACHA config CRYPTO_NHPOLY1305_NEON tristate "NEON accelerated NHPoly1305 hash function (for Adiantum)"
arch/arm/crypto/chacha-glue.c +40 -1
67 lines not on this page · raw message
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 10/34 crypto: mips/chacha - import 32r2 ChaCha code from Zinc
In reply to Ard BiesheuvelFrom: "Jason A. Donenfeld" <[email protected]>
This imports the accelerated MIPS 32r2 ChaCha20 implementation from the Zinc patch set.
--- arch/mips/crypto/chacha-core.S | 424 ++++++++++++++++++++ 1 file changed, 424 insertions(+)
arch/mips/crypto/chacha-core.S new file +424 -0
@@ -0,0 +1,424 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright (C) 2016-2018 René van Dorst <[email protected]>. All Rights Reserved. + * Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. + */ + +#define MASK_U32 0x3c +#define CHACHA20_BLOCK_SIZE 64 +#define STACK_SIZE 32 + +#define X0 $t0 +#define X1 $t1 +#define X2 $t2 +#define X3 $t3 +#define X4 $t4 +#define X5 $t5 +#define X6 $t6 +#define X7 $t7 +#define X8 $t8 +#define X9 $t9 +#define X10 $v1 +#define X11 $s6 +#define X12 $s5 +#define X13 $s4 +#define X14 $s3 +#define X15 $s2 +/* Use regs which are overwritten on exit for Tx so we don't leak clear data. */ +#define T0 $s1 +#define T1 $s0 +#define T(n) T ## n +#define X(n) X ## n + +/* Input arguments */ +#define STATE $a0 +#define OUT $a1 +#define IN $a2 +#define BYTES $a3 + +/* Output argument */ +/* NONCE[0] is kept in a register and not in memory. + * We don't want to touch original value in memory. + * Must be incremented every loop iteration. + */ +#define NONCE_0 $v0 + +/* SAVED_X and SAVED_CA are set in the jump table. + * Use regs which are overwritten on exit else we don't leak clear data. + * They are used to handling the last bytes which are not multiple of 4. + */ +#define SAVED_X X15 +#define SAVED_CA $s7 + +#define IS_UNALIGNED $s7 + +#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ +#define MSB 0 +#define LSB 3 +#define ROTx rotl +#define ROTR(n) rotr n, 24 +#define CPU_TO_LE32(n) \ + wsbh n; \ + rotr n, 16; +#else +#define MSB 3 +#define LSB 0 +#define ROTx rotr +#define CPU_TO_LE32(n) +#define ROTR(n) +#endif + +#define FOR_EACH_WORD(x) \ + x( 0); \ + x( 1); \ + x( 2); \ + x( 3); \ + x( 4); \ + x( 5); \ + x( 6); \ + x( 7); \ + x( 8); \ + x( 9); \ + x(10); \ + x(11); \ + x(12); \ + x(13); \ + x(14); \ + x(15); + +#define FOR_EACH_WORD_REV(x) \ + x(15); \ + x(14); \ + x(13); \ + x(12); \ + x(11); \ + x(10); \ + x( 9); \ + x( 8); \ + x( 7); \ + x( 6); \ + x( 5); \ + x( 4); \ + x( 3); \ + x( 2); \ + x( 1); \ + x( 0); + +#define PLUS_ONE_0 1 +#define PLUS_ONE_1 2 +#define PLUS_ONE_2 3 +#define PLUS_ONE_3 4 +#define PLUS_ONE_4 5 +#define PLUS_ONE_5 6 +#define PLUS_ONE_6 7 +#define PLUS_ONE_7 8 +#define PLUS_ONE_8 9 +#define PLUS_ONE_9 10 +#define PLUS_ONE_10 11 +#define PLUS_ONE_11 12 +#define PLUS_ONE_12 13 +#define PLUS_ONE_13 14 +#define PLUS_ONE_14 15 +#define PLUS_ONE_15 16 +#define PLUS_ONE(x) PLUS_ONE_ ## x +#define _CONCAT3(a,b,c) a ## b ## c +#define CONCAT3(a,b,c) _CONCAT3(a,b,c) + +#define STORE_UNALIGNED(x) \ +CONCAT3(.Lchacha20_mips_xor_unaligned_, PLUS_ONE(x), _b: ;) \ + .if (x != 12); \ + lw T0, (x*4)(STATE); \ + .endif; \ + lwl T1, (x*4)+MSB ## (IN); \ + lwr T1, (x*4)+LSB ## (IN); \ + .if (x == 12); \ + addu X ## x, NONCE_0; \ + .else; \ + addu X ## x, T0; \ + .endif; \ + CPU_TO_LE32(X ## x); \ + xor X ## x, T1; \ + swl X ## x, (x*4)+MSB ## (OUT); \ + swr X ## x, (x*4)+LSB ## (OUT); + +#define STORE_ALIGNED(x) \ +CONCAT3(.Lchacha20_mips_xor_aligned_, PLUS_ONE(x), _b: ;) \ + .if (x != 12); \ + lw T0, (x*4)(STATE); \ + .endif; \ + lw T1, (x*4) ## (IN); \ + .if (x == 12); \ + addu X ## x, NONCE_0; \ + .else; \ + addu X ## x, T0; \ + .endif; \ + CPU_TO_LE32(X ## x); \ + xor X ## x, T1; \ + sw X ## x, (x*4) ## (OUT); + +/* Jump table macro. + * Used for setup and handling the last bytes, which are not multiple of 4. + * X15 is free to store Xn + * Every jumptable entry must be equal in size. + */ +#define JMPTBL_ALIGNED(x) \ +.Lchacha20_mips_jmptbl_aligned_ ## x: ; \ + .set noreorder; \ + b .Lchacha20_mips_xor_aligned_ ## x ## _b; \ + .if (x == 12); \ + addu SAVED_X, X ## x, NONCE_0; \ + .else; \ + addu SAVED_X, X ## x, SAVED_CA; \ + .endif; \ + .set reorder + +#define JMPTBL_UNALIGNED(x) \ +.Lchacha20_mips_jmptbl_unaligned_ ## x: ; \ + .set noreorder; \ + b .Lchacha20_mips_xor_unaligned_ ## x ## _b; \ + .if (x == 12); \ + addu SAVED_X, X ## x, NONCE_0; \ + .else; \ + addu SAVED_X, X ## x, SAVED_CA; \ + .endif; \ + .set reorder + +#define AXR(A, B, C, D, K, L, M, N, V, W, Y, Z, S) \ + addu X(A), X(K); \ + addu X(B), X(L); \ + addu X(C), X(M); \ + addu X(D), X(N); \ + xor X(V), X(A); \ + xor X(W), X(B); \ + xor X(Y), X(C); \ + xor X(Z), X(D); \ + rotl X(V), S; \ + rotl X(W), S; \ + rotl X(Y), S; \ + rotl X(Z), S; + +.text +.set reorder +.set noat +.globl chacha20_mips +.ent chacha20_mips +chacha20_mips: + .frame $sp, STACK_SIZE, $ra + + addiu $sp, -STACK_SIZE + + /* Return bytes = 0. */ + beqz BYTES, .Lchacha20_mips_end + + lw NONCE_0, 48(STATE) + + /* Save s0-s7 */ + sw $s0, 0($sp) + sw $s1, 4($sp) + sw $s2, 8($sp) + sw $s3, 12($sp) + sw $s4, 16($sp) + sw $s5, 20($sp) + sw $s6, 24($sp) + sw $s7, 28($sp) + + /* Test IN or OUT is unaligned. + * IS_UNALIGNED = ( IN | OUT ) & 0x00000003 + */ + or IS_UNALIGNED, IN, OUT + andi IS_UNALIGNED, 0x3 + + /* Set number of rounds */ + li $at, 20 + + b .Lchacha20_rounds_start + +.align 4 +.Loop_chacha20_rounds: + addiu IN, CHACHA20_BLOCK_SIZE + addiu OUT, CHACHA20_BLOCK_SIZE + addiu NONCE_0, 1 + +.Lchacha20_rounds_start: + lw X0, 0(STATE) + lw X1, 4(STATE) + lw X2, 8(STATE) + lw X3, 12(STATE) + + lw X4, 16(STATE) + lw X5, 20(STATE) + lw X6, 24(STATE) + lw X7, 28(STATE) + lw X8, 32(STATE) + lw X9, 36(STATE) + lw X10, 40(STATE) + lw X11, 44(STATE) + + move X12, NONCE_0 + lw X13, 52(STATE) + lw X14, 56(STATE) + lw X15, 60(STATE) + +.Loop_chacha20_xor_rounds: + addiu $at, -2 + AXR( 0, 1, 2, 3, 4, 5, 6, 7, 12,13,14,15, 16); + AXR( 8, 9,10,11, 12,13,14,15, 4, 5, 6, 7, 12); + AXR( 0, 1, 2, 3, 4, 5, 6, 7, 12,13,14,15, 8); + AXR( 8, 9,10,11, 12,13,14,15, 4, 5, 6, 7, 7); + AXR( 0, 1, 2, 3, 5, 6, 7, 4, 15,12,13,14, 16); + AXR(10,11, 8, 9, 15,12,13,14, 5, 6, 7, 4, 12); + AXR( 0, 1, 2, 3, 5, 6, 7, 4, 15,12,13,14, 8); + AXR(10,11, 8, 9, 15,12,13,14, 5, 6, 7, 4, 7); + bnez $at, .Loop_chacha20_xor_rounds + + addiu BYTES, -(CHACHA20_BLOCK_SIZE) + + /* Is data src/dst unaligned? Jump */ + bnez IS_UNALIGNED, .Loop_chacha20_unaligned + + /* Set number rounds here to fill delayslot. */ + li $at, 20 + + /* BYTES < 0, it has no full block. */ + bltz BYTES, .Lchacha20_mips_no_full_block_aligned + + FOR_EACH_WORD_REV(STORE_ALIGNED) + + /* BYTES > 0? Loop again. */ + bgtz BYTES, .Loop_chacha20_rounds + + /* Place this here to fill delay slot */ + addiu NONCE_0, 1 + + /* BYTES < 0? Handle last bytes */ + bltz BYTES, .Lchacha20_mips_xor_bytes + +.Lchacha20_mips_xor_done: + /* Restore used registers */ + lw $s0, 0($sp) + lw $s1, 4($sp) + lw $s2, 8($sp) + lw $s3, 12($sp) + lw $s4, 16($sp) + lw $s5, 20($sp) + lw $s6, 24($sp) + lw $s7, 28($sp) + + /* Write NONCE_0 back to right location in state */ + sw NONCE_0, 48(STATE) + +.Lchacha20_mips_end: + addiu $sp, STACK_SIZE + jr $ra + +.Lchacha20_mips_no_full_block_aligned: + /* Restore the offset on BYTES */ + addiu BYTES, CHACHA20_BLOCK_SIZE + + /* Get number of full WORDS */ + andi $at, BYTES, MASK_U32 + + /* Load upper half of jump table addr */ + lui T0, %hi(.Lchacha20_mips_jmptbl_aligned_0) + + /* Calculate lower half jump table offset */ + ins T0, $at, 1, 6 + + /* Add offset to STATE */ + addu T1, STATE, $at + + /* Add lower half jump table addr */ + addiu T0, %lo(.Lchacha20_mips_jmptbl_aligned_0) + + /* Read value from STATE */ + lw SAVED_CA, 0(T1) + + /* Store remaining bytecounter as negative value */ + subu BYTES, $at, BYTES + + jr T0 + + /* Jump table */ + FOR_EACH_WORD(JMPTBL_ALIGNED) + + +.Loop_chacha20_unaligned: + /* Set number rounds here to fill delayslot. */ + li $at, 20 + + /* BYTES > 0, it has no full block. */ + bltz BYTES, .Lchacha20_mips_no_full_block_unaligned + + FOR_EACH_WORD_REV(STORE_UNALIGNED) + + /* BYTES > 0? Loop again. */ + bgtz BYTES, .Loop_chacha20_rounds + + /* Write NONCE_0 back to right location in state */ + sw NONCE_0, 48(STATE) + + .set noreorder + /* Fall through to byte handling */ + bgez BYTES, .Lchacha20_mips_xor_done +.Lchacha20_mips_xor_unaligned_0_b: +.Lchacha20_mips_xor_aligned_0_b: + /* Place this here to fill delay slot */ + addiu NONCE_0, 1 + .set reorder + +.Lchacha20_mips_xor_bytes: + addu IN, $at + addu OUT, $at + /* First byte */ + lbu T1, 0(IN) + addiu $at, BYTES, 1 + CPU_TO_LE32(SAVED_X) + ROTR(SAVED_X) + xor T1, SAVED_X + sb T1, 0(OUT) + beqz $at, .Lchacha20_mips_xor_done + /* Second byte */ + lbu T1, 1(IN) + addiu $at, BYTES, 2 + ROTx SAVED_X, 8 + xor T1, SAVED_X + sb T1, 1(OUT) + beqz $at, .Lchacha20_mips_xor_done + /* Third byte */ + lbu T1, 2(IN) + ROTx SAVED_X, 8 + xor T1, SAVED_X + sb T1, 2(OUT) + b .Lchacha20_mips_xor_done + +.Lchacha20_mips_no_full_block_unaligned: + /* Restore the offset on BYTES */ + addiu BYTES, CHACHA20_BLOCK_SIZE + + /* Get number of full WORDS */ + andi $at, BYTES, MASK_U32 + + /* Load upper half of jump table addr */ + lui T0, %hi(.Lchacha20_mips_jmptbl_unaligned_0) + + /* Calculate lower half jump table offset */ + ins T0, $at, 1, 6 + + /* Add offset to STATE */ + addu T1, STATE, $at + + /* Add lower half jump table addr */ + addiu T0, %lo(.Lchacha20_mips_jmptbl_unaligned_0) + + /* Read value from STATE */ + lw SAVED_CA, 0(T1) + + /* Store remaining bytecounter as negative value */ + subu BYTES, $at, BYTES + + jr T0 + + /* Jump table */ + FOR_EACH_WORD(JMPTBL_UNALIGNED) +.end chacha20_mips +.set at
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 11/34 crypto: mips/chacha - wire up accelerated 32r2 code from Zinc
In reply to Ard BiesheuvelThis integrates the accelerated MIPS 32r2 implementation of ChaCha into both the API and library interfaces of the kernel crypto stack.
The significance of this is that, in addition to becoming available as an accelerated library implementation, it can also be used by existing crypto API code such as Adiantum (for block encryption on ultra low performance cores) or IPsec using chacha20poly1305. These are use cases that have already opted into using the abstract crypto API. In order to support Adiantum, the core assembler routine has been adapted to take the round count as a function argument rather than hardcoding it to 20.
--- arch/mips/Makefile | 2 +- arch/mips/crypto/Makefile | 4 + arch/mips/crypto/chacha-core.S | 159 ++++++++++++++------ arch/mips/crypto/chacha-glue.c | 150 ++++++++++++++++++ crypto/Kconfig | 6 + 5 files changed, 277 insertions(+), 44 deletions(-)
arch/mips/Makefile +1 -1
@@ -323,7 +323,7 @@ libs-$(CONFIG_MIPS_FP_SUPPORT) += arch/mips/math-emu/ # See arch/mips/Kbuild for content of core part of the kernel core-y += arch/mips/ -drivers-$(CONFIG_MIPS_CRC_SUPPORT) += arch/mips/crypto/ +drivers-y += arch/mips/crypto/ drivers-$(CONFIG_OPROFILE) += arch/mips/oprofile/ # suspend and hibernation support
arch/mips/crypto/Makefile +4 -0
@@ -4,3 +4,7 @@ # obj-$(CONFIG_CRYPTO_CRC32_MIPS) += crc32-mips.o + +obj-$(CONFIG_CRYPTO_CHACHA_MIPS) += chacha-mips.o +chacha-mips-y := chacha-core.o chacha-glue.o +AFLAGS_chacha-core.o += -O2 # needed to fill branch delay slots
arch/mips/crypto/chacha-core.S +116 -43
316 lines not on this page · raw message
arch/mips/crypto/chacha-glue.c new file +150 -0
150 lines not on this page · raw message
crypto/Kconfig +6 -0
@@ -1442,6 +1442,12 @@ config CRYPTO_CHACHA20_X86_64 SSSE3, AVX2, and AVX-512VL optimized implementations of the ChaCha20, XChaCha20, and XChaCha12 stream ciphers. +config CRYPTO_CHACHA_MIPS + tristate "ChaCha stream cipher algorithms (MIPS 32r2 optimized)" + depends on CPU_MIPS32_R2 + select CRYPTO_BLKCIPHER + select CRYPTO_ARCH_HAVE_LIB_CHACHA + config CRYPTO_SEED tristate "SEED cipher algorithm" select CRYPTO_ALGAPI
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 12/34 crypto: chacha - unexport chacha_generic routines
In reply to Ard BiesheuvelNow that all users of generic ChaCha code have moved to the core library, there is no longer a need for the generic ChaCha skcpiher driver to export parts of it implementation for reuse by other drivers. So drop the exports, and make the symbols static.
--- crypto/chacha_generic.c | 26 ++++++-------------- include/crypto/internal/chacha.h | 10 -------- 2 files changed, 8 insertions(+), 28 deletions(-)
crypto/chacha_generic.c +8 -18
@@ -21,7 +21,7 @@ static int chacha_stream_xor(struct skcipher_request *req, err =skcipher_walk_virt (&walk, req, false); -crypto_chacha_init (state, ctx, iv); +chacha_init_generic (state, ctx->key, iv);while (walk.nbytes >0 ) {unsigned int nbytes = walk.nbytes;
@@ -37,36 +37,27 @@ static int chacha_stream_xor(struct skcipher_request *req,return err; } -void crypto_chacha_init (u32 *state,const struct chacha_ctx *ctx,const u8 *iv) -{ -chacha_init_generic (state, ctx->key, iv); -} -EXPORT_SYMBOL_GPL (crypto_chacha_init); - -int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keysize) +static int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key, +unsigned int keysize) {return chacha_setkey (tfm, key, keysize,20 ); } -EXPORT_SYMBOL_GPL (crypto_chacha20_setkey); -int crypto_chacha12_setkey (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keysize) +static int crypto_chacha12_setkey (struct crypto_skcipher *tfm,const u8 *key, +unsigned int keysize) {return chacha_setkey (tfm, key, keysize,12 ); } -EXPORT_SYMBOL_GPL (crypto_chacha12_setkey); -int crypto_chacha_crypt (struct skcipher_request *req) +static int crypto_chacha_crypt (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm);return chacha_stream_xor (req, ctx, req->iv); } -EXPORT_SYMBOL_GPL (crypto_chacha_crypt); -int crypto_xchacha_crypt (struct skcipher_request *req) +static int crypto_xchacha_crypt (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm);
@@ -75,7 +66,7 @@ int crypto_xchacha_crypt(struct skcipher_request *req)u8 real_iv[16 ];/* Compute the subkey given the original key and first 128 nonce bits */ -crypto_chacha_init (state, ctx, req->iv); +chacha_init_generic (state, ctx->key, req->iv);hchacha_block_generic (state, subctx.key, ctx->nrounds); subctx.nrounds = ctx->nrounds;
@@ -86,7 +77,6 @@ int crypto_xchacha_crypt(struct skcipher_request *req)/* Generate the stream and XOR it with the data */ return chacha_stream_xor (req, &subctx, real_iv); } -EXPORT_SYMBOL_GPL (crypto_xchacha_crypt);static struct skcipher_alg algs[] = { {
include/crypto/internal/chacha.h +0 -10
@@ -12,8 +12,6 @@ struct chacha_ctx {int nrounds; }; -void crypto_chacha_init (u32 *state,const struct chacha_ctx *ctx,const u8 *iv); -static inline int chacha_setkey (struct crypto_skcipher *tfm,const u8 *key,unsigned int keysize,int nrounds) {
@@ -42,12 +40,4 @@ static int inline chacha12_setkey(struct crypto_skcipher *tfm, const u8 *key, return chacha_setkey(tfm, key, keysize, 12); } -int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keysize); -int crypto_chacha12_setkey (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keysize); - -int crypto_chacha_crypt (struct skcipher_request *req); -int crypto_xchacha_crypt (struct skcipher_request *req); -#endif /* _CRYPTO_CHACHA_H */
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 13/34 crypto: poly1305 - move core routines into a separate library
In reply to Ard BiesheuvelMove the core Poly1305 routines shared between the generic Poly1305 shash driver and the Adiantum and NHPoly1305 drivers into a separate library so that using just this pieces does not pull in the crypto API pieces of the generic Poly1305 routine.
In a subsequent patch, we will augment this generic library with init/update/final routines so that Poyl1305 algorithm can be used directly without the need for using the crypto API's shash abstraction.
--- arch/x86/crypto/poly1305_glue.c | 2 +- crypto/Kconfig | 5 +- crypto/adiantum.c | 5 +- crypto/nhpoly1305.c | 3 +- crypto/poly1305_generic.c | 195 +------------------- include/crypto/internal/poly1305.h | 67 +++++++ include/crypto/poly1305.h | 23 --- lib/crypto/Kconfig | 3 + lib/crypto/Makefile | 3 + lib/crypto/poly1305.c | 158 ++++++++++++++++ 10 files changed, 248 insertions(+), 216 deletions(-)
arch/x86/crypto/poly1305_glue.c +1 -1
@@ -7,8 +7,8 @@#include <crypto/algapi.h> #include <crypto/internal/hash.h> +#include <crypto/internal/poly1305.h> #include <crypto/internal/simd.h> -#include <crypto/poly1305.h> #include <linux/crypto.h> #include <linux/kernel.h> #include <linux/module.h>
crypto/Kconfig +3 -2
@@ -446,7 +446,7 @@ config CRYPTO_KEYWRAP config CRYPTO_NHPOLY1305 tristate select CRYPTO_HASH - select CRYPTO_POLY1305 + select CRYPTO_LIB_POLY1305_GENERIC config CRYPTO_NHPOLY1305_SSE2 tristate "NHPoly1305 hash function (x86_64 SSE2 implementation)"
@@ -467,7 +467,7 @@ config CRYPTO_NHPOLY1305_AVX2 config CRYPTO_ADIANTUM tristate "Adiantum support" select CRYPTO_CHACHA20 - select CRYPTO_POLY1305 + select CRYPTO_LIB_POLY1305_GENERIC select CRYPTO_NHPOLY1305 select CRYPTO_MANAGER help
@@ -703,6 +703,7 @@ config CRYPTO_GHASH config CRYPTO_POLY1305 tristate "Poly1305 authenticator algorithm" select CRYPTO_HASH + select CRYPTO_LIB_POLY1305_GENERIC help Poly1305 authenticator algorithm, RFC7539.
crypto/adiantum.c +3 -2
@@ -33,6 +33,7 @@#include <crypto/b128ops.h> #include <crypto/chacha.h> #include <crypto/internal/hash.h> +#include <crypto/internal/poly1305.h> #include <crypto/internal/skcipher.h> #include <crypto/nhpoly1305.h> #include <crypto/scatterwalk.h>
@@ -242,11 +243,11 @@ static void adiantum_hash_header(struct skcipher_request *req)BUILD_BUG_ON (sizeof (header) %POLY1305_BLOCK_SIZE !=0 );poly1305_core_blocks (&state, &tctx->header_hash_key, - &header,sizeof (header) /POLY1305_BLOCK_SIZE ); + &header,sizeof (header) /POLY1305_BLOCK_SIZE ,1 );BUILD_BUG_ON (TWEAK_SIZE %POLY1305_BLOCK_SIZE !=0 );poly1305_core_blocks (&state, &tctx->header_hash_key, req->iv, -TWEAK_SIZE /POLY1305_BLOCK_SIZE ); +TWEAK_SIZE /POLY1305_BLOCK_SIZE ,1 );poly1305_core_emit (&state, &rctx->header_hash); }
crypto/nhpoly1305.c +2 -1
@@ -33,6 +33,7 @@#include <asm/unaligned.h> #include <crypto/algapi.h> #include <crypto/internal/hash.h> +#include <crypto/internal/poly1305.h> #include <crypto/nhpoly1305.h> #include <linux/crypto.h> #include <linux/kernel.h>
@@ -78,7 +79,7 @@ static void process_nh_hash_value(struct nhpoly1305_state *state,BUILD_BUG_ON (NH_HASH_BYTES %POLY1305_BLOCK_SIZE !=0 );poly1305_core_blocks (&state->poly_state, &key->poly_key, state->nh_hash, -NH_HASH_BYTES /POLY1305_BLOCK_SIZE ); +NH_HASH_BYTES /POLY1305_BLOCK_SIZE ,1 ); }/*
crypto/poly1305_generic.c +8 -187
241 lines not on this page · raw message
include/crypto/internal/poly1305.h new file +67 -0
67 lines not on this page · raw message
include/crypto/poly1305.h +0 -23
@@ -38,27 +38,4 @@ struct poly1305_desc_ctx { bool sset; }; -/* -* Poly1305 core functions. These implement the ε-almost-∆-universal hash -* function underlying the Poly1305 MAC, i.e. they don't add an encrypted nonce -* ("s key") at the end. They also only support block-aligned inputs. -*/ -void poly1305_core_setkey (struct poly1305_key *key,const u8 *raw_key); -static inline void poly1305_core_init (struct poly1305_state *state) -{ -memset (state->h,0 ,sizeof (state->h)); -} -void poly1305_core_blocks (struct poly1305_state *state, -const struct poly1305_key *key, -const void *src,unsigned int nblocks); -void poly1305_core_emit (const struct poly1305_state *state,void *dst); - -/* Crypto API helper functions for the Poly1305 MAC */ -int crypto_poly1305_init (struct shash_desc *desc); -unsigned int crypto_poly1305_setdesckey (struct poly1305_desc_ctx *dctx, -const u8 *src,unsigned int srclen); -int crypto_poly1305_update (struct shash_desc *desc, -const u8 *src,unsigned int srclen); -int crypto_poly1305_final (struct shash_desc *desc,u8 *dst); -#endif
lib/crypto/Kconfig +3 -0
@@ -37,5 +37,8 @@ config CRYPTO_LIB_CHACHA config CRYPTO_LIB_DES tristate +config CRYPTO_LIB_POLY1305_GENERIC + tristate + config CRYPTO_LIB_SHA256 tristate
lib/crypto/Makefile +3 -0
@@ -13,5 +13,8 @@ libarc4-y := arc4.o obj-$(CONFIG_CRYPTO_LIB_DES) += libdes.o libdes-y := des.o +obj-$(CONFIG_CRYPTO_LIB_POLY1305_GENERIC) += libpoly1305.o +libpoly1305-y := poly1305.o + obj-$(CONFIG_CRYPTO_LIB_SHA256) += libsha256.o libsha256-y := sha256.o
lib/crypto/poly1305.c new file +158 -0
158 lines not on this page · raw message
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 14/34 crypto: x86/poly1305 - unify Poly1305 state struct with generic code
In reply to Ard BiesheuvelIn preparation of exposing a Poly1305 library interface directly from the accelerated x86 driver, align the state descriptor of the x86 code with the one used by the generic driver. This is needed to make the library interface unified between all implementations.
--- arch/x86/crypto/poly1305_glue.c | 88 +++++++------------- crypto/poly1305_generic.c | 6 +- include/crypto/internal/poly1305.h | 4 +- include/crypto/poly1305.h | 18 ++-- 4 files changed, 43 insertions(+), 73 deletions(-)
arch/x86/crypto/poly1305_glue.c +29 -59
@@ -14,40 +14,14 @@#include <linux/module.h> #include <asm/simd.h> -struct poly1305_simd_desc_ctx { -struct poly1305_desc_ctx base; -/* derived key u set? */ -bool uset; -#ifdef CONFIG_AS_AVX2 -/* derived keys r^3, r^4 set? */ -bool wset; -#endif -/* derived Poly1305 key r^2 */ -u32 u[5 ]; -/* ... silently appended r^3 and r^4 when using AVX2 */ -}; - asmlinkage void poly1305_block_sse2(u32 *h, const u8 *src,const u32 *r,unsigned int blocks); asmlinkage void poly1305_2block_sse2(u32 *h, const u8 *src, const u32 *r,unsigned int blocks,const u32 *u); -#ifdef CONFIG_AS_AVX2 asmlinkage void poly1305_4block_avx2(u32 *h, const u8 *src, const u32 *r,unsigned int blocks,const u32 *u); -static bool poly1305_use_avx2; -#endif -static int poly1305_simd_init (struct shash_desc *desc) -{ -struct poly1305_simd_desc_ctx *sctx =shash_desc_ctx (desc); - - sctx->uset = false; -#ifdef CONFIG_AS_AVX2 - sctx->wset = false; -#endif - -return crypto_poly1305_init (desc); -} +static bool poly1305_use_avx2__ro_after_init ;static void poly1305_simd_mult (u32 *a,const u32 *b) {
@@ -63,53 +37,49 @@ static void poly1305_simd_mult(u32 *a, const u32 *b)static unsigned int poly1305_simd_blocks (struct poly1305_desc_ctx *dctx,const u8 *src,unsigned int srclen) { -struct poly1305_simd_desc_ctx *sctx;unsigned int blocks, datalen; -BUILD_BUG_ON (offsetof(struct poly1305_simd_desc_ctx , base)); - sctx = container_of(dctx, struct poly1305_simd_desc_ctx, base); -if (unlikely (!dctx->sset)) { datalen =crypto_poly1305_setdesckey (dctx, src, srclen); src += srclen - datalen; srclen = datalen; } -#ifdef CONFIG_AS_AVX2 -if (poly1305_use_avx2 && srclen >=POLY1305_BLOCK_SIZE *4 ) { -if (unlikely (!sctx->wset)) { -if (!sctx->uset) { -memcpy (sctx->u, dctx->r.r,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u, dctx->r.r); - sctx->uset = true; +if (IS_ENABLED (CONFIG_AS_AVX2 ) && + poly1305_use_avx2 && + srclen >=POLY1305_BLOCK_SIZE *4 ) { +if (unlikely (dctx->rset <4 )) { +if (dctx->rset <2 ) { + dctx->r[1 ] = dctx->r[0 ]; +poly1305_simd_mult (dctx->r[1 ].r, dctx->r[0 ].r); } -memcpy (sctx->u +5 , sctx->u,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u +5 , dctx->r.r); -memcpy (sctx->u +10 , sctx->u +5 ,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u +10 , dctx->r.r); - sctx->wset = true; + dctx->r[2 ] = dctx->r[1 ]; +poly1305_simd_mult (dctx->r[2 ].r, dctx->r[0 ].r); + dctx->r[3 ] = dctx->r[2 ]; +poly1305_simd_mult (dctx->r[3 ].r, dctx->r[0 ].r); + dctx->rset =4 ; } blocks = srclen / (POLY1305_BLOCK_SIZE *4 ); -poly1305_4block_avx2 (dctx->h.h, src, dctx->r.r, blocks, - sctx->u); +poly1305_4block_avx2 (dctx->h.h, src, dctx->r[0 ].r, blocks, + dctx->r[1 ].r); src +=POLY1305_BLOCK_SIZE *4 * blocks; srclen -=POLY1305_BLOCK_SIZE *4 * blocks; } -#endif +if (likely (srclen >=POLY1305_BLOCK_SIZE *2 )) { -if (unlikely (!sctx->uset)) { -memcpy (sctx->u, dctx->r.r,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u, dctx->r.r); - sctx->uset = true; +if (unlikely (dctx->rset <2 )) { + dctx->r[1 ] = dctx->r[0 ]; +poly1305_simd_mult (dctx->r[1 ].r, dctx->r[0 ].r); + dctx->rset =2 ; } blocks = srclen / (POLY1305_BLOCK_SIZE *2 ); -poly1305_2block_sse2 (dctx->h.h, src, dctx->r.r, blocks, - sctx->u); +poly1305_2block_sse2 (dctx->h.h, src, dctx->r[0 ].r, + blocks, dctx->r[1 ].r); src +=POLY1305_BLOCK_SIZE *2 * blocks; srclen -=POLY1305_BLOCK_SIZE *2 * blocks; }if (srclen >=POLY1305_BLOCK_SIZE ) { -poly1305_block_sse2 (dctx->h.h, src, dctx->r.r,1 ); +poly1305_block_sse2 (dctx->h.h, src, dctx->r[0 ].r,1 ); srclen -=POLY1305_BLOCK_SIZE ; }return srclen;
@@ -159,10 +129,10 @@ static int poly1305_simd_update(struct shash_desc *desc,static struct shash_alg alg = { .digestsize =POLY1305_DIGEST_SIZE , - .init = poly1305_simd_init, + .init = crypto_poly1305_init, .update = poly1305_simd_update, .final = crypto_poly1305_final, - .descsize =sizeof (struct poly1305_simd_desc_ctx ), + .descsize =sizeof (struct poly1305_desc_ctx ), .base = { .cra_name ="poly1305" , .cra_driver_name ="poly1305-simd" ,
@@ -177,14 +147,14 @@ static int __init poly1305_simd_mod_init(void)if (!boot_cpu_has (X86_FEATURE_XMM2 ))return -ENODEV ; -#ifdef CONFIG_AS_AVX2 - poly1305_use_avx2 =boot_cpu_has (X86_FEATURE_AVX ) && + poly1305_use_avx2 =IS_ENABLED (CONFIG_AS_AVX2 ) && +boot_cpu_has (X86_FEATURE_AVX ) &&boot_cpu_has (X86_FEATURE_AVX2 ) &&cpu_has_xfeatures (XFEATURE_MASK_SSE |XFEATURE_MASK_YMM ,NULL ); - alg.descsize =sizeof (struct poly1305_simd_desc_ctx ); + alg.descsize =sizeof (struct poly1305_desc_ctx ) +5 *sizeof (u32);if (poly1305_use_avx2) alg.descsize +=10 *sizeof (u32); -#endif +return crypto_register_shash (&alg); }
crypto/poly1305_generic.c +3 -3
@@ -25,7 +25,7 @@ int crypto_poly1305_init(struct shash_desc *desc)poly1305_core_init (&dctx->h); dctx->buflen =0 ; - dctx->rset = false; + dctx->rset =0 ; dctx->sset = false;return 0 ;
@@ -43,7 +43,7 @@ static void poly1305_blocks(struct poly1305_desc_ctx *dctx, const u8 *src, srclen = datalen; } -poly1305_core_blocks (&dctx->h, &dctx->r, src, +poly1305_core_blocks (&dctx->h, dctx->r, src, srclen /POLY1305_BLOCK_SIZE ,1 ); }
@@ -95,7 +95,7 @@ int crypto_poly1305_final(struct shash_desc *desc, u8 *dst) dctx->buf[dctx->buflen++] =1 ;memset (dctx->buf + dctx->buflen,0 ,POLY1305_BLOCK_SIZE - dctx->buflen); -poly1305_core_blocks (&dctx->h, &dctx->r, dctx->buf,1 ,0 ); +poly1305_core_blocks (&dctx->h, dctx->r, dctx->buf,1 ,0 ); }poly1305_core_emit (&dctx->h, digest);
include/crypto/internal/poly1305.h +2 -2
@@ -46,10 +46,10 @@ unsigned int crypto_poly1305_setdesckey(struct poly1305_desc_ctx *dctx, {if (!dctx->sset) {if (!dctx->rset && srclen >=POLY1305_BLOCK_SIZE ) { -poly1305_core_setkey (&dctx->r, src); +poly1305_core_setkey (dctx->r, src); src +=POLY1305_BLOCK_SIZE ; srclen -=POLY1305_BLOCK_SIZE ; - dctx->rset = true; + dctx->rset =1 ; }if (srclen >=POLY1305_BLOCK_SIZE ) { dctx->s[0 ] =get_unaligned_le32 (src +0 );
include/crypto/poly1305.h +9 -9
@@ -22,20 +22,20 @@ struct poly1305_state { };struct poly1305_desc_ctx { -/* key */ -struct poly1305_key r; -/* finalize key */ -u32 s[4 ]; -/* accumulator */ -struct poly1305_state h;/* partial buffer */ u8 buf[POLY1305_BLOCK_SIZE ];/* bytes used in partial buffer */ unsigned int buflen; -/* r key has been set */ -bool rset; -/* s key has been set */ +/* how many keys have been set in r[] */ +unsigned short rset; +/* whether s[] has been set */ bool sset; +/* finalize key */ +u32 s[4 ]; +/* accumulator */ +struct poly1305_state h; +/* key */ +struct poly1305_key r[1 ]; };#endif
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 15/34 crypto: poly1305 - expose init/update/final library interface
In reply to Ard BiesheuvelExpose the existing generic Poly1305 code via a init/update/final library interface so that callers are not required to go through the crypto API's shash abstraction to access it. At the same time, make some preparations so that the library implementation can be superseded by an accelerated arch-specific version in the future.
--- crypto/poly1305_generic.c | 22 +----- include/crypto/poly1305.h | 38 +++++++++- lib/crypto/Kconfig | 26 +++++++ lib/crypto/poly1305.c | 74 ++++++++++++++++++++ 4 files changed, 138 insertions(+), 22 deletions(-)
crypto/poly1305_generic.c +1 -21
@@ -85,31 +85,11 @@ EXPORT_SYMBOL_GPL(crypto_poly1305_update);int crypto_poly1305_final (struct shash_desc *desc,u8 *dst) {struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc); -__le32 digest[4 ]; -u64 f =0 ;if (unlikely (!dctx->sset))return -ENOKEY ; -if (unlikely (dctx->buflen)) { - dctx->buf[dctx->buflen++] =1 ; -memset (dctx->buf + dctx->buflen,0 , -POLY1305_BLOCK_SIZE - dctx->buflen); -poly1305_core_blocks (&dctx->h, dctx->r, dctx->buf,1 ,0 ); - } - -poly1305_core_emit (&dctx->h, digest); - -/* mac = (h + s) % (2^128) */ - f = (f >>32 ) +le32_to_cpu (digest[0 ]) + dctx->s[0 ]; -put_unaligned_le32 (f, dst +0 ); - f = (f >>32 ) +le32_to_cpu (digest[1 ]) + dctx->s[1 ]; -put_unaligned_le32 (f, dst +4 ); - f = (f >>32 ) +le32_to_cpu (digest[2 ]) + dctx->s[2 ]; -put_unaligned_le32 (f, dst +8 ); - f = (f >>32 ) +le32_to_cpu (digest[3 ]) + dctx->s[3 ]; -put_unaligned_le32 (f, dst +12 ); - +poly1305_final_generic (dctx, dst);return 0 ; }EXPORT_SYMBOL_GPL (crypto_poly1305_final);
include/crypto/poly1305.h +37 -1
@@ -35,7 +35,43 @@ struct poly1305_desc_ctx {/* accumulator */ struct poly1305_state h;/* key */ - struct poly1305_key r[1]; +struct poly1305_key r[CONFIG_CRYPTO_LIB_POLY1305_RSIZE ]; }; +void poly1305_init_arch (struct poly1305_desc_ctx *desc,const u8 *key); +void poly1305_init_generic (struct poly1305_desc_ctx *desc,const u8 *key); + +static inline void poly1305_init (struct poly1305_desc_ctx *desc,const u8 *key) +{ +if (IS_ENABLED (CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305 )) +poly1305_init_arch (desc, key); +else +poly1305_init_generic (desc, key); +} + +void poly1305_update_arch (struct poly1305_desc_ctx *desc,const u8 *src, +unsigned int nbytes); +void poly1305_update_generic (struct poly1305_desc_ctx *desc,const u8 *src, +unsigned int nbytes); + +static inline void poly1305_update (struct poly1305_desc_ctx *desc, +const u8 *src,unsigned int nbytes) +{ +if (IS_ENABLED (CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305 )) +poly1305_update_arch (desc, src, nbytes); +else +poly1305_update_generic (desc, src, nbytes); +} + +void poly1305_final_arch (struct poly1305_desc_ctx *desc,u8 *digest); +void poly1305_final_generic (struct poly1305_desc_ctx *desc,u8 *digest); + +static inline void poly1305_final (struct poly1305_desc_ctx *desc,u8 *digest) +{ +if (IS_ENABLED (CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305 )) +poly1305_final_arch (desc, digest); +else +poly1305_final_generic (desc, digest); +} +#endif
lib/crypto/Kconfig +26 -0
@@ -37,8 +37,34 @@ config CRYPTO_LIB_CHACHA config CRYPTO_LIB_DES tristate +config CRYPTO_LIB_POLY1305_RSIZE + int + default 1 + +config CRYPTO_ARCH_HAVE_LIB_POLY1305 + tristate + help + Declares whether the architecture provides an arch-specific + accelerated implementation of the Poly1305 library interface, + either builtin or as a module. + config CRYPTO_LIB_POLY1305_GENERIC tristate + help + This symbol can be depended upon by arch implementations of the + Poly1305 library interface that require the generic code as a + fallback, e.g., for SIMD implementations. If no arch specific + implementation is enabled, this implementation serves the users + of CRYPTO_LIB_POLY1305. + +config CRYPTO_LIB_POLY1305 + tristate "Poly1305 library interface" + depends on CRYPTO_ARCH_HAVE_LIB_POLY1305 || !CRYPTO_ARCH_HAVE_LIB_POLY1305 + select CRYPTO_LIB_POLY1305_GENERIC if CRYPTO_ARCH_HAVE_LIB_POLY1305=n + help + Enable the Poly1305 library interface. This interface may be fulfilled + by either the generic implementation or an arch-specific one, if one + is available and enabled. config CRYPTO_LIB_SHA256 tristate
lib/crypto/poly1305.c +74 -0
79 lines not on this page · raw message
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 16/34 crypto: x86/poly1305 - depend on generic library not generic shash
In reply to Ard BiesheuvelRemove the dependency on the generic Poly1305 driver. Instead, depend on the generic library so that we only reuse code without pulling in the generic skcipher implementation as well.
While at it, remove the logic that prefers the non-SIMD path for short inputs - this is no longer necessary after recent FPU handling changes on x86.
Since this removes the last remaining user of the routines exported by the generic shash driver, unexport them and make them static.
--- arch/x86/crypto/poly1305_glue.c | 66 ++++++++++++++++---- crypto/Kconfig | 2 +- crypto/poly1305_generic.c | 11 ++-- include/crypto/internal/poly1305.h | 9 --- 4 files changed, 60 insertions(+), 28 deletions(-)
arch/x86/crypto/poly1305_glue.c +55 -11
@@ -34,6 +34,24 @@ static void poly1305_simd_mult(u32 *a, const u32 *b)poly1305_block_sse2 (a, m, b,1 ); } +static unsigned int poly1305_scalar_blocks (struct poly1305_desc_ctx *dctx, +const u8 *src,unsigned int srclen) +{ +unsigned int datalen; + +if (unlikely (!dctx->sset)) { + datalen =crypto_poly1305_setdesckey (dctx, src, srclen); + src += srclen - datalen; + srclen = datalen; + } +if (srclen >=POLY1305_BLOCK_SIZE ) { +poly1305_core_blocks (&dctx->h, dctx->r, src, + srclen /POLY1305_BLOCK_SIZE ,1 ); + srclen %=POLY1305_BLOCK_SIZE ; + } +return srclen; +} +static unsigned int poly1305_simd_blocks (struct poly1305_desc_ctx *dctx,const u8 *src,unsigned int srclen) {
@@ -91,12 +109,6 @@ static int poly1305_simd_update(struct shash_desc *desc,struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc);unsigned int bytes; -/* kernel_fpu_begin/end is costly, use fallback for small updates */ -if (srclen <=288 || !crypto_simd_usable ()) -return crypto_poly1305_update (desc, src, srclen); - -kernel_fpu_begin (); -if (unlikely (dctx->buflen)) { bytes =min (srclen,POLY1305_BLOCK_SIZE - dctx->buflen);memcpy (dctx->buf + dctx->buflen, src, bytes);
@@ -105,25 +117,57 @@ static int poly1305_simd_update(struct shash_desc *desc, dctx->buflen += bytes;if (dctx->buflen ==POLY1305_BLOCK_SIZE ) { -poly1305_simd_blocks (dctx, dctx->buf, -POLY1305_BLOCK_SIZE ); +if (likely (crypto_simd_usable ())) { +kernel_fpu_begin (); +poly1305_simd_blocks (dctx, dctx->buf, +POLY1305_BLOCK_SIZE ); +kernel_fpu_end (); + }else { +poly1305_scalar_blocks (dctx, dctx->buf, +POLY1305_BLOCK_SIZE ); + } dctx->buflen =0 ; } }if (likely (srclen >=POLY1305_BLOCK_SIZE )) { - bytes =poly1305_simd_blocks (dctx, src, srclen); +if (likely (crypto_simd_usable ())) { +kernel_fpu_begin (); + bytes =poly1305_simd_blocks (dctx, src, srclen); +kernel_fpu_end (); + }else { + bytes =poly1305_scalar_blocks (dctx, src, srclen); + } src += srclen - bytes; srclen = bytes; } -kernel_fpu_end (); -if (unlikely (srclen)) { dctx->buflen = srclen;memcpy (dctx->buf, src, srclen); } +} + +static int crypto_poly1305_init (struct shash_desc *desc) +{ +struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc); + +poly1305_core_init (&dctx->h); + dctx->buflen =0 ; + dctx->rset =0 ; + dctx->sset = false; + +return 0 ; +} + +static int crypto_poly1305_final (struct shash_desc *desc,u8 *dst) +{ +struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc); + +if (unlikely (!dctx->sset)) +return -ENOKEY ; +poly1305_final_generic (dctx, dst);return 0 ; }
crypto/Kconfig +1 -1
@@ -714,7 +714,7 @@ config CRYPTO_POLY1305 config CRYPTO_POLY1305_X86_64 tristate "Poly1305 authenticator algorithm (x86_64/SSE2/AVX2)" depends on X86 && 64BIT - select CRYPTO_POLY1305 + select CRYPTO_LIB_POLY1305_GENERIC help Poly1305 authenticator algorithm, RFC7539.
crypto/poly1305_generic.c +4 -7
@@ -19,7 +19,7 @@#include <linux/module.h> #include <asm/unaligned.h> -int crypto_poly1305_init (struct shash_desc *desc) +static int crypto_poly1305_init (struct shash_desc *desc) {struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc);
@@ -30,7 +30,6 @@ int crypto_poly1305_init(struct shash_desc *desc)return 0 ; } -EXPORT_SYMBOL_GPL (crypto_poly1305_init);static void poly1305_blocks (struct poly1305_desc_ctx *dctx,const u8 *src,unsigned int srclen)
@@ -47,8 +46,8 @@ static void poly1305_blocks(struct poly1305_desc_ctx *dctx, const u8 *src, srclen / POLY1305_BLOCK_SIZE, 1); } -int crypto_poly1305_update (struct shash_desc *desc, -const u8 *src,unsigned int srclen) +static int crypto_poly1305_update (struct shash_desc *desc, +const u8 *src,unsigned int srclen) {struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc);unsigned int bytes;
@@ -80,9 +79,8 @@ int crypto_poly1305_update(struct shash_desc *desc,return 0 ; } -EXPORT_SYMBOL_GPL (crypto_poly1305_update); -int crypto_poly1305_final (struct shash_desc *desc,u8 *dst) +static int crypto_poly1305_final (struct shash_desc *desc,u8 *dst) {struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc);
@@ -92,7 +90,6 @@ int crypto_poly1305_final(struct shash_desc *desc, u8 *dst)poly1305_final_generic (dctx, dst);return 0 ; } -EXPORT_SYMBOL_GPL (crypto_poly1305_final);static struct shash_alg poly1305_alg = { .digestsize =POLY1305_DIGEST_SIZE ,
include/crypto/internal/poly1305.h +0 -9
@@ -10,8 +10,6 @@#include <linux/types.h> #include <crypto/poly1305.h> -struct shash_desc ; -/* * Poly1305 core functions. These implement the ε-almost-∆-universal hash * function underlying the Poly1305 MAC, i.e. they don't add an encrypted nonce
@@ -28,13 +26,6 @@ void poly1305_core_blocks(struct poly1305_state *state, unsigned int nblocks, u32 hibit);void poly1305_core_emit (const struct poly1305_state *state,void *dst); -/* Crypto API helper functions for the Poly1305 MAC */ -int crypto_poly1305_init (struct shash_desc *desc); - -int crypto_poly1305_update (struct shash_desc *desc, -const u8 *src,unsigned int srclen); -int crypto_poly1305_final (struct shash_desc *desc,u8 *dst); -/* * Poly1305 requires a unique key for each tag, which implies that we can't set * it on the tfm that gets accessed by multiple users simultaneously. Instead we
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 17/34 crypto: x86/poly1305 - expose existing driver as poly1305 library
In reply to Ard BiesheuvelImplement the arch init/update/final Poly1305 library routines in the accelerated SIMD driver for x86 so they are accessible to users of the Poly1305 library interface as well.
--- arch/x86/crypto/poly1305_glue.c | 57 ++++++++++++++------ crypto/Kconfig | 1 + lib/crypto/Kconfig | 1 + 3 files changed, 43 insertions(+), 16 deletions(-)
arch/x86/crypto/poly1305_glue.c +41 -16
@@ -10,6 +10,7 @@#include <crypto/internal/poly1305.h> #include <crypto/internal/simd.h> #include <linux/crypto.h> +#include <linux/jump_label.h> #include <linux/kernel.h> #include <linux/module.h> #include <asm/simd.h>
@@ -21,7 +22,8 @@ asmlinkage void poly1305_2block_sse2(u32 *h, const u8 *src, const u32 *r, asmlinkage void poly1305_4block_avx2(u32 *h, const u8 *src, const u32 *r,unsigned int blocks,const u32 *u); -static bool poly1305_use_avx2__ro_after_init ; +static __ro_after_init DEFINE_STATIC_KEY_FALSE (poly1305_use_simd); +static __ro_after_init DEFINE_STATIC_KEY_FALSE (poly1305_use_avx2);static void poly1305_simd_mult (u32 *a,const u32 *b) {
@@ -64,7 +66,7 @@ static unsigned int poly1305_simd_blocks(struct poly1305_desc_ctx *dctx, }if (IS_ENABLED (CONFIG_AS_AVX2 ) && - poly1305_use_avx2 && +static_branch_likely (&poly1305_use_avx2) && srclen >=POLY1305_BLOCK_SIZE *4 ) {if (unlikely (dctx->rset <4 )) {if (dctx->rset <2 ) {
@@ -103,10 +105,15 @@ static unsigned int poly1305_simd_blocks(struct poly1305_desc_ctx *dctx,return srclen; } -static int poly1305_simd_update (struct shash_desc *desc, -const u8 *src,unsigned int srclen) +void poly1305_init_arch (struct poly1305_desc_ctx *desc,const u8 *key) +{ +poly1305_init_generic (desc, key); +} +EXPORT_SYMBOL (poly1305_init_arch); + +void poly1305_update_arch (struct poly1305_desc_ctx *dctx,const u8 *src, +unsigned int srclen) { -struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc);unsigned int bytes;if (unlikely (dctx->buflen)) {
@@ -117,7 +124,8 @@ static int poly1305_simd_update(struct shash_desc *desc, dctx->buflen += bytes;if (dctx->buflen ==POLY1305_BLOCK_SIZE ) { -if (likely (crypto_simd_usable ())) { +if (static_branch_likely (&poly1305_use_simd) && +likely (crypto_simd_usable ())) {kernel_fpu_begin ();poly1305_simd_blocks (dctx, dctx->buf,POLY1305_BLOCK_SIZE );
@@ -131,7 +139,8 @@ static int poly1305_simd_update(struct shash_desc *desc, }if (likely (srclen >=POLY1305_BLOCK_SIZE )) { -if (likely (crypto_simd_usable ())) { +if (static_branch_likely (&poly1305_use_simd) && +likely (crypto_simd_usable ())) {kernel_fpu_begin (); bytes =poly1305_simd_blocks (dctx, src, srclen);kernel_fpu_end ();
@@ -147,6 +156,13 @@ static int poly1305_simd_update(struct shash_desc *desc,memcpy (dctx->buf, src, srclen); } } +EXPORT_SYMBOL (poly1305_update_arch); + +void poly1305_final_arch (struct poly1305_desc_ctx *desc,u8 *digest) +{ +poly1305_final_generic (desc, digest); +} +EXPORT_SYMBOL (poly1305_final_arch);static int crypto_poly1305_init (struct shash_desc *desc) {
@@ -171,6 +187,15 @@ static int crypto_poly1305_final(struct shash_desc *desc, u8 *dst)return 0 ; } +static int poly1305_simd_update (struct shash_desc *desc, +const u8 *src,unsigned int srclen) +{ +struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc); + +poly1305_update_arch (dctx, src, srclen); +return 0 ; +} +static struct shash_alg alg = { .digestsize =POLY1305_DIGEST_SIZE , .init = crypto_poly1305_init,
@@ -189,15 +214,15 @@ static struct shash_alg alg = {static int __init poly1305_simd_mod_init (void ) {if (!boot_cpu_has (X86_FEATURE_XMM2 )) -return -ENODEV ; - - poly1305_use_avx2 =IS_ENABLED (CONFIG_AS_AVX2 ) && -boot_cpu_has (X86_FEATURE_AVX ) && -boot_cpu_has (X86_FEATURE_AVX2 ) && -cpu_has_xfeatures (XFEATURE_MASK_SSE |XFEATURE_MASK_YMM ,NULL ); - alg.descsize =sizeof (struct poly1305_desc_ctx ) +5 *sizeof (u32); -if (poly1305_use_avx2) - alg.descsize +=10 *sizeof (u32); +return 0 ; + +static_branch_enable (&poly1305_use_simd); + +if (IS_ENABLED (CONFIG_AS_AVX2 ) && +boot_cpu_has (X86_FEATURE_AVX ) && +boot_cpu_has (X86_FEATURE_AVX2 ) && +cpu_has_xfeatures (XFEATURE_MASK_SSE |XFEATURE_MASK_YMM ,NULL )) +static_branch_enable (&poly1305_use_avx2);return crypto_register_shash (&alg); }
crypto/Kconfig +1 -0
@@ -715,6 +715,7 @@ config CRYPTO_POLY1305_X86_64 tristate "Poly1305 authenticator algorithm (x86_64/SSE2/AVX2)" depends on X86 && 64BIT select CRYPTO_LIB_POLY1305_GENERIC + select CRYPTO_ARCH_HAVE_LIB_POLY1305 help Poly1305 authenticator algorithm, RFC7539.
lib/crypto/Kconfig +1 -0
@@ -39,6 +39,7 @@ config CRYPTO_LIB_DES config CRYPTO_LIB_POLY1305_RSIZE int + default 4 if X86_64 default 1 config CRYPTO_ARCH_HAVE_LIB_POLY1305
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 18/34 crypto: arm64/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON implementation
In reply to Ard BiesheuvelThis is a straight import of the OpenSSL/CRYPTOGAMS Poly1305 implementation for NEON authored by Andy Polyakov, and contributed by him to the OpenSSL project. The file 'poly1305-armv8.pl' is taken straight from this upstream GitHub repository [0] at commit ec55a08dc0244ce570c4fc7cade330c60798952f, and already contains all the changes required to build it as part of a Linux kernel module.
[0] https://github.com/dot-asm/cryptogams
--- arch/arm64/crypto/Kconfig | 6 + arch/arm64/crypto/Makefile | 10 +- arch/arm64/crypto/poly1305-armv8.pl | 913 ++++++++++++++++++++ arch/arm64/crypto/poly1305-core.S_shipped | 835 ++++++++++++++++++ arch/arm64/crypto/poly1305-glue.c | 237 +++++ lib/crypto/Kconfig | 1 + 6 files changed, 2001 insertions(+), 1 deletion(-)
arch/arm64/crypto/Kconfig +6 -0
@@ -106,6 +106,12 @@ config CRYPTO_CHACHA20_NEON select CRYPTO_LIB_CHACHA_GENERIC select CRYPTO_ARCH_HAVE_LIB_CHACHA +config CRYPTO_POLY1305_NEON + tristate "Poly1305 hash function using scalar or NEON instructions" + depends on KERNEL_MODE_NEON + select CRYPTO_HASH + select CRYPTO_ARCH_HAVE_LIB_POLY1305 + config CRYPTO_NHPOLY1305_NEON tristate "NHPoly1305 hash function using NEON instructions (for Adiantum)" depends on KERNEL_MODE_NEON
arch/arm64/crypto/Makefile +9 -1
@@ -50,6 +50,10 @@ sha512-arm64-y := sha512-glue.o sha512-core.o obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha-neon.o chacha-neon-y := chacha-neon-core.o chacha-neon-glue.o +obj-$(CONFIG_CRYPTO_POLY1305_NEON) += poly1305-neon.o +poly1305-neon-y := poly1305-core.o poly1305-glue.o +AFLAGS_poly1305-core.o += -Dpoly1305_init=poly1305_init_arm64 + obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) += nhpoly1305-neon.o nhpoly1305-neon-y := nh-neon-core.o nhpoly1305-neon-glue.o
@@ -68,11 +72,15 @@ ifdef REGENERATE_ARM64_CRYPTO quiet_cmd_perlasm = PERLASM $@ cmd_perlasm = $(PERL) $(<) void $(@) +$(src)/poly1305-core.S_shipped: $(src)/poly1305-armv8.pl + $(call cmd,perlasm) + $(src)/sha256-core.S_shipped: $(src)/sha512-armv8.pl $(call cmd,perlasm) $(src)/sha512-core.S_shipped: $(src)/sha512-armv8.pl $(call cmd,perlasm) + endif -clean-files += sha256-core.S sha512-core.S +clean-files += poly1305-core.S sha256-core.S sha512-core.S
arch/arm64/crypto/poly1305-armv8.pl new file +913 -0
913 lines not on this page · raw message
arch/arm64/crypto/poly1305-core.S_shipped new file +835 -0
835 lines not on this page · raw message
arch/arm64/crypto/poly1305-glue.c new file +237 -0
237 lines not on this page · raw message
lib/crypto/Kconfig +1 -0
@@ -40,6 +40,7 @@ config CRYPTO_LIB_DES config CRYPTO_LIB_POLY1305_RSIZE int default 4 if X86_64 + default 9 if ARM64 default 1 config CRYPTO_ARCH_HAVE_LIB_POLY1305
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 19/34 crypto: arm/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON implementation
In reply to Ard BiesheuvelThis is a straight import of the OpenSSL/CRYPTOGAMS Poly1305 implementation for NEON authored by Andy Polyakov, and contributed by him to the OpenSSL project. The file 'poly1305-armv4.pl' is taken straight from this upstream GitHub repository [0] at commit ec55a08dc0244ce570c4fc7cade330c60798952f, and already contains all the changes required to build it as part of a Linux kernel module.
[0] https://github.com/dot-asm/cryptogams
--- arch/arm/crypto/Kconfig | 5 + arch/arm/crypto/Makefile | 12 +- arch/arm/crypto/poly1305-armv4.pl | 1236 ++++++++++++++++++++ arch/arm/crypto/poly1305-core.S_shipped | 1158 ++++++++++++++++++ arch/arm/crypto/poly1305-glue.c | 276 +++++ lib/crypto/Kconfig | 2 +- 6 files changed, 2687 insertions(+), 2 deletions(-)
arch/arm/crypto/Kconfig +5 -0
@@ -132,6 +132,11 @@ config CRYPTO_CHACHA20_NEON select CRYPTO_SKCIPHER select CRYPTO_ARCH_HAVE_LIB_CHACHA +config CRYPTO_POLY1305_ARM + tristate "Accelerated scalar and SIMD Poly1305 hash implementations" + select CRYPTO_HASH + select CRYPTO_ARCH_HAVE_LIB_POLY1305 + config CRYPTO_NHPOLY1305_NEON tristate "NEON accelerated NHPoly1305 hash function (for Adiantum)" depends on KERNEL_MODE_NEON
arch/arm/crypto/Makefile +11 -1
@@ -10,6 +10,7 @@ obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha-neon.o +obj-$(CONFIG_CRYPTO_POLY1305_ARM) += poly1305-arm.o obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) += nhpoly1305-neon.o obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o
@@ -35,12 +36,16 @@ crct10dif-arm-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o chacha-neon-y := chacha-scalar-core.o chacha-glue.o chacha-neon-$(CONFIG_KERNEL_MODE_NEON) += chacha-neon-core.o +poly1305-arm-y := poly1305-core.o poly1305-glue.o nhpoly1305-neon-y := nh-neon-core.o nhpoly1305-neon-glue.o ifdef REGENERATE_ARM_CRYPTO quiet_cmd_perl = PERL $@ cmd_perl = $(PERL) $(<) > $(@) +$(src)/poly1305-core.S_shipped: $(src)/poly1305-armv4.pl + $(call cmd,perl) + $(src)/sha256-core.S_shipped: $(src)/sha256-armv4.pl $(call cmd,perl)
@@ -48,4 +53,9 @@ $(src)/sha512-core.S_shipped: $(src)/sha512-armv4.pl $(call cmd,perl) endif -clean-files += sha256-core.S sha512-core.S +clean-files += poly1305-core.S sha256-core.S sha512-core.S + +# massage the perlasm code a bit so we only get the NEON routine if we need it +poly1305-aflags-$(CONFIG_CPU_V7) := -U__LINUX_ARM_ARCH__ -D__LINUX_ARM_ARCH__=5 +poly1305-aflags-$(CONFIG_KERNEL_MODE_NEON) := -U__LINUX_ARM_ARCH__ -D__LINUX_ARM_ARCH__=7 +AFLAGS_poly1305-core.o += $(poly1305-aflags-y)
arch/arm/crypto/poly1305-armv4.pl new file +1236 -0
1,236 lines not on this page · raw message
arch/arm/crypto/poly1305-core.S_shipped new file +1158 -0
1,158 lines not on this page · raw message
arch/arm/crypto/poly1305-glue.c new file +276 -0
276 lines not on this page · raw message
lib/crypto/Kconfig +1 -1
@@ -40,7 +40,7 @@ config CRYPTO_LIB_DES config CRYPTO_LIB_POLY1305_RSIZE int default 4 if X86_64 - default 9 if ARM64 + default 9 if ARM || ARM64 default 1 config CRYPTO_ARCH_HAVE_LIB_POLY1305
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 20/34 crypto: mips/poly1305 - incorporate OpenSSL/CRYPTOGAMS optimized implementation
In reply to Ard BiesheuvelThis is a straight import of the OpenSSL/CRYPTOGAMS Poly1305 implementation for MIPS authored by Andy Polyakov, a prior 64-bit only version of which has been contributed by him to the OpenSSL project. The file 'poly1305-mips.pl' is taken straight from this upstream GitHub repository [0] at commit d22ade312a7af958ec955620b0d241cf42c37feb, and already contains all the changes required to build it as part of a Linux kernel module.
[0] https://github.com/dot-asm/cryptogams
--- arch/mips/crypto/Makefile | 14 + arch/mips/crypto/poly1305-glue.c | 203 ++++ arch/mips/crypto/poly1305-mips.pl | 1273 ++++++++++++++++++++ crypto/Kconfig | 5 + lib/crypto/Kconfig | 1 + 5 files changed, 1496 insertions(+)
arch/mips/crypto/Makefile +14 -0
@@ -8,3 +8,17 @@ obj-$(CONFIG_CRYPTO_CRC32_MIPS) += crc32-mips.o obj-$(CONFIG_CRYPTO_CHACHA_MIPS) += chacha-mips.o chacha-mips-y := chacha-core.o chacha-glue.o AFLAGS_chacha-core.o += -O2 # needed to fill branch delay slots + +obj-$(CONFIG_CRYPTO_POLY1305_MIPS) += poly1305-mips.o +poly1305-mips-y := poly1305-core.o poly1305-glue.o + +perlasm-flavour-$(CONFIG_CPU_MIPS32) := o32 +perlasm-flavour-$(CONFIG_CPU_MIPS64) := 64 + +quiet_cmd_perlasm = PERLASM $@ + cmd_perlasm = $(PERL) $(<) $(perlasm-flavour-y) $(@) + +$(obj)/poly1305-core.S: $(src)/poly1305-mips.pl FORCE + $(call if_changed,perlasm) + +targets += poly1305-core.S
arch/mips/crypto/poly1305-glue.c new file +203 -0
203 lines not on this page · raw message
arch/mips/crypto/poly1305-mips.pl new file +1273 -0
1,273 lines not on this page · raw message
crypto/Kconfig +5 -0
@@ -724,6 +724,11 @@ config CRYPTO_POLY1305_X86_64 in IETF protocols. This is the x86_64 assembler implementation using SIMD instructions. +config CRYPTO_POLY1305_MIPS + tristate "Poly1305 authenticator algorithm (MIPS optimized)" + depends on CPU_MIPS32 || (CPU_MIPS64 && 64BIT) + select CRYPTO_ARCH_HAVE_LIB_POLY1305 + config CRYPTO_MD4 tristate "MD4 digest algorithm" select CRYPTO_HASH
lib/crypto/Kconfig +1 -0
@@ -39,6 +39,7 @@ config CRYPTO_LIB_DES config CRYPTO_LIB_POLY1305_RSIZE int + default 2 if MIPS default 4 if X86_64 default 9 if ARM || ARM64 default 1
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 21/34 int128: move __uint128_t compiler test to Kconfig
In reply to Ard BiesheuvelIn order to use 128-bit integer arithmetic in C code, the architecture needs to have declared support for it by setting ARCH_SUPPORTS_INT128, and it requires a version of the toolchain that supports this at build time. This is why all existing tests for ARCH_SUPPORTS_INT128 also test whether __SIZEOF_INT128__ is defined, since this is only the case for compilers that can support 128-bit integers.
Let's fold this additional test into the Kconfig declaration of ARCH_SUPPORTS_INT128 so that we can also use the symbol in Makefiles, e.g., to decide whether a certain object needs to be included in the first place.
--- arch/arm64/Kconfig | 2 +- arch/riscv/Kconfig | 2 +- arch/x86/Kconfig | 2 +- crypto/ecc.c | 2 +- init/Kconfig | 4 ++++ lib/ubsan.c | 2 +- lib/ubsan.h | 2 +- 7 files changed, 10 insertions(+), 6 deletions(-)
arch/arm64/Kconfig +1 -1
@@ -67,7 +67,7 @@ config ARM64 select ARCH_USE_QUEUED_SPINLOCKS select ARCH_SUPPORTS_MEMORY_FAILURE select ARCH_SUPPORTS_ATOMIC_RMW - select ARCH_SUPPORTS_INT128 if GCC_VERSION >= 50000 || CC_IS_CLANG + select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 && (GCC_VERSION >= 50000 || CC_IS_CLANG) select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT
arch/riscv/Kconfig +1 -1
@@ -164,7 +164,7 @@ config ARCH_RV32I config ARCH_RV64I bool "RV64I" select 64BIT - select ARCH_SUPPORTS_INT128 if GCC_VERSION >= 50000 + select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 && GCC_VERSION >= 50000 select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FTRACE_MCOUNT_RECORD
arch/x86/Kconfig +1 -1
@@ -24,7 +24,7 @@ config X86_64 depends on 64BIT # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE - select ARCH_SUPPORTS_INT128 + select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA
crypto/ecc.c +1 -1
@@ -336,7 +336,7 @@ static u64 vli_usub(u64 *result, const u64 *left, u64 right,static uint128_t mul_64_64 (u64 left,u64 right) {uint128_t result; -#if defined(CONFIG_ARCH_SUPPORTS_INT128 ) && defined(__SIZEOF_INT128__) +#if defined(CONFIG_ARCH_SUPPORTS_INT128 )unsigned __int128 m = (unsigned __int128 )left * right; result.m_low = m;
init/Kconfig +4 -0
@@ -785,6 +785,10 @@ config ARCH_SUPPORTS_NUMA_BALANCING config ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH bool +config CC_HAS_INT128 + def_bool y + depends on !$(cc-option,-D__SIZEOF_INT128__=0) + # # For architectures that know their GCC __int128 support is sound #
lib/ubsan.c +1 -1
@@ -119,7 +119,7 @@ static void val_to_string(char *str, size_t size, struct type_descriptor *type, {if (type_is_int (type)) {if (type_bit_width (type) ==128 ) { -#if defined(CONFIG_ARCH_SUPPORTS_INT128 ) && defined(__SIZEOF_INT128__) +#if defined(CONFIG_ARCH_SUPPORTS_INT128 )u_max val =get_unsigned_val (type, value); scnprintf(str, size, "0x%08x%08x%08x%08x",
lib/ubsan.h +1 -1
@@ -78,7 +78,7 @@ struct invalid_value_data {struct type_descriptor *type; }; -#if defined(CONFIG_ARCH_SUPPORTS_INT128 ) && defined(__SIZEOF_INT128__) +#if defined(CONFIG_ARCH_SUPPORTS_INT128 )typedef __int128 s_max ;typedef unsigned __int128 u_max ;#else
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 22/34 crypto: BLAKE2s - generic C library implementation and selftest
In reply to Ard BiesheuvelFrom: "Jason A. Donenfeld" <[email protected]>
The C implementation was originally based on Samuel Neves' public domain reference implementation but has since been heavily modified for the kernel. We're able to do compile-time optimizations by moving some scaffolding around the final function into the header file.
Information: https://blake2.net/
[ardb: - move from lib/zinc to lib/crypto
- remove simd handling
- rewrote selftest for better coverage
- use fixed digest length for blake2s_hmac() and rename to blake2s256_hmac() ]
--- include/crypto/blake2s.h | 106 ++++ include/crypto/internal/blake2s.h | 19 + lib/crypto/Kconfig | 25 + lib/crypto/Makefile | 10 + lib/crypto/blake2s-generic.c | 111 ++++ lib/crypto/blake2s-selftest.c | 622 ++++++++++++++++++++ lib/crypto/blake2s.c | 126 ++++ 7 files changed, 1019 insertions(+)
include/crypto/blake2s.h new file +106 -0
@@ -0,0 +1,106 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef BLAKE2S_H +#define BLAKE2S_H + +#include <linux/types.h> +#include <linux/kernel.h> +#include <linux/string.h> + +#include <asm/bug.h> + +enum blake2s_lengths { +BLAKE2S_BLOCK_SIZE =64 , +BLAKE2S_HASH_SIZE =32 , +BLAKE2S_KEY_SIZE =32 , + +BLAKE2S_128_HASH_SIZE =16 , +BLAKE2S_160_HASH_SIZE =20 , +BLAKE2S_224_HASH_SIZE =28 , +BLAKE2S_256_HASH_SIZE =32 , +}; + +struct blake2s_state { +u32 h[8 ]; +u32 t[2 ]; +u32 f[2 ]; +u8 buf[BLAKE2S_BLOCK_SIZE ]; +unsigned int buflen; +unsigned int outlen; +}; + +enum blake2s_iv { +BLAKE2S_IV0 =0x6A09E667UL , +BLAKE2S_IV1 =0xBB67AE85UL , +BLAKE2S_IV2 =0x3C6EF372UL , +BLAKE2S_IV3 =0xA54FF53AUL , +BLAKE2S_IV4 =0x510E527FUL , +BLAKE2S_IV5 =0x9B05688CUL , +BLAKE2S_IV6 =0x1F83D9ABUL , +BLAKE2S_IV7 =0x5BE0CD19UL , +}; + +void blake2s_update (struct blake2s_state *state,const u8 *in,size_t inlen); +void blake2s_final (struct blake2s_state *state,u8 *out); + +static inline void blake2s_init_param (struct blake2s_state *state, +const u32 param) +{ + *state = (struct blake2s_state ){{ +BLAKE2S_IV0 ^ param, +BLAKE2S_IV1 , +BLAKE2S_IV2 , +BLAKE2S_IV3 , +BLAKE2S_IV4 , +BLAKE2S_IV5 , +BLAKE2S_IV6 , +BLAKE2S_IV7 , + }}; +} + +static inline void blake2s_init (struct blake2s_state *state, +const size_t outlen) +{ +blake2s_init_param (state,0x01010000 | outlen); + state->outlen = outlen; +} + +static inline void blake2s_init_key (struct blake2s_state *state, +const size_t outlen,const void *key, +const size_t keylen) +{ +WARN_ON (IS_ENABLED (DEBUG ) && (!outlen || outlen >BLAKE2S_HASH_SIZE || + !key || !keylen || keylen >BLAKE2S_KEY_SIZE )); + +blake2s_init_param (state,0x01010000 | keylen <<8 | outlen); +memcpy (state->buf, key, keylen); + state->buflen =BLAKE2S_BLOCK_SIZE ; + state->outlen = outlen; +} + +static inline void blake2s (u8 *out,const u8 *in,const u8 *key, +const size_t outlen,const size_t inlen, +const size_t keylen) +{ +struct blake2s_state state; + +WARN_ON (IS_ENABLED (DEBUG ) && ((!in && inlen >0 ) || !out || !outlen || + outlen >BLAKE2S_HASH_SIZE || keylen >BLAKE2S_KEY_SIZE || + (!key && keylen))); + +if (keylen) +blake2s_init_key (&state, outlen, key, keylen); +else +blake2s_init (&state, outlen); + +blake2s_update (&state, in, inlen); +blake2s_final (&state, out); +} + +void blake2s256_hmac (u8 *out,const u8 *in,const u8 *key,const size_t inlen, +const size_t keylen); + +#endif /* BLAKE2S_H */
include/crypto/internal/blake2s.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ + +#ifndef BLAKE2S_INTERNAL_H +#define BLAKE2S_INTERNAL_H + +#include <crypto/blake2s.h> + +void blake2s_compress_generic (struct blake2s_state *state,const u8 *block, +size_t nblocks,const u32 inc); + +void blake2s_compress_arch (struct blake2s_state *state,const u8 *block, +size_t nblocks,const u32 inc); + +static inline void blake2s_set_lastblock (struct blake2s_state *state) +{ + state->f[0 ] =-1 ; +} + +#endif /* BLAKE2S_INTERNAL_H */
lib/crypto/Kconfig +25 -0
@@ -8,6 +8,31 @@ config CRYPTO_LIB_AES config CRYPTO_LIB_ARC4 tristate +config CRYPTO_ARCH_HAVE_LIB_BLAKE2S + tristate + help + Declares whether the architecture provides an arch-specific + accelerated implementation of the Blake2s library interface, + either builtin or as a module. + +config CRYPTO_LIB_BLAKE2S_GENERIC + tristate + help + This symbol can be depended upon by arch implementations of the + Blake2s library interface that require the generic code as a + fallback, e.g., for SIMD implementations. If no arch specific + implementation is enabled, this implementation serves the users + of CRYPTO_LIB_BLAKE2S. + +config CRYPTO_LIB_BLAKE2S + tristate "BLAKE2s hash function library" + depends on CRYPTO_ARCH_HAVE_LIB_BLAKE2S || !CRYPTO_ARCH_HAVE_LIB_BLAKE2S + select CRYPTO_LIB_BLAKE2S_GENERIC if CRYPTO_ARCH_HAVE_LIB_BLAKE2S=n + help + Enable the Blake2s library interface. This interface may be fulfilled + by either the generic implementation or an arch-specific one, if one + is available and enabled. + config CRYPTO_ARCH_HAVE_LIB_CHACHA tristate help
lib/crypto/Makefile +10 -0
@@ -10,6 +10,12 @@ libaes-y := aes.o obj-$(CONFIG_CRYPTO_LIB_ARC4) += libarc4.o libarc4-y := arc4.o +obj-$(CONFIG_CRYPTO_LIB_BLAKE2S_GENERIC) += libblake2s-generic.o +libblake2s-generic-y += blake2s-generic.o + +obj-$(CONFIG_CRYPTO_LIB_BLAKE2S) += libblake2s.o +libblake2s-y += blake2s.o + obj-$(CONFIG_CRYPTO_LIB_DES) += libdes.o libdes-y := des.o
@@ -18,3 +24,7 @@ libpoly1305-y := poly1305.o obj-$(CONFIG_CRYPTO_LIB_SHA256) += libsha256.o libsha256-y := sha256.o + +ifneq ($(CONFIG_CRYPTO_MANAGER_DISABLE_TESTS),y) +libblake2s-y += blake2s-selftest.o +endif
lib/crypto/blake2s-generic.c new file +111 -0
111 lines not on this page · raw message
lib/crypto/blake2s-selftest.c new file +622 -0
622 lines not on this page · raw message
lib/crypto/blake2s.c new file +126 -0
126 lines not on this page · raw message
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 23/34 crypto: testmgr - add test cases for Blake2s
In reply to Ard BiesheuvelAs suggested by Eric for the Blake2b implementation contributed by David, introduce a set of test vectors for Blake2s covering different digest and key sizes.
blake2s-128 blake2s-160 blake2s-224 blake2s-256
---------------------------------------------------
len=0 | klen=0 klen=1 klen=16 klen=32
len=1 | klen=16 klen=32 klen=0 klen=1
len=7 | klen=32 klen=0 klen=1 klen=16
len=15 | klen=1 klen=16 klen=32 klen=0
len=64 | klen=0 klen=1 klen=16 klen=32
len=247 | klen=16 klen=32 klen=0 klen=1
len=256 | klen=32 klen=0 klen=1 klen=16--- crypto/testmgr.c | 24 ++ crypto/testmgr.h | 295 +++++++++++++++++--- 2 files changed, 280 insertions(+), 39 deletions(-)
crypto/testmgr.c +24 -0
@@ -4050,6 +4050,30 @@ static const struct alg_test_desc alg_test_descs[] = { .suite = { .hash = __VECS(blake2b_512_tv_template) } + }, { + .alg = "blake2s-128", + .test = alg_test_hash, + .suite = { + .hash = __VECS(blakes2s_128_tv_template) + } + }, { + .alg = "blake2s-160", + .test = alg_test_hash, + .suite = { + .hash = __VECS(blakes2s_160_tv_template) + } + }, { + .alg = "blake2s-224", + .test = alg_test_hash, + .suite = { + .hash = __VECS(blakes2s_224_tv_template) + } + }, { + .alg = "blake2s-256", + .test = alg_test_hash, + .suite = { + .hash = __VECS(blakes2s_256_tv_template) + } }, { .alg = "cbc(aes)", .test = alg_test_skcipher,
crypto/testmgr.h +256 -39
461 lines not on this page · raw message
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 24/34 crypto: blake2s - implement generic shash driver
In reply to Ard BiesheuvelWire up our newly added Blake2s implementation via the shash API.
--- crypto/Kconfig | 18 +++ crypto/Makefile | 1 + crypto/blake2s_generic.c | 171 ++++++++++++++++++++ include/crypto/internal/blake2s.h | 5 + 4 files changed, 195 insertions(+)
crypto/Kconfig +18 -0
@@ -656,6 +656,24 @@ config CRYPTO_BLAKE2B See https://blake2.net for further information. +config CRYPTO_BLAKE2S + tristate "BLAKE2s digest algorithm" + select CRYPTO_LIB_BLAKE2S_GENERIC + select CRYPTO_HASH + help + Implementation of cryptographic hash function BLAKE2s + optimized for 8-32bit platforms and can produce digests of any size + between 1 to 32. The keyed hash is also implemented. + + This module provides the following algorithms: + + - blake2s-128 + - blake2s-160 + - blake2s-224 + - blake2s-256 + + See https://blake2.net for further information. + config CRYPTO_CRCT10DIF tristate "CRCT10DIF algorithm" select CRYPTO_HASH
crypto/Makefile +1 -0
@@ -74,6 +74,7 @@ obj-$(CONFIG_CRYPTO_WP512) += wp512.o CFLAGS_wp512.o := $(call cc-option,-fno-schedule-insns) # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=79149 obj-$(CONFIG_CRYPTO_TGR192) += tgr192.o obj-$(CONFIG_CRYPTO_BLAKE2B) += blake2b_generic.o +obj-$(CONFIG_CRYPTO_BLAKE2S) += blake2s_generic.o obj-$(CONFIG_CRYPTO_GF128MUL) += gf128mul.o obj-$(CONFIG_CRYPTO_ECB) += ecb.o obj-$(CONFIG_CRYPTO_CBC) += cbc.o
crypto/blake2s_generic.c new file +171 -0
171 lines not on this page · raw message
include/crypto/internal/blake2s.h +5 -0
@@ -5,6 +5,11 @@#include <crypto/blake2s.h> +struct blake2s_tfm_ctx { +u8 key[BLAKE2S_KEY_SIZE ]; +unsigned int keylen; +}; +void blake2s_compress_generic (struct blake2s_state *state,const u8 *block,size_t nblocks,const u32 inc);
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 25/34 crypto: BLAKE2s - x86_64 SIMD implementation
In reply to Ard BiesheuvelFrom: "Jason A. Donenfeld" <[email protected]>
These implementations from Samuel Neves support AVX and AVX-512VL. Originally this used AVX-512F, but Skylake thermal throttling made AVX-512VL more attractive and possible to do with negligable difference.
--- arch/x86/crypto/Makefile | 2 + arch/x86/crypto/blake2s-core.S | 258 ++++++++++++++++++++ arch/x86/crypto/blake2s-glue.c | 233 ++++++++++++++++++ crypto/Kconfig | 6 + 4 files changed, 499 insertions(+)
arch/x86/crypto/Makefile +2 -0
@@ -48,6 +48,7 @@ ifeq ($(avx_supported),yes) obj-$(CONFIG_CRYPTO_CAST6_AVX_X86_64) += cast6-avx-x86_64.o obj-$(CONFIG_CRYPTO_TWOFISH_AVX_X86_64) += twofish-avx-x86_64.o obj-$(CONFIG_CRYPTO_SERPENT_AVX_X86_64) += serpent-avx-x86_64.o + obj-$(CONFIG_CRYPTO_BLAKE2S_X86) += blake2s-x86_64.o endif # These modules require assembler to support AVX2.
@@ -70,6 +71,7 @@ serpent-sse2-x86_64-y := serpent-sse2-x86_64-asm_64.o serpent_sse2_glue.o aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o nhpoly1305-sse2-y := nh-sse2-x86_64.o nhpoly1305-sse2-glue.o +blake2s-x86_64-y := blake2s-core.o blake2s-glue.o ifeq ($(avx_supported),yes) camellia-aesni-avx-x86_64-y := camellia-aesni-avx-asm_64.o \
arch/x86/crypto/blake2s-core.S new file +258 -0
258 lines not on this page · raw message
arch/x86/crypto/blake2s-glue.c new file +233 -0
233 lines not on this page · raw message
crypto/Kconfig +6 -0
@@ -674,6 +674,12 @@ config CRYPTO_BLAKE2S See https://blake2.net for further information. +config CRYPTO_BLAKE2S_X86 + tristate "BLAKE2s digest algorithm (x86 accelerated version)" + depends on X86 && 64BIT + select CRYPTO_LIB_BLAKE2S_GENERIC + select CRYPTO_ARCH_HAVE_LIB_BLAKE2S + config CRYPTO_CRCT10DIF tristate "CRCT10DIF algorithm" select CRYPTO_HASH
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 26/34 crypto: Curve25519 - generic C library implementations
In reply to Ard BiesheuvelFrom: "Jason A. Donenfeld" <[email protected]>
This contains two formally verified C implementations of the Curve25519 scalar multiplication function, one for 32-bit systems, and one for 64-bit systems whose compiler supports efficient 128-bit integer types. Not only are these implementations formally verified, but they are also the fastest available C implementations. They have been modified to be friendly to kernel space and to be generally less horrendous looking, but still an effort has been made to retain their formally verified characteristic, and so the C might look slightly unidiomatic.
The 64-bit version comes from HACL*: https://github.com/project-everest/hacl-star
The 32-bit version comes from Fiat: https://github.com/mit-plv/fiat-crypto
Information: https://cr.yp.to/ecdh.html
[ardb: - move from lib/zinc to lib/crypto
- replace .c #includes with Kconfig based object selection
- drop simd handling and simplify support for per-arch versions ]
--- include/crypto/curve25519.h | 71 ++ lib/crypto/Kconfig | 25 + lib/crypto/Makefile | 5 + lib/crypto/curve25519-fiat32.c | 864 ++++++++++++++++++++ lib/crypto/curve25519-hacl64.c | 788 ++++++++++++++++++ lib/crypto/curve25519.c | 25 + 6 files changed, 1778 insertions(+)
include/crypto/curve25519.h new file +71 -0
@@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef CURVE25519_H +#define CURVE25519_H + +#include <crypto/algapi.h> // For crypto_memneq. +#include <linux/types.h> +#include <linux/random.h> + +enum curve25519_lengths { +CURVE25519_KEY_SIZE =32 +}; + +extern const u8 curve25519_null_point[]; +extern const u8 curve25519_base_point[]; + +void curve25519_generic (u8 out[CURVE25519_KEY_SIZE ], +const u8 scalar[CURVE25519_KEY_SIZE ], +const u8 point[CURVE25519_KEY_SIZE ]); + +void curve25519_arch (u8 out[CURVE25519_KEY_SIZE ], +const u8 scalar[CURVE25519_KEY_SIZE ], +const u8 point[CURVE25519_KEY_SIZE ]); + +void curve25519_base_arch (u8 pub[CURVE25519_KEY_SIZE ], +const u8 secret[CURVE25519_KEY_SIZE ]); + +static inline +bool __must_check curve25519 (u8 mypublic[CURVE25519_KEY_SIZE ], +const u8 secret[CURVE25519_KEY_SIZE ], +const u8 basepoint[CURVE25519_KEY_SIZE ]) +{ +if (IS_ENABLED (CONFIG_CRYPTO_ARCH_HAVE_LIB_CURVE25519 )) +curve25519_arch (mypublic, secret, basepoint); +else +curve25519_generic (mypublic, secret, basepoint); +return crypto_memneq (mypublic, curve25519_null_point, +CURVE25519_KEY_SIZE ); +} + +static inline bool +__must_check curve25519_generate_public (u8 pub[CURVE25519_KEY_SIZE ], +const u8 secret[CURVE25519_KEY_SIZE ]) +{ +if (unlikely (!crypto_memneq (secret, curve25519_null_point, +CURVE25519_KEY_SIZE ))) +return false; + +if (IS_ENABLED (CONFIG_CRYPTO_ARCH_HAVE_LIB_CURVE25519 )) +curve25519_base_arch (pub, secret); +else +curve25519_generic (pub, secret, curve25519_base_point); +return crypto_memneq (pub, curve25519_null_point,CURVE25519_KEY_SIZE ); +} + +static inline void curve25519_clamp_secret (u8 secret[CURVE25519_KEY_SIZE ]) +{ + secret[0 ] &=248 ; + secret[31 ] = (secret[31 ] &127 ) |64 ; +} + +static inline void curve25519_generate_secret (u8 secret[CURVE25519_KEY_SIZE ]) +{ +get_random_bytes_wait (secret,CURVE25519_KEY_SIZE ); +curve25519_clamp_secret (secret); +} + +#endif /* CURVE25519_H */
lib/crypto/Kconfig +25 -0
@@ -59,6 +59,31 @@ config CRYPTO_LIB_CHACHA by either the generic implementation or an arch-specific one, if one is available and enabled. +config CRYPTO_ARCH_HAVE_LIB_CURVE25519 + tristate + help + Declares whether the architecture provides an arch-specific + accelerated implementation of the Curve25519 library interface, + either builtin or as a module. + +config CRYPTO_LIB_CURVE25519_GENERIC + tristate + help + This symbol can be depended upon by arch implementations of the + Curve25519 library interface that require the generic code as a + fallback, e.g., for SIMD implementations. If no arch specific + implementation is enabled, this implementation serves the users + of CRYPTO_LIB_CURVE25519. + +config CRYPTO_LIB_CURVE25519 + tristate "Curve25519 scalar multiplication library" + depends on CRYPTO_ARCH_HAVE_LIB_CURVE25519 || !CRYPTO_ARCH_HAVE_LIB_CURVE25519 + select CRYPTO_LIB_CURVE25519_GENERIC if CRYPTO_ARCH_HAVE_LIB_CURVE25519=n + help + Enable the Curve25519 library interface. This interface may be + fulfilled by either the generic implementation or an arch-specific + one, if one is available and enabled. + config CRYPTO_LIB_DES tristate
lib/crypto/Makefile +5 -0
@@ -16,6 +16,11 @@ libblake2s-generic-y += blake2s-generic.o obj-$(CONFIG_CRYPTO_LIB_BLAKE2S) += libblake2s.o libblake2s-y += blake2s.o +obj-$(CONFIG_CRYPTO_LIB_CURVE25519_GENERIC) += libcurve25519.o +libcurve25519-y := curve25519-fiat32.o +libcurve25519-$(CONFIG_ARCH_SUPPORTS_INT128) := curve25519-hacl64.o +libcurve25519-y += curve25519.o + obj-$(CONFIG_CRYPTO_LIB_DES) += libdes.o libdes-y := des.o
lib/crypto/curve25519-fiat32.c new file +864 -0
864 lines not on this page · raw message
lib/crypto/curve25519-hacl64.c new file +788 -0
788 lines not on this page · raw message
lib/crypto/curve25519.c new file +25 -0
@@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 OR MIT +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +* +* This is an implementation of the Curve25519 ECDH algorithm, using either +* a 32-bit implementation or a 64-bit implementation with 128-bit integers, +* depending on what is supported by the target compiler. +* +* Information: https://cr.yp.to/ecdh.html +*/ + +#include <crypto/curve25519.h> +#include <linux/module.h> +#include <linux/init.h> + +const u8 curve25519_null_point[CURVE25519_KEY_SIZE ]__aligned(32) = {0 }; +const u8 curve25519_base_point[CURVE25519_KEY_SIZE ]__aligned(32) = {9 }; + +EXPORT_SYMBOL (curve25519_null_point); +EXPORT_SYMBOL (curve25519_base_point); +EXPORT_SYMBOL (curve25519_generic); + +MODULE_LICENSE ("GPL v2" ); +MODULE_DESCRIPTION ("Curve25519 scalar multiplication" ); +MODULE_AUTHOR ("Jason A. Donenfeld <[email protected]>" );
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 27/34 crypto: curve25519 - add kpp selftest
In reply to Ard BiesheuvelIn preparation of introducing KPP implementations of Curve25519, import the set of test cases proposed by the Zinc patch set, but converted to the KPP format.
--- crypto/testmgr.c | 6 + crypto/testmgr.h | 1225 ++++++++++++++++++++ 2 files changed, 1231 insertions(+)
crypto/testmgr.c +6 -0
@@ -4317,6 +4317,12 @@ static const struct alg_test_desc alg_test_descs[] = { .alg = "cts(cbc(paes))", .test = alg_test_null, .fips_allowed = 1, + }, { + .alg = "curve25519", + .test = alg_test_kpp, + .suite = { + .kpp = __VECS(curve25519_tv_template) + } }, { .alg = "deflate", .test = alg_test_comp,
crypto/testmgr.h +1225 -0
1,231 lines not on this page · raw message
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 28/34 crypto: curve25519 - implement generic KPP driver
In reply to Ard BiesheuvelExpose the generic Curve25519 library via the crypto API KPP interface.
--- crypto/Kconfig | 5 ++ crypto/Makefile | 1 + crypto/curve25519-generic.c | 90 ++++++++++++++++++++ 3 files changed, 96 insertions(+)
crypto/Kconfig +5 -0
@@ -264,6 +264,11 @@ config CRYPTO_ECRDSA standard algorithms (called GOST algorithms). Only signature verification is implemented. +config CRYPTO_CURVE25519 + tristate "Curve25519 algorithm" + select CRYPTO_KPP + select CRYPTO_LIB_CURVE25519_GENERIC + comment "Authenticated Encryption with Associated Data" config CRYPTO_CCM
crypto/Makefile +1 -0
@@ -167,6 +167,7 @@ obj-$(CONFIG_CRYPTO_ZSTD) += zstd.o obj-$(CONFIG_CRYPTO_OFB) += ofb.o obj-$(CONFIG_CRYPTO_ECC) += ecc.o obj-$(CONFIG_CRYPTO_ESSIV) += essiv.o +obj-$(CONFIG_CRYPTO_CURVE25519) += curve25519-generic.o ecdh_generic-y += ecdh.o ecdh_generic-y += ecdh_helper.o
crypto/curve25519-generic.c new file +90 -0
90 lines not on this page · raw message
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 29/34 crypto: lib/curve25519 - work around Clang stack spilling issue
In reply to Ard BiesheuvelArnd reports that the 32-bit generic library code for Curve25119 ends up using an excessive amount of stack space when built with Clang:
lib/crypto/curve25519-fiat32.c:756:6: error: stack frame size
of 1384 bytes in function 'curve25519_generic'
[-Werror,-Wframe-larger-than=]Let's give some hints to the compiler regarding which routines should not be inlined, to prevent it from running out of registers and spilling to the stack. The resulting code performs identically under both GCC and Clang, and makes the warning go away.
--- lib/crypto/curve25519-fiat32.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
lib/crypto/curve25519-fiat32.c +5 -5
@@ -223,7 +223,7 @@ static __always_inline void fe_1(fe *h) h->v[0 ] =1 ; } -static void fe_add_impl (u32 out[10 ],const u32 in1[10 ],const u32 in2[10 ]) +static noinline void fe_add_impl(u32 out[10], const u32 in1[10], const u32 in2[10]) { {const u32 x20 = in1[9 ]; {const u32 x21 = in1[8 ];
@@ -266,7 +266,7 @@ static __always_inline void fe_add(fe_loose *h, const fe *f, const fe *g)fe_add_impl (h->v, f->v, g->v); } -static void fe_sub_impl (u32 out[10 ],const u32 in1[10 ],const u32 in2[10 ]) +static noinline void fe_sub_impl(u32 out[10], const u32 in1[10], const u32 in2[10]) { {const u32 x20 = in1[9 ]; {const u32 x21 = in1[8 ];
@@ -309,7 +309,7 @@ static __always_inline void fe_sub(fe_loose *h, const fe *f, const fe *g)fe_sub_impl (h->v, f->v, g->v); } -static void fe_mul_impl (u32 out[10 ],const u32 in1[10 ],const u32 in2[10 ]) +static noinline void fe_mul_impl(u32 out[10], const u32 in1[10], const u32 in2[10]) { {const u32 x20 = in1[9 ]; {const u32 x21 = in1[8 ];
@@ -441,7 +441,7 @@ fe_mul_tll(fe *h, const fe_loose *f, const fe_loose *g)fe_mul_impl (h->v, f->v, g->v); } -static void fe_sqr_impl (u32 out[10 ],const u32 in1[10 ]) +static noinline void fe_sqr_impl(u32 out[10], const u32 in1[10]) { {const u32 x17 = in1[9 ]; {const u32 x18 = in1[8 ];
@@ -619,7 +619,7 @@ static __always_inline void fe_invert(fe *out, const fe *z)* * Preconditions: b in {0,1} */ -static __always_inline void fe_cswap(fe *f, fe *g, unsigned int b) +static noinline void fe_cswap(fe *f, fe *g, unsigned int b) {unsigned i; b =0 - b;
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 30/34 crypto: Curve25519 - x86_64 library and KPP implementations
In reply to Ard BiesheuvelFrom: "Jason A. Donenfeld" <[email protected]>
This implementation is the fastest available x86_64 implementation, and unlike Sandy2x, it doesn't requie use of the floating point registers at all. Instead it makes use of BMI2 and ADX, available on recent microarchitectures. The implementation was written by Armando Faz-Hernández with contributions (upstream) from Samuel Neves and me, in addition to further changes in the kernel implementation from us.
[ardb: - move to arch/x86/crypto
- wire into lib/crypto framework
- implement crypto API KPP hooks ]
--- arch/x86/crypto/Makefile | 1 + arch/x86/crypto/curve25519-x86_64.c | 2475 ++++++++++++++++++++ crypto/Kconfig | 6 + 3 files changed, 2482 insertions(+)
arch/x86/crypto/Makefile +1 -0
@@ -39,6 +39,7 @@ obj-$(CONFIG_CRYPTO_AEGIS128_AESNI_SSE2) += aegis128-aesni.o obj-$(CONFIG_CRYPTO_NHPOLY1305_SSE2) += nhpoly1305-sse2.o obj-$(CONFIG_CRYPTO_NHPOLY1305_AVX2) += nhpoly1305-avx2.o +obj-$(CONFIG_CRYPTO_CURVE25519_X86) += curve25519-x86_64.o # These modules require assembler to support AVX. ifeq ($(avx_supported),yes)
arch/x86/crypto/curve25519-x86_64.c new file +2475 -0
2,475 lines not on this page · raw message
crypto/Kconfig +6 -0
@@ -269,6 +269,12 @@ config CRYPTO_CURVE25519 select CRYPTO_KPP select CRYPTO_LIB_CURVE25519_GENERIC +config CRYPTO_CURVE25519_X86 + tristate "x86_64 accelerated Curve25519 scalar multiplication library" + depends on X86 && 64BIT + select CRYPTO_LIB_CURVE25519_GENERIC + select CRYPTO_ARCH_HAVE_LIB_CURVE25519 + comment "Authenticated Encryption with Associated Data" config CRYPTO_CCM
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 31/34 crypto: arm - import Bernstein and Schwabe's Curve25519 ARM implementation
In reply to Ard BiesheuvelFrom: "Jason A. Donenfeld" <[email protected]>
This comes from Dan Bernstein and Peter Schwabe's public domain NEON code, and is included here in raw form so that subsequent commits that fix these up for the kernel can see how it has changed. This code does have some entirely cosmetic formatting differences, adding indentation and so forth, so that when we actually port it for use in the kernel in the subsequent commit, it's obvious what's changed in the process.
This code originates from SUPERCOP 20180818, available at
<https://bench.cr.yp.to/supercop.html>.
--- arch/arm/crypto/curve25519-core.S | 2105 ++++++++++++++++++++ 1 file changed, 2105 insertions(+)
arch/arm/crypto/curve25519-core.S new file +2105 -0
@@ -0,0 +1,2105 @@ +/* + * Public domain code from Daniel J. Bernstein and Peter Schwabe, from + * SUPERCOP's curve25519/neon2/scalarmult.s. + */ + +.fpu neon +.text +.align 4 +.global _crypto_scalarmult_curve25519_neon2 +.global crypto_scalarmult_curve25519_neon2 +.type _crypto_scalarmult_curve25519_neon2 STT_FUNC +.type crypto_scalarmult_curve25519_neon2 STT_FUNC + _crypto_scalarmult_curve25519_neon2: + crypto_scalarmult_curve25519_neon2: + vpush {q4, q5, q6, q7} + mov r12, sp + sub sp, sp, #736 + and sp, sp, #0xffffffe0 + strd r4, [sp, #0] + strd r6, [sp, #8] + strd r8, [sp, #16] + strd r10, [sp, #24] + str r12, [sp, #480] + str r14, [sp, #484] + mov r0, r0 + mov r1, r1 + mov r2, r2 + add r3, sp, #32 + ldr r4, =0 + ldr r5, =254 + vmov.i32 q0, #1 + vshr.u64 q1, q0, #7 + vshr.u64 q0, q0, #8 + vmov.i32 d4, #19 + vmov.i32 d5, #38 + add r6, sp, #512 + vst1.8 {d2-d3}, [r6, : 128] + add r6, sp, #528 + vst1.8 {d0-d1}, [r6, : 128] + add r6, sp, #544 + vst1.8 {d4-d5}, [r6, : 128] + add r6, r3, #0 + vmov.i32 q2, #0 + vst1.8 {d4-d5}, [r6, : 128]! + vst1.8 {d4-d5}, [r6, : 128]! + vst1.8 d4, [r6, : 64] + add r6, r3, #0 + ldr r7, =960 + sub r7, r7, #2 + neg r7, r7 + sub r7, r7, r7, LSL #7 + str r7, [r6] + add r6, sp, #704 + vld1.8 {d4-d5}, [r1]! + vld1.8 {d6-d7}, [r1] + vst1.8 {d4-d5}, [r6, : 128]! + vst1.8 {d6-d7}, [r6, : 128] + sub r1, r6, #16 + ldrb r6, [r1] + and r6, r6, #248 + strb r6, [r1] + ldrb r6, [r1, #31] + and r6, r6, #127 + orr r6, r6, #64 + strb r6, [r1, #31] + vmov.i64 q2, #0xffffffff + vshr.u64 q3, q2, #7 + vshr.u64 q2, q2, #6 + vld1.8 {d8}, [r2] + vld1.8 {d10}, [r2] + add r2, r2, #6 + vld1.8 {d12}, [r2] + vld1.8 {d14}, [r2] + add r2, r2, #6 + vld1.8 {d16}, [r2] + add r2, r2, #4 + vld1.8 {d18}, [r2] + vld1.8 {d20}, [r2] + add r2, r2, #6 + vld1.8 {d22}, [r2] + add r2, r2, #2 + vld1.8 {d24}, [r2] + vld1.8 {d26}, [r2] + vshr.u64 q5, q5, #26 + vshr.u64 q6, q6, #3 + vshr.u64 q7, q7, #29 + vshr.u64 q8, q8, #6 + vshr.u64 q10, q10, #25 + vshr.u64 q11, q11, #3 + vshr.u64 q12, q12, #12 + vshr.u64 q13, q13, #38 + vand q4, q4, q2 + vand q6, q6, q2 + vand q8, q8, q2 + vand q10, q10, q2 + vand q2, q12, q2 + vand q5, q5, q3 + vand q7, q7, q3 + vand q9, q9, q3 + vand q11, q11, q3 + vand q3, q13, q3 + add r2, r3, #48 + vadd.i64 q12, q4, q1 + vadd.i64 q13, q10, q1 + vshr.s64 q12, q12, #26 + vshr.s64 q13, q13, #26 + vadd.i64 q5, q5, q12 + vshl.i64 q12, q12, #26 + vadd.i64 q14, q5, q0 + vadd.i64 q11, q11, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q15, q11, q0 + vsub.i64 q4, q4, q12 + vshr.s64 q12, q14, #25 + vsub.i64 q10, q10, q13 + vshr.s64 q13, q15, #25 + vadd.i64 q6, q6, q12 + vshl.i64 q12, q12, #25 + vadd.i64 q14, q6, q1 + vadd.i64 q2, q2, q13 + vsub.i64 q5, q5, q12 + vshr.s64 q12, q14, #26 + vshl.i64 q13, q13, #25 + vadd.i64 q14, q2, q1 + vadd.i64 q7, q7, q12 + vshl.i64 q12, q12, #26 + vadd.i64 q15, q7, q0 + vsub.i64 q11, q11, q13 + vshr.s64 q13, q14, #26 + vsub.i64 q6, q6, q12 + vshr.s64 q12, q15, #25 + vadd.i64 q3, q3, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q14, q3, q0 + vadd.i64 q8, q8, q12 + vshl.i64 q12, q12, #25 + vadd.i64 q15, q8, q1 + add r2, r2, #8 + vsub.i64 q2, q2, q13 + vshr.s64 q13, q14, #25 + vsub.i64 q7, q7, q12 + vshr.s64 q12, q15, #26 + vadd.i64 q14, q13, q13 + vadd.i64 q9, q9, q12 + vtrn.32 d12, d14 + vshl.i64 q12, q12, #26 + vtrn.32 d13, d15 + vadd.i64 q0, q9, q0 + vadd.i64 q4, q4, q14 + vst1.8 d12, [r2, : 64]! + vshl.i64 q6, q13, #4 + vsub.i64 q7, q8, q12 + vshr.s64 q0, q0, #25 + vadd.i64 q4, q4, q6 + vadd.i64 q6, q10, q0 + vshl.i64 q0, q0, #25 + vadd.i64 q8, q6, q1 + vadd.i64 q4, q4, q13 + vshl.i64 q10, q13, #25 + vadd.i64 q1, q4, q1 + vsub.i64 q0, q9, q0 + vshr.s64 q8, q8, #26 + vsub.i64 q3, q3, q10 + vtrn.32 d14, d0 + vshr.s64 q1, q1, #26 + vtrn.32 d15, d1 + vadd.i64 q0, q11, q8 + vst1.8 d14, [r2, : 64] + vshl.i64 q7, q8, #26 + vadd.i64 q5, q5, q1 + vtrn.32 d4, d6 + vshl.i64 q1, q1, #26 + vtrn.32 d5, d7 + vsub.i64 q3, q6, q7 + add r2, r2, #16 + vsub.i64 q1, q4, q1 + vst1.8 d4, [r2, : 64] + vtrn.32 d6, d0 + vtrn.32 d7, d1 + sub r2, r2, #8 + vtrn.32 d2, d10 + vtrn.32 d3, d11 + vst1.8 d6, [r2, : 64] + sub r2, r2, #24 + vst1.8 d2, [r2, : 64] + add r2, r3, #96 + vmov.i32 q0, #0 + vmov.i64 d2, #0xff + vmov.i64 d3, #0 + vshr.u32 q1, q1, #7 + vst1.8 {d2-d3}, [r2, : 128]! + vst1.8 {d0-d1}, [r2, : 128]! + vst1.8 d0, [r2, : 64] + add r2, r3, #144 + vmov.i32 q0, #0 + vst1.8 {d0-d1}, [r2, : 128]! + vst1.8 {d0-d1}, [r2, : 128]! + vst1.8 d0, [r2, : 64] + add r2, r3, #240 + vmov.i32 q0, #0 + vmov.i64 d2, #0xff + vmov.i64 d3, #0 + vshr.u32 q1, q1, #7 + vst1.8 {d2-d3}, [r2, : 128]! + vst1.8 {d0-d1}, [r2, : 128]! + vst1.8 d0, [r2, : 64] + add r2, r3, #48 + add r6, r3, #192 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d4}, [r2, : 64] + vst1.8 {d0-d1}, [r6, : 128]! + vst1.8 {d2-d3}, [r6, : 128]! + vst1.8 d4, [r6, : 64] +._mainloop: + mov r2, r5, LSR #3 + and r6, r5, #7 + ldrb r2, [r1, r2] + mov r2, r2, LSR r6 + and r2, r2, #1 + str r5, [sp, #488] + eor r4, r4, r2 + str r2, [sp, #492] + neg r2, r4 + add r4, r3, #96 + add r5, r3, #192 + add r6, r3, #144 + vld1.8 {d8-d9}, [r4, : 128]! + add r7, r3, #240 + vld1.8 {d10-d11}, [r5, : 128]! + veor q6, q4, q5 + vld1.8 {d14-d15}, [r6, : 128]! + vdup.i32 q8, r2 + vld1.8 {d18-d19}, [r7, : 128]! + veor q10, q7, q9 + vld1.8 {d22-d23}, [r4, : 128]! + vand q6, q6, q8 + vld1.8 {d24-d25}, [r5, : 128]! + vand q10, q10, q8 + vld1.8 {d26-d27}, [r6, : 128]! + veor q4, q4, q6 + vld1.8 {d28-d29}, [r7, : 128]! + veor q5, q5, q6 + vld1.8 {d0}, [r4, : 64] + veor q6, q7, q10 + vld1.8 {d2}, [r5, : 64] + veor q7, q9, q10 + vld1.8 {d4}, [r6, : 64] + veor q9, q11, q12 + vld1.8 {d6}, [r7, : 64] + veor q10, q0, q1 + sub r2, r4, #32 + vand q9, q9, q8 + sub r4, r5, #32 + vand q10, q10, q8 + sub r5, r6, #32 + veor q11, q11, q9 + sub r6, r7, #32 + veor q0, q0, q10 + veor q9, q12, q9 + veor q1, q1, q10 + veor q10, q13, q14 + veor q12, q2, q3 + vand q10, q10, q8 + vand q8, q12, q8 + veor q12, q13, q10 + veor q2, q2, q8 + veor q10, q14, q10 + veor q3, q3, q8 + vadd.i32 q8, q4, q6 + vsub.i32 q4, q4, q6 + vst1.8 {d16-d17}, [r2, : 128]! + vadd.i32 q6, q11, q12 + vst1.8 {d8-d9}, [r5, : 128]! + vsub.i32 q4, q11, q12 + vst1.8 {d12-d13}, [r2, : 128]! + vadd.i32 q6, q0, q2 + vst1.8 {d8-d9}, [r5, : 128]! + vsub.i32 q0, q0, q2 + vst1.8 d12, [r2, : 64] + vadd.i32 q2, q5, q7 + vst1.8 d0, [r5, : 64] + vsub.i32 q0, q5, q7 + vst1.8 {d4-d5}, [r4, : 128]! + vadd.i32 q2, q9, q10 + vst1.8 {d0-d1}, [r6, : 128]! + vsub.i32 q0, q9, q10 + vst1.8 {d4-d5}, [r4, : 128]! + vadd.i32 q2, q1, q3 + vst1.8 {d0-d1}, [r6, : 128]! + vsub.i32 q0, q1, q3 + vst1.8 d4, [r4, : 64] + vst1.8 d0, [r6, : 64] + add r2, sp, #544 + add r4, r3, #96 + add r5, r3, #144 + vld1.8 {d0-d1}, [r2, : 128] + vld1.8 {d2-d3}, [r4, : 128]! + vld1.8 {d4-d5}, [r5, : 128]! + vzip.i32 q1, q2 + vld1.8 {d6-d7}, [r4, : 128]! + vld1.8 {d8-d9}, [r5, : 128]! + vshl.i32 q5, q1, #1 + vzip.i32 q3, q4 + vshl.i32 q6, q2, #1 + vld1.8 {d14}, [r4, : 64] + vshl.i32 q8, q3, #1 + vld1.8 {d15}, [r5, : 64] + vshl.i32 q9, q4, #1 + vmul.i32 d21, d7, d1 + vtrn.32 d14, d15 + vmul.i32 q11, q4, q0 + vmul.i32 q0, q7, q0 + vmull.s32 q12, d2, d2 + vmlal.s32 q12, d11, d1 + vmlal.s32 q12, d12, d0 + vmlal.s32 q12, d13, d23 + vmlal.s32 q12, d16, d22 + vmlal.s32 q12, d7, d21 + vmull.s32 q10, d2, d11 + vmlal.s32 q10, d4, d1 + vmlal.s32 q10, d13, d0 + vmlal.s32 q10, d6, d23 + vmlal.s32 q10, d17, d22 + vmull.s32 q13, d10, d4 + vmlal.s32 q13, d11, d3 + vmlal.s32 q13, d13, d1 + vmlal.s32 q13, d16, d0 + vmlal.s32 q13, d17, d23 + vmlal.s32 q13, d8, d22 + vmull.s32 q1, d10, d5 + vmlal.s32 q1, d11, d4 + vmlal.s32 q1, d6, d1 + vmlal.s32 q1, d17, d0 + vmlal.s32 q1, d8, d23 + vmull.s32 q14, d10, d6 + vmlal.s32 q14, d11, d13 + vmlal.s32 q14, d4, d4 + vmlal.s32 q14, d17, d1 + vmlal.s32 q14, d18, d0 + vmlal.s32 q14, d9, d23 + vmull.s32 q11, d10, d7 + vmlal.s32 q11, d11, d6 + vmlal.s32 q11, d12, d5 + vmlal.s32 q11, d8, d1 + vmlal.s32 q11, d19, d0 + vmull.s32 q15, d10, d8 + vmlal.s32 q15, d11, d17 + vmlal.s32 q15, d12, d6 + vmlal.s32 q15, d13, d5 + vmlal.s32 q15, d19, d1 + vmlal.s32 q15, d14, d0 + vmull.s32 q2, d10, d9 + vmlal.s32 q2, d11, d8 + vmlal.s32 q2, d12, d7 + vmlal.s32 q2, d13, d6 + vmlal.s32 q2, d14, d1 + vmull.s32 q0, d15, d1 + vmlal.s32 q0, d10, d14 + vmlal.s32 q0, d11, d19 + vmlal.s32 q0, d12, d8 + vmlal.s32 q0, d13, d17 + vmlal.s32 q0, d6, d6 + add r2, sp, #512 + vld1.8 {d18-d19}, [r2, : 128] + vmull.s32 q3, d16, d7 + vmlal.s32 q3, d10, d15 + vmlal.s32 q3, d11, d14 + vmlal.s32 q3, d12, d9 + vmlal.s32 q3, d13, d8 + add r2, sp, #528 + vld1.8 {d8-d9}, [r2, : 128] + vadd.i64 q5, q12, q9 + vadd.i64 q6, q15, q9 + vshr.s64 q5, q5, #26 + vshr.s64 q6, q6, #26 + vadd.i64 q7, q10, q5 + vshl.i64 q5, q5, #26 + vadd.i64 q8, q7, q4 + vadd.i64 q2, q2, q6 + vshl.i64 q6, q6, #26 + vadd.i64 q10, q2, q4 + vsub.i64 q5, q12, q5 + vshr.s64 q8, q8, #25 + vsub.i64 q6, q15, q6 + vshr.s64 q10, q10, #25 + vadd.i64 q12, q13, q8 + vshl.i64 q8, q8, #25 + vadd.i64 q13, q12, q9 + vadd.i64 q0, q0, q10 + vsub.i64 q7, q7, q8 + vshr.s64 q8, q13, #26 + vshl.i64 q10, q10, #25 + vadd.i64 q13, q0, q9 + vadd.i64 q1, q1, q8 + vshl.i64 q8, q8, #26 + vadd.i64 q15, q1, q4 + vsub.i64 q2, q2, q10 + vshr.s64 q10, q13, #26 + vsub.i64 q8, q12, q8 + vshr.s64 q12, q15, #25 + vadd.i64 q3, q3, q10 + vshl.i64 q10, q10, #26 + vadd.i64 q13, q3, q4 + vadd.i64 q14, q14, q12 + add r2, r3, #288 + vshl.i64 q12, q12, #25 + add r4, r3, #336 + vadd.i64 q15, q14, q9 + add r2, r2, #8 + vsub.i64 q0, q0, q10 + add r4, r4, #8 + vshr.s64 q10, q13, #25 + vsub.i64 q1, q1, q12 + vshr.s64 q12, q15, #26 + vadd.i64 q13, q10, q10 + vadd.i64 q11, q11, q12 + vtrn.32 d16, d2 + vshl.i64 q12, q12, #26 + vtrn.32 d17, d3 + vadd.i64 q1, q11, q4 + vadd.i64 q4, q5, q13 + vst1.8 d16, [r2, : 64]! + vshl.i64 q5, q10, #4 + vst1.8 d17, [r4, : 64]! + vsub.i64 q8, q14, q12 + vshr.s64 q1, q1, #25 + vadd.i64 q4, q4, q5 + vadd.i64 q5, q6, q1 + vshl.i64 q1, q1, #25 + vadd.i64 q6, q5, q9 + vadd.i64 q4, q4, q10 + vshl.i64 q10, q10, #25 + vadd.i64 q9, q4, q9 + vsub.i64 q1, q11, q1 + vshr.s64 q6, q6, #26 + vsub.i64 q3, q3, q10 + vtrn.32 d16, d2 + vshr.s64 q9, q9, #26 + vtrn.32 d17, d3 + vadd.i64 q1, q2, q6 + vst1.8 d16, [r2, : 64] + vshl.i64 q2, q6, #26 + vst1.8 d17, [r4, : 64] + vadd.i64 q6, q7, q9 + vtrn.32 d0, d6 + vshl.i64 q7, q9, #26 + vtrn.32 d1, d7 + vsub.i64 q2, q5, q2 + add r2, r2, #16 + vsub.i64 q3, q4, q7 + vst1.8 d0, [r2, : 64] + add r4, r4, #16 + vst1.8 d1, [r4, : 64] + vtrn.32 d4, d2 + vtrn.32 d5, d3 + sub r2, r2, #8 + sub r4, r4, #8 + vtrn.32 d6, d12 + vtrn.32 d7, d13 + vst1.8 d4, [r2, : 64] + vst1.8 d5, [r4, : 64] + sub r2, r2, #24 + sub r4, r4, #24 + vst1.8 d6, [r2, : 64] + vst1.8 d7, [r4, : 64] + add r2, r3, #240 + add r4, r3, #96 + vld1.8 {d0-d1}, [r4, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vld1.8 {d4}, [r4, : 64] + add r4, r3, #144 + vld1.8 {d6-d7}, [r4, : 128]! + vtrn.32 q0, q3 + vld1.8 {d8-d9}, [r4, : 128]! + vshl.i32 q5, q0, #4 + vtrn.32 q1, q4 + vshl.i32 q6, q3, #4 + vadd.i32 q5, q5, q0 + vadd.i32 q6, q6, q3 + vshl.i32 q7, q1, #4 + vld1.8 {d5}, [r4, : 64] + vshl.i32 q8, q4, #4 + vtrn.32 d4, d5 + vadd.i32 q7, q7, q1 + vadd.i32 q8, q8, q4 + vld1.8 {d18-d19}, [r2, : 128]! + vshl.i32 q10, q2, #4 + vld1.8 {d22-d23}, [r2, : 128]! + vadd.i32 q10, q10, q2 + vld1.8 {d24}, [r2, : 64] + vadd.i32 q5, q5, q0 + add r2, r3, #192 + vld1.8 {d26-d27}, [r2, : 128]! + vadd.i32 q6, q6, q3 + vld1.8 {d28-d29}, [r2, : 128]! + vadd.i32 q8, q8, q4 + vld1.8 {d25}, [r2, : 64] + vadd.i32 q10, q10, q2 + vtrn.32 q9, q13 + vadd.i32 q7, q7, q1 + vadd.i32 q5, q5, q0 + vtrn.32 q11, q14 + vadd.i32 q6, q6, q3 + add r2, sp, #560 + vadd.i32 q10, q10, q2 + vtrn.32 d24, d25 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q6, q13, #1 + add r2, sp, #576 + vst1.8 {d20-d21}, [r2, : 128] + vshl.i32 q10, q14, #1 + add r2, sp, #592 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q15, q12, #1 + vadd.i32 q8, q8, q4 + vext.32 d10, d31, d30, #0 + vadd.i32 q7, q7, q1 + add r2, sp, #608 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q8, d18, d5 + vmlal.s32 q8, d26, d4 + vmlal.s32 q8, d19, d9 + vmlal.s32 q8, d27, d3 + vmlal.s32 q8, d22, d8 + vmlal.s32 q8, d28, d2 + vmlal.s32 q8, d23, d7 + vmlal.s32 q8, d29, d1 + vmlal.s32 q8, d24, d6 + vmlal.s32 q8, d25, d0 + add r2, sp, #624 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q2, d18, d4 + vmlal.s32 q2, d12, d9 + vmlal.s32 q2, d13, d8 + vmlal.s32 q2, d19, d3 + vmlal.s32 q2, d22, d2 + vmlal.s32 q2, d23, d1 + vmlal.s32 q2, d24, d0 + add r2, sp, #640 + vst1.8 {d20-d21}, [r2, : 128] + vmull.s32 q7, d18, d9 + vmlal.s32 q7, d26, d3 + vmlal.s32 q7, d19, d8 + vmlal.s32 q7, d27, d2 + vmlal.s32 q7, d22, d7 + vmlal.s32 q7, d28, d1 + vmlal.s32 q7, d23, d6 + vmlal.s32 q7, d29, d0 + add r2, sp, #656 + vst1.8 {d10-d11}, [r2, : 128] + vmull.s32 q5, d18, d3 + vmlal.s32 q5, d19, d2 + vmlal.s32 q5, d22, d1 + vmlal.s32 q5, d23, d0 + vmlal.s32 q5, d12, d8 + add r2, sp, #672 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q4, d18, d8 + vmlal.s32 q4, d26, d2 + vmlal.s32 q4, d19, d7 + vmlal.s32 q4, d27, d1 + vmlal.s32 q4, d22, d6 + vmlal.s32 q4, d28, d0 + vmull.s32 q8, d18, d7 + vmlal.s32 q8, d26, d1 + vmlal.s32 q8, d19, d6 + vmlal.s32 q8, d27, d0 + add r2, sp, #576 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q7, d24, d21 + vmlal.s32 q7, d25, d20 + vmlal.s32 q4, d23, d21 + vmlal.s32 q4, d29, d20 + vmlal.s32 q8, d22, d21 + vmlal.s32 q8, d28, d20 + vmlal.s32 q5, d24, d20 + add r2, sp, #576 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q7, d18, d6 + vmlal.s32 q7, d26, d0 + add r2, sp, #656 + vld1.8 {d30-d31}, [r2, : 128] + vmlal.s32 q2, d30, d21 + vmlal.s32 q7, d19, d21 + vmlal.s32 q7, d27, d20 + add r2, sp, #624 + vld1.8 {d26-d27}, [r2, : 128] + vmlal.s32 q4, d25, d27 + vmlal.s32 q8, d29, d27 + vmlal.s32 q8, d25, d26 + vmlal.s32 q7, d28, d27 + vmlal.s32 q7, d29, d26 + add r2, sp, #608 + vld1.8 {d28-d29}, [r2, : 128] + vmlal.s32 q4, d24, d29 + vmlal.s32 q8, d23, d29 + vmlal.s32 q8, d24, d28 + vmlal.s32 q7, d22, d29 + vmlal.s32 q7, d23, d28 + add r2, sp, #608 + vst1.8 {d8-d9}, [r2, : 128] + add r2, sp, #560 + vld1.8 {d8-d9}, [r2, : 128] + vmlal.s32 q7, d24, d9 + vmlal.s32 q7, d25, d31 + vmull.s32 q1, d18, d2 + vmlal.s32 q1, d19, d1 + vmlal.s32 q1, d22, d0 + vmlal.s32 q1, d24, d27 + vmlal.s32 q1, d23, d20 + vmlal.s32 q1, d12, d7 + vmlal.s32 q1, d13, d6 + vmull.s32 q6, d18, d1 + vmlal.s32 q6, d19, d0 + vmlal.s32 q6, d23, d27 + vmlal.s32 q6, d22, d20 + vmlal.s32 q6, d24, d26 + vmull.s32 q0, d18, d0 + vmlal.s32 q0, d22, d27 + vmlal.s32 q0, d23, d26 + vmlal.s32 q0, d24, d31 + vmlal.s32 q0, d19, d20 + add r2, sp, #640 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q2, d18, d7 + vmlal.s32 q2, d19, d6 + vmlal.s32 q5, d18, d6 + vmlal.s32 q5, d19, d21 + vmlal.s32 q1, d18, d21 + vmlal.s32 q1, d19, d29 + vmlal.s32 q0, d18, d28 + vmlal.s32 q0, d19, d9 + vmlal.s32 q6, d18, d29 + vmlal.s32 q6, d19, d28 + add r2, sp, #592 + vld1.8 {d18-d19}, [r2, : 128] + add r2, sp, #512 + vld1.8 {d22-d23}, [r2, : 128] + vmlal.s32 q5, d19, d7 + vmlal.s32 q0, d18, d21 + vmlal.s32 q0, d19, d29 + vmlal.s32 q6, d18, d6 + add r2, sp, #528 + vld1.8 {d6-d7}, [r2, : 128] + vmlal.s32 q6, d19, d21 + add r2, sp, #576 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q0, d30, d8 + add r2, sp, #672 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q5, d30, d29 + add r2, sp, #608 + vld1.8 {d24-d25}, [r2, : 128] + vmlal.s32 q1, d30, d28 + vadd.i64 q13, q0, q11 + vadd.i64 q14, q5, q11 + vmlal.s32 q6, d30, d9 + vshr.s64 q4, q13, #26 + vshr.s64 q13, q14, #26 + vadd.i64 q7, q7, q4 + vshl.i64 q4, q4, #26 + vadd.i64 q14, q7, q3 + vadd.i64 q9, q9, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q15, q9, q3 + vsub.i64 q0, q0, q4 + vshr.s64 q4, q14, #25 + vsub.i64 q5, q5, q13 + vshr.s64 q13, q15, #25 + vadd.i64 q6, q6, q4 + vshl.i64 q4, q4, #25 + vadd.i64 q14, q6, q11 + vadd.i64 q2, q2, q13 + vsub.i64 q4, q7, q4 + vshr.s64 q7, q14, #26 + vshl.i64 q13, q13, #25 + vadd.i64 q14, q2, q11 + vadd.i64 q8, q8, q7 + vshl.i64 q7, q7, #26 + vadd.i64 q15, q8, q3 + vsub.i64 q9, q9, q13 + vshr.s64 q13, q14, #26 + vsub.i64 q6, q6, q7 + vshr.s64 q7, q15, #25 + vadd.i64 q10, q10, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q14, q10, q3 + vadd.i64 q1, q1, q7 + add r2, r3, #144 + vshl.i64 q7, q7, #25 + add r4, r3, #96 + vadd.i64 q15, q1, q11 + add r2, r2, #8 + vsub.i64 q2, q2, q13 + add r4, r4, #8 + vshr.s64 q13, q14, #25 + vsub.i64 q7, q8, q7 + vshr.s64 q8, q15, #26 + vadd.i64 q14, q13, q13 + vadd.i64 q12, q12, q8 + vtrn.32 d12, d14 + vshl.i64 q8, q8, #26 + vtrn.32 d13, d15 + vadd.i64 q3, q12, q3 + vadd.i64 q0, q0, q14 + vst1.8 d12, [r2, : 64]! + vshl.i64 q7, q13, #4 + vst1.8 d13, [r4, : 64]! + vsub.i64 q1, q1, q8 + vshr.s64 q3, q3, #25 + vadd.i64 q0, q0, q7 + vadd.i64 q5, q5, q3 + vshl.i64 q3, q3, #25 + vadd.i64 q6, q5, q11 + vadd.i64 q0, q0, q13 + vshl.i64 q7, q13, #25 + vadd.i64 q8, q0, q11 + vsub.i64 q3, q12, q3 + vshr.s64 q6, q6, #26 + vsub.i64 q7, q10, q7 + vtrn.32 d2, d6 + vshr.s64 q8, q8, #26 + vtrn.32 d3, d7 + vadd.i64 q3, q9, q6 + vst1.8 d2, [r2, : 64] + vshl.i64 q6, q6, #26 + vst1.8 d3, [r4, : 64] + vadd.i64 q1, q4, q8 + vtrn.32 d4, d14 + vshl.i64 q4, q8, #26 + vtrn.32 d5, d15 + vsub.i64 q5, q5, q6 + add r2, r2, #16 + vsub.i64 q0, q0, q4 + vst1.8 d4, [r2, : 64] + add r4, r4, #16 + vst1.8 d5, [r4, : 64] + vtrn.32 d10, d6 + vtrn.32 d11, d7 + sub r2, r2, #8 + sub r4, r4, #8 + vtrn.32 d0, d2 + vtrn.32 d1, d3 + vst1.8 d10, [r2, : 64] + vst1.8 d11, [r4, : 64] + sub r2, r2, #24 + sub r4, r4, #24 + vst1.8 d0, [r2, : 64] + vst1.8 d1, [r4, : 64] + add r2, r3, #288 + add r4, r3, #336 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vsub.i32 q0, q0, q1 + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d4-d5}, [r4, : 128]! + vsub.i32 q1, q1, q2 + add r5, r3, #240 + vld1.8 {d4}, [r2, : 64] + vld1.8 {d6}, [r4, : 64] + vsub.i32 q2, q2, q3 + vst1.8 {d0-d1}, [r5, : 128]! + vst1.8 {d2-d3}, [r5, : 128]! + vst1.8 d4, [r5, : 64] + add r2, r3, #144 + add r4, r3, #96 + add r5, r3, #144 + add r6, r3, #192 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vsub.i32 q2, q0, q1 + vadd.i32 q0, q0, q1 + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d6-d7}, [r4, : 128]! + vsub.i32 q4, q1, q3 + vadd.i32 q1, q1, q3 + vld1.8 {d6}, [r2, : 64] + vld1.8 {d10}, [r4, : 64] + vsub.i32 q6, q3, q5 + vadd.i32 q3, q3, q5 + vst1.8 {d4-d5}, [r5, : 128]! + vst1.8 {d0-d1}, [r6, : 128]! + vst1.8 {d8-d9}, [r5, : 128]! + vst1.8 {d2-d3}, [r6, : 128]! + vst1.8 d12, [r5, : 64] + vst1.8 d6, [r6, : 64] + add r2, r3, #0 + add r4, r3, #240 + vld1.8 {d0-d1}, [r4, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vld1.8 {d4}, [r4, : 64] + add r4, r3, #336 + vld1.8 {d6-d7}, [r4, : 128]! + vtrn.32 q0, q3 + vld1.8 {d8-d9}, [r4, : 128]! + vshl.i32 q5, q0, #4 + vtrn.32 q1, q4 + vshl.i32 q6, q3, #4 + vadd.i32 q5, q5, q0 + vadd.i32 q6, q6, q3 + vshl.i32 q7, q1, #4 + vld1.8 {d5}, [r4, : 64] + vshl.i32 q8, q4, #4 + vtrn.32 d4, d5 + vadd.i32 q7, q7, q1 + vadd.i32 q8, q8, q4 + vld1.8 {d18-d19}, [r2, : 128]! + vshl.i32 q10, q2, #4 + vld1.8 {d22-d23}, [r2, : 128]! + vadd.i32 q10, q10, q2 + vld1.8 {d24}, [r2, : 64] + vadd.i32 q5, q5, q0 + add r2, r3, #288 + vld1.8 {d26-d27}, [r2, : 128]! + vadd.i32 q6, q6, q3 + vld1.8 {d28-d29}, [r2, : 128]! + vadd.i32 q8, q8, q4 + vld1.8 {d25}, [r2, : 64] + vadd.i32 q10, q10, q2 + vtrn.32 q9, q13 + vadd.i32 q7, q7, q1 + vadd.i32 q5, q5, q0 + vtrn.32 q11, q14 + vadd.i32 q6, q6, q3 + add r2, sp, #560 + vadd.i32 q10, q10, q2 + vtrn.32 d24, d25 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q6, q13, #1 + add r2, sp, #576 + vst1.8 {d20-d21}, [r2, : 128] + vshl.i32 q10, q14, #1 + add r2, sp, #592 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q15, q12, #1 + vadd.i32 q8, q8, q4 + vext.32 d10, d31, d30, #0 + vadd.i32 q7, q7, q1 + add r2, sp, #608 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q8, d18, d5 + vmlal.s32 q8, d26, d4 + vmlal.s32 q8, d19, d9 + vmlal.s32 q8, d27, d3 + vmlal.s32 q8, d22, d8 + vmlal.s32 q8, d28, d2 + vmlal.s32 q8, d23, d7 + vmlal.s32 q8, d29, d1 + vmlal.s32 q8, d24, d6 + vmlal.s32 q8, d25, d0 + add r2, sp, #624 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q2, d18, d4 + vmlal.s32 q2, d12, d9 + vmlal.s32 q2, d13, d8 + vmlal.s32 q2, d19, d3 + vmlal.s32 q2, d22, d2 + vmlal.s32 q2, d23, d1 + vmlal.s32 q2, d24, d0 + add r2, sp, #640 + vst1.8 {d20-d21}, [r2, : 128] + vmull.s32 q7, d18, d9 + vmlal.s32 q7, d26, d3 + vmlal.s32 q7, d19, d8 + vmlal.s32 q7, d27, d2 + vmlal.s32 q7, d22, d7 + vmlal.s32 q7, d28, d1 + vmlal.s32 q7, d23, d6 + vmlal.s32 q7, d29, d0 + add r2, sp, #656 + vst1.8 {d10-d11}, [r2, : 128] + vmull.s32 q5, d18, d3 + vmlal.s32 q5, d19, d2 + vmlal.s32 q5, d22, d1 + vmlal.s32 q5, d23, d0 + vmlal.s32 q5, d12, d8 + add r2, sp, #672 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q4, d18, d8 + vmlal.s32 q4, d26, d2 + vmlal.s32 q4, d19, d7 + vmlal.s32 q4, d27, d1 + vmlal.s32 q4, d22, d6 + vmlal.s32 q4, d28, d0 + vmull.s32 q8, d18, d7 + vmlal.s32 q8, d26, d1 + vmlal.s32 q8, d19, d6 + vmlal.s32 q8, d27, d0 + add r2, sp, #576 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q7, d24, d21 + vmlal.s32 q7, d25, d20 + vmlal.s32 q4, d23, d21 + vmlal.s32 q4, d29, d20 + vmlal.s32 q8, d22, d21 + vmlal.s32 q8, d28, d20 + vmlal.s32 q5, d24, d20 + add r2, sp, #576 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q7, d18, d6 + vmlal.s32 q7, d26, d0 + add r2, sp, #656 + vld1.8 {d30-d31}, [r2, : 128] + vmlal.s32 q2, d30, d21 + vmlal.s32 q7, d19, d21 + vmlal.s32 q7, d27, d20 + add r2, sp, #624 + vld1.8 {d26-d27}, [r2, : 128] + vmlal.s32 q4, d25, d27 + vmlal.s32 q8, d29, d27 + vmlal.s32 q8, d25, d26 + vmlal.s32 q7, d28, d27 + vmlal.s32 q7, d29, d26 + add r2, sp, #608 + vld1.8 {d28-d29}, [r2, : 128] + vmlal.s32 q4, d24, d29 + vmlal.s32 q8, d23, d29 + vmlal.s32 q8, d24, d28 + vmlal.s32 q7, d22, d29 + vmlal.s32 q7, d23, d28 + add r2, sp, #608 + vst1.8 {d8-d9}, [r2, : 128] + add r2, sp, #560 + vld1.8 {d8-d9}, [r2, : 128] + vmlal.s32 q7, d24, d9 + vmlal.s32 q7, d25, d31 + vmull.s32 q1, d18, d2 + vmlal.s32 q1, d19, d1 + vmlal.s32 q1, d22, d0 + vmlal.s32 q1, d24, d27 + vmlal.s32 q1, d23, d20 + vmlal.s32 q1, d12, d7 + vmlal.s32 q1, d13, d6 + vmull.s32 q6, d18, d1 + vmlal.s32 q6, d19, d0 + vmlal.s32 q6, d23, d27 + vmlal.s32 q6, d22, d20 + vmlal.s32 q6, d24, d26 + vmull.s32 q0, d18, d0 + vmlal.s32 q0, d22, d27 + vmlal.s32 q0, d23, d26 + vmlal.s32 q0, d24, d31 + vmlal.s32 q0, d19, d20 + add r2, sp, #640 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q2, d18, d7 + vmlal.s32 q2, d19, d6 + vmlal.s32 q5, d18, d6 + vmlal.s32 q5, d19, d21 + vmlal.s32 q1, d18, d21 + vmlal.s32 q1, d19, d29 + vmlal.s32 q0, d18, d28 + vmlal.s32 q0, d19, d9 + vmlal.s32 q6, d18, d29 + vmlal.s32 q6, d19, d28 + add r2, sp, #592 + vld1.8 {d18-d19}, [r2, : 128] + add r2, sp, #512 + vld1.8 {d22-d23}, [r2, : 128] + vmlal.s32 q5, d19, d7 + vmlal.s32 q0, d18, d21 + vmlal.s32 q0, d19, d29 + vmlal.s32 q6, d18, d6 + add r2, sp, #528 + vld1.8 {d6-d7}, [r2, : 128] + vmlal.s32 q6, d19, d21 + add r2, sp, #576 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q0, d30, d8 + add r2, sp, #672 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q5, d30, d29 + add r2, sp, #608 + vld1.8 {d24-d25}, [r2, : 128] + vmlal.s32 q1, d30, d28 + vadd.i64 q13, q0, q11 + vadd.i64 q14, q5, q11 + vmlal.s32 q6, d30, d9 + vshr.s64 q4, q13, #26 + vshr.s64 q13, q14, #26 + vadd.i64 q7, q7, q4 + vshl.i64 q4, q4, #26 + vadd.i64 q14, q7, q3 + vadd.i64 q9, q9, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q15, q9, q3 + vsub.i64 q0, q0, q4 + vshr.s64 q4, q14, #25 + vsub.i64 q5, q5, q13 + vshr.s64 q13, q15, #25 + vadd.i64 q6, q6, q4 + vshl.i64 q4, q4, #25 + vadd.i64 q14, q6, q11 + vadd.i64 q2, q2, q13 + vsub.i64 q4, q7, q4 + vshr.s64 q7, q14, #26 + vshl.i64 q13, q13, #25 + vadd.i64 q14, q2, q11 + vadd.i64 q8, q8, q7 + vshl.i64 q7, q7, #26 + vadd.i64 q15, q8, q3 + vsub.i64 q9, q9, q13 + vshr.s64 q13, q14, #26 + vsub.i64 q6, q6, q7 + vshr.s64 q7, q15, #25 + vadd.i64 q10, q10, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q14, q10, q3 + vadd.i64 q1, q1, q7 + add r2, r3, #288 + vshl.i64 q7, q7, #25 + add r4, r3, #96 + vadd.i64 q15, q1, q11 + add r2, r2, #8 + vsub.i64 q2, q2, q13 + add r4, r4, #8 + vshr.s64 q13, q14, #25 + vsub.i64 q7, q8, q7 + vshr.s64 q8, q15, #26 + vadd.i64 q14, q13, q13 + vadd.i64 q12, q12, q8 + vtrn.32 d12, d14 + vshl.i64 q8, q8, #26 + vtrn.32 d13, d15 + vadd.i64 q3, q12, q3 + vadd.i64 q0, q0, q14 + vst1.8 d12, [r2, : 64]! + vshl.i64 q7, q13, #4 + vst1.8 d13, [r4, : 64]! + vsub.i64 q1, q1, q8 + vshr.s64 q3, q3, #25 + vadd.i64 q0, q0, q7 + vadd.i64 q5, q5, q3 + vshl.i64 q3, q3, #25 + vadd.i64 q6, q5, q11 + vadd.i64 q0, q0, q13 + vshl.i64 q7, q13, #25 + vadd.i64 q8, q0, q11 + vsub.i64 q3, q12, q3 + vshr.s64 q6, q6, #26 + vsub.i64 q7, q10, q7 + vtrn.32 d2, d6 + vshr.s64 q8, q8, #26 + vtrn.32 d3, d7 + vadd.i64 q3, q9, q6 + vst1.8 d2, [r2, : 64] + vshl.i64 q6, q6, #26 + vst1.8 d3, [r4, : 64] + vadd.i64 q1, q4, q8 + vtrn.32 d4, d14 + vshl.i64 q4, q8, #26 + vtrn.32 d5, d15 + vsub.i64 q5, q5, q6 + add r2, r2, #16 + vsub.i64 q0, q0, q4 + vst1.8 d4, [r2, : 64] + add r4, r4, #16 + vst1.8 d5, [r4, : 64] + vtrn.32 d10, d6 + vtrn.32 d11, d7 + sub r2, r2, #8 + sub r4, r4, #8 + vtrn.32 d0, d2 + vtrn.32 d1, d3 + vst1.8 d10, [r2, : 64] + vst1.8 d11, [r4, : 64] + sub r2, r2, #24 + sub r4, r4, #24 + vst1.8 d0, [r2, : 64] + vst1.8 d1, [r4, : 64] + add r2, sp, #544 + add r4, r3, #144 + add r5, r3, #192 + vld1.8 {d0-d1}, [r2, : 128] + vld1.8 {d2-d3}, [r4, : 128]! + vld1.8 {d4-d5}, [r5, : 128]! + vzip.i32 q1, q2 + vld1.8 {d6-d7}, [r4, : 128]! + vld1.8 {d8-d9}, [r5, : 128]! + vshl.i32 q5, q1, #1 + vzip.i32 q3, q4 + vshl.i32 q6, q2, #1 + vld1.8 {d14}, [r4, : 64] + vshl.i32 q8, q3, #1 + vld1.8 {d15}, [r5, : 64] + vshl.i32 q9, q4, #1 + vmul.i32 d21, d7, d1 + vtrn.32 d14, d15 + vmul.i32 q11, q4, q0 + vmul.i32 q0, q7, q0 + vmull.s32 q12, d2, d2 + vmlal.s32 q12, d11, d1 + vmlal.s32 q12, d12, d0 + vmlal.s32 q12, d13, d23 + vmlal.s32 q12, d16, d22 + vmlal.s32 q12, d7, d21 + vmull.s32 q10, d2, d11 + vmlal.s32 q10, d4, d1 + vmlal.s32 q10, d13, d0 + vmlal.s32 q10, d6, d23 + vmlal.s32 q10, d17, d22 + vmull.s32 q13, d10, d4 + vmlal.s32 q13, d11, d3 + vmlal.s32 q13, d13, d1 + vmlal.s32 q13, d16, d0 + vmlal.s32 q13, d17, d23 + vmlal.s32 q13, d8, d22 + vmull.s32 q1, d10, d5 + vmlal.s32 q1, d11, d4 + vmlal.s32 q1, d6, d1 + vmlal.s32 q1, d17, d0 + vmlal.s32 q1, d8, d23 + vmull.s32 q14, d10, d6 + vmlal.s32 q14, d11, d13 + vmlal.s32 q14, d4, d4 + vmlal.s32 q14, d17, d1 + vmlal.s32 q14, d18, d0 + vmlal.s32 q14, d9, d23 + vmull.s32 q11, d10, d7 + vmlal.s32 q11, d11, d6 + vmlal.s32 q11, d12, d5 + vmlal.s32 q11, d8, d1 + vmlal.s32 q11, d19, d0 + vmull.s32 q15, d10, d8 + vmlal.s32 q15, d11, d17 + vmlal.s32 q15, d12, d6 + vmlal.s32 q15, d13, d5 + vmlal.s32 q15, d19, d1 + vmlal.s32 q15, d14, d0 + vmull.s32 q2, d10, d9 + vmlal.s32 q2, d11, d8 + vmlal.s32 q2, d12, d7 + vmlal.s32 q2, d13, d6 + vmlal.s32 q2, d14, d1 + vmull.s32 q0, d15, d1 + vmlal.s32 q0, d10, d14 + vmlal.s32 q0, d11, d19 + vmlal.s32 q0, d12, d8 + vmlal.s32 q0, d13, d17 + vmlal.s32 q0, d6, d6 + add r2, sp, #512 + vld1.8 {d18-d19}, [r2, : 128] + vmull.s32 q3, d16, d7 + vmlal.s32 q3, d10, d15 + vmlal.s32 q3, d11, d14 + vmlal.s32 q3, d12, d9 + vmlal.s32 q3, d13, d8 + add r2, sp, #528 + vld1.8 {d8-d9}, [r2, : 128] + vadd.i64 q5, q12, q9 + vadd.i64 q6, q15, q9 + vshr.s64 q5, q5, #26 + vshr.s64 q6, q6, #26 + vadd.i64 q7, q10, q5 + vshl.i64 q5, q5, #26 + vadd.i64 q8, q7, q4 + vadd.i64 q2, q2, q6 + vshl.i64 q6, q6, #26 + vadd.i64 q10, q2, q4 + vsub.i64 q5, q12, q5 + vshr.s64 q8, q8, #25 + vsub.i64 q6, q15, q6 + vshr.s64 q10, q10, #25 + vadd.i64 q12, q13, q8 + vshl.i64 q8, q8, #25 + vadd.i64 q13, q12, q9 + vadd.i64 q0, q0, q10 + vsub.i64 q7, q7, q8 + vshr.s64 q8, q13, #26 + vshl.i64 q10, q10, #25 + vadd.i64 q13, q0, q9 + vadd.i64 q1, q1, q8 + vshl.i64 q8, q8, #26 + vadd.i64 q15, q1, q4 + vsub.i64 q2, q2, q10 + vshr.s64 q10, q13, #26 + vsub.i64 q8, q12, q8 + vshr.s64 q12, q15, #25 + vadd.i64 q3, q3, q10 + vshl.i64 q10, q10, #26 + vadd.i64 q13, q3, q4 + vadd.i64 q14, q14, q12 + add r2, r3, #144 + vshl.i64 q12, q12, #25 + add r4, r3, #192 + vadd.i64 q15, q14, q9 + add r2, r2, #8 + vsub.i64 q0, q0, q10 + add r4, r4, #8 + vshr.s64 q10, q13, #25 + vsub.i64 q1, q1, q12 + vshr.s64 q12, q15, #26 + vadd.i64 q13, q10, q10 + vadd.i64 q11, q11, q12 + vtrn.32 d16, d2 + vshl.i64 q12, q12, #26 + vtrn.32 d17, d3 + vadd.i64 q1, q11, q4 + vadd.i64 q4, q5, q13 + vst1.8 d16, [r2, : 64]! + vshl.i64 q5, q10, #4 + vst1.8 d17, [r4, : 64]! + vsub.i64 q8, q14, q12 + vshr.s64 q1, q1, #25 + vadd.i64 q4, q4, q5 + vadd.i64 q5, q6, q1 + vshl.i64 q1, q1, #25 + vadd.i64 q6, q5, q9 + vadd.i64 q4, q4, q10 + vshl.i64 q10, q10, #25 + vadd.i64 q9, q4, q9 + vsub.i64 q1, q11, q1 + vshr.s64 q6, q6, #26 + vsub.i64 q3, q3, q10 + vtrn.32 d16, d2 + vshr.s64 q9, q9, #26 + vtrn.32 d17, d3 + vadd.i64 q1, q2, q6 + vst1.8 d16, [r2, : 64] + vshl.i64 q2, q6, #26 + vst1.8 d17, [r4, : 64] + vadd.i64 q6, q7, q9 + vtrn.32 d0, d6 + vshl.i64 q7, q9, #26 + vtrn.32 d1, d7 + vsub.i64 q2, q5, q2 + add r2, r2, #16 + vsub.i64 q3, q4, q7 + vst1.8 d0, [r2, : 64] + add r4, r4, #16 + vst1.8 d1, [r4, : 64] + vtrn.32 d4, d2 + vtrn.32 d5, d3 + sub r2, r2, #8 + sub r4, r4, #8 + vtrn.32 d6, d12 + vtrn.32 d7, d13 + vst1.8 d4, [r2, : 64] + vst1.8 d5, [r4, : 64] + sub r2, r2, #24 + sub r4, r4, #24 + vst1.8 d6, [r2, : 64] + vst1.8 d7, [r4, : 64] + add r2, r3, #336 + add r4, r3, #288 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vadd.i32 q0, q0, q1 + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d4-d5}, [r4, : 128]! + vadd.i32 q1, q1, q2 + add r5, r3, #288 + vld1.8 {d4}, [r2, : 64] + vld1.8 {d6}, [r4, : 64] + vadd.i32 q2, q2, q3 + vst1.8 {d0-d1}, [r5, : 128]! + vst1.8 {d2-d3}, [r5, : 128]! + vst1.8 d4, [r5, : 64] + add r2, r3, #48 + add r4, r3, #144 + vld1.8 {d0-d1}, [r4, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vld1.8 {d4}, [r4, : 64] + add r4, r3, #288 + vld1.8 {d6-d7}, [r4, : 128]! + vtrn.32 q0, q3 + vld1.8 {d8-d9}, [r4, : 128]! + vshl.i32 q5, q0, #4 + vtrn.32 q1, q4 + vshl.i32 q6, q3, #4 + vadd.i32 q5, q5, q0 + vadd.i32 q6, q6, q3 + vshl.i32 q7, q1, #4 + vld1.8 {d5}, [r4, : 64] + vshl.i32 q8, q4, #4 + vtrn.32 d4, d5 + vadd.i32 q7, q7, q1 + vadd.i32 q8, q8, q4 + vld1.8 {d18-d19}, [r2, : 128]! + vshl.i32 q10, q2, #4 + vld1.8 {d22-d23}, [r2, : 128]! + vadd.i32 q10, q10, q2 + vld1.8 {d24}, [r2, : 64] + vadd.i32 q5, q5, q0 + add r2, r3, #240 + vld1.8 {d26-d27}, [r2, : 128]! + vadd.i32 q6, q6, q3 + vld1.8 {d28-d29}, [r2, : 128]! + vadd.i32 q8, q8, q4 + vld1.8 {d25}, [r2, : 64] + vadd.i32 q10, q10, q2 + vtrn.32 q9, q13 + vadd.i32 q7, q7, q1 + vadd.i32 q5, q5, q0 + vtrn.32 q11, q14 + vadd.i32 q6, q6, q3 + add r2, sp, #560 + vadd.i32 q10, q10, q2 + vtrn.32 d24, d25 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q6, q13, #1 + add r2, sp, #576 + vst1.8 {d20-d21}, [r2, : 128] + vshl.i32 q10, q14, #1 + add r2, sp, #592 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q15, q12, #1 + vadd.i32 q8, q8, q4 + vext.32 d10, d31, d30, #0 + vadd.i32 q7, q7, q1 + add r2, sp, #608 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q8, d18, d5 + vmlal.s32 q8, d26, d4 + vmlal.s32 q8, d19, d9 + vmlal.s32 q8, d27, d3 + vmlal.s32 q8, d22, d8 + vmlal.s32 q8, d28, d2 + vmlal.s32 q8, d23, d7 + vmlal.s32 q8, d29, d1 + vmlal.s32 q8, d24, d6 + vmlal.s32 q8, d25, d0 + add r2, sp, #624 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q2, d18, d4 + vmlal.s32 q2, d12, d9 + vmlal.s32 q2, d13, d8 + vmlal.s32 q2, d19, d3 + vmlal.s32 q2, d22, d2 + vmlal.s32 q2, d23, d1 + vmlal.s32 q2, d24, d0 + add r2, sp, #640 + vst1.8 {d20-d21}, [r2, : 128] + vmull.s32 q7, d18, d9 + vmlal.s32 q7, d26, d3 + vmlal.s32 q7, d19, d8 + vmlal.s32 q7, d27, d2 + vmlal.s32 q7, d22, d7 + vmlal.s32 q7, d28, d1 + vmlal.s32 q7, d23, d6 + vmlal.s32 q7, d29, d0 + add r2, sp, #656 + vst1.8 {d10-d11}, [r2, : 128] + vmull.s32 q5, d18, d3 + vmlal.s32 q5, d19, d2 + vmlal.s32 q5, d22, d1 + vmlal.s32 q5, d23, d0 + vmlal.s32 q5, d12, d8 + add r2, sp, #672 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q4, d18, d8 + vmlal.s32 q4, d26, d2 + vmlal.s32 q4, d19, d7 + vmlal.s32 q4, d27, d1 + vmlal.s32 q4, d22, d6 + vmlal.s32 q4, d28, d0 + vmull.s32 q8, d18, d7 + vmlal.s32 q8, d26, d1 + vmlal.s32 q8, d19, d6 + vmlal.s32 q8, d27, d0 + add r2, sp, #576 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q7, d24, d21 + vmlal.s32 q7, d25, d20 + vmlal.s32 q4, d23, d21 + vmlal.s32 q4, d29, d20 + vmlal.s32 q8, d22, d21 + vmlal.s32 q8, d28, d20 + vmlal.s32 q5, d24, d20 + add r2, sp, #576 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q7, d18, d6 + vmlal.s32 q7, d26, d0 + add r2, sp, #656 + vld1.8 {d30-d31}, [r2, : 128] + vmlal.s32 q2, d30, d21 + vmlal.s32 q7, d19, d21 + vmlal.s32 q7, d27, d20 + add r2, sp, #624 + vld1.8 {d26-d27}, [r2, : 128] + vmlal.s32 q4, d25, d27 + vmlal.s32 q8, d29, d27 + vmlal.s32 q8, d25, d26 + vmlal.s32 q7, d28, d27 + vmlal.s32 q7, d29, d26 + add r2, sp, #608 + vld1.8 {d28-d29}, [r2, : 128] + vmlal.s32 q4, d24, d29 + vmlal.s32 q8, d23, d29 + vmlal.s32 q8, d24, d28 + vmlal.s32 q7, d22, d29 + vmlal.s32 q7, d23, d28 + add r2, sp, #608 + vst1.8 {d8-d9}, [r2, : 128] + add r2, sp, #560 + vld1.8 {d8-d9}, [r2, : 128] + vmlal.s32 q7, d24, d9 + vmlal.s32 q7, d25, d31 + vmull.s32 q1, d18, d2 + vmlal.s32 q1, d19, d1 + vmlal.s32 q1, d22, d0 + vmlal.s32 q1, d24, d27 + vmlal.s32 q1, d23, d20 + vmlal.s32 q1, d12, d7 + vmlal.s32 q1, d13, d6 + vmull.s32 q6, d18, d1 + vmlal.s32 q6, d19, d0 + vmlal.s32 q6, d23, d27 + vmlal.s32 q6, d22, d20 + vmlal.s32 q6, d24, d26 + vmull.s32 q0, d18, d0 + vmlal.s32 q0, d22, d27 + vmlal.s32 q0, d23, d26 + vmlal.s32 q0, d24, d31 + vmlal.s32 q0, d19, d20 + add r2, sp, #640 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q2, d18, d7 + vmlal.s32 q2, d19, d6 + vmlal.s32 q5, d18, d6 + vmlal.s32 q5, d19, d21 + vmlal.s32 q1, d18, d21 + vmlal.s32 q1, d19, d29 + vmlal.s32 q0, d18, d28 + vmlal.s32 q0, d19, d9 + vmlal.s32 q6, d18, d29 + vmlal.s32 q6, d19, d28 + add r2, sp, #592 + vld1.8 {d18-d19}, [r2, : 128] + add r2, sp, #512 + vld1.8 {d22-d23}, [r2, : 128] + vmlal.s32 q5, d19, d7 + vmlal.s32 q0, d18, d21 + vmlal.s32 q0, d19, d29 + vmlal.s32 q6, d18, d6 + add r2, sp, #528 + vld1.8 {d6-d7}, [r2, : 128] + vmlal.s32 q6, d19, d21 + add r2, sp, #576 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q0, d30, d8 + add r2, sp, #672 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q5, d30, d29 + add r2, sp, #608 + vld1.8 {d24-d25}, [r2, : 128] + vmlal.s32 q1, d30, d28 + vadd.i64 q13, q0, q11 + vadd.i64 q14, q5, q11 + vmlal.s32 q6, d30, d9 + vshr.s64 q4, q13, #26 + vshr.s64 q13, q14, #26 + vadd.i64 q7, q7, q4 + vshl.i64 q4, q4, #26 + vadd.i64 q14, q7, q3 + vadd.i64 q9, q9, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q15, q9, q3 + vsub.i64 q0, q0, q4 + vshr.s64 q4, q14, #25 + vsub.i64 q5, q5, q13 + vshr.s64 q13, q15, #25 + vadd.i64 q6, q6, q4 + vshl.i64 q4, q4, #25 + vadd.i64 q14, q6, q11 + vadd.i64 q2, q2, q13 + vsub.i64 q4, q7, q4 + vshr.s64 q7, q14, #26 + vshl.i64 q13, q13, #25 + vadd.i64 q14, q2, q11 + vadd.i64 q8, q8, q7 + vshl.i64 q7, q7, #26 + vadd.i64 q15, q8, q3 + vsub.i64 q9, q9, q13 + vshr.s64 q13, q14, #26 + vsub.i64 q6, q6, q7 + vshr.s64 q7, q15, #25 + vadd.i64 q10, q10, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q14, q10, q3 + vadd.i64 q1, q1, q7 + add r2, r3, #240 + vshl.i64 q7, q7, #25 + add r4, r3, #144 + vadd.i64 q15, q1, q11 + add r2, r2, #8 + vsub.i64 q2, q2, q13 + add r4, r4, #8 + vshr.s64 q13, q14, #25 + vsub.i64 q7, q8, q7 + vshr.s64 q8, q15, #26 + vadd.i64 q14, q13, q13 + vadd.i64 q12, q12, q8 + vtrn.32 d12, d14 + vshl.i64 q8, q8, #26 + vtrn.32 d13, d15 + vadd.i64 q3, q12, q3 + vadd.i64 q0, q0, q14 + vst1.8 d12, [r2, : 64]! + vshl.i64 q7, q13, #4 + vst1.8 d13, [r4, : 64]! + vsub.i64 q1, q1, q8 + vshr.s64 q3, q3, #25 + vadd.i64 q0, q0, q7 + vadd.i64 q5, q5, q3 + vshl.i64 q3, q3, #25 + vadd.i64 q6, q5, q11 + vadd.i64 q0, q0, q13 + vshl.i64 q7, q13, #25 + vadd.i64 q8, q0, q11 + vsub.i64 q3, q12, q3 + vshr.s64 q6, q6, #26 + vsub.i64 q7, q10, q7 + vtrn.32 d2, d6 + vshr.s64 q8, q8, #26 + vtrn.32 d3, d7 + vadd.i64 q3, q9, q6 + vst1.8 d2, [r2, : 64] + vshl.i64 q6, q6, #26 + vst1.8 d3, [r4, : 64] + vadd.i64 q1, q4, q8 + vtrn.32 d4, d14 + vshl.i64 q4, q8, #26 + vtrn.32 d5, d15 + vsub.i64 q5, q5, q6 + add r2, r2, #16 + vsub.i64 q0, q0, q4 + vst1.8 d4, [r2, : 64] + add r4, r4, #16 + vst1.8 d5, [r4, : 64] + vtrn.32 d10, d6 + vtrn.32 d11, d7 + sub r2, r2, #8 + sub r4, r4, #8 + vtrn.32 d0, d2 + vtrn.32 d1, d3 + vst1.8 d10, [r2, : 64] + vst1.8 d11, [r4, : 64] + sub r2, r2, #24 + sub r4, r4, #24 + vst1.8 d0, [r2, : 64] + vst1.8 d1, [r4, : 64] + ldr r2, [sp, #488] + ldr r4, [sp, #492] + subs r5, r2, #1 + bge ._mainloop + add r1, r3, #144 + add r2, r3, #336 + vld1.8 {d0-d1}, [r1, : 128]! + vld1.8 {d2-d3}, [r1, : 128]! + vld1.8 {d4}, [r1, : 64] + vst1.8 {d0-d1}, [r2, : 128]! + vst1.8 {d2-d3}, [r2, : 128]! + vst1.8 d4, [r2, : 64] + ldr r1, =0 +._invertloop: + add r2, r3, #144 + ldr r4, =0 + ldr r5, =2 + cmp r1, #1 + ldreq r5, =1 + addeq r2, r3, #336 + addeq r4, r3, #48 + cmp r1, #2 + ldreq r5, =1 + addeq r2, r3, #48 + cmp r1, #3 + ldreq r5, =5 + addeq r4, r3, #336 + cmp r1, #4 + ldreq r5, =10 + cmp r1, #5 + ldreq r5, =20 + cmp r1, #6 + ldreq r5, =10 + addeq r2, r3, #336 + addeq r4, r3, #336 + cmp r1, #7 + ldreq r5, =50 + cmp r1, #8 + ldreq r5, =100 + cmp r1, #9 + ldreq r5, =50 + addeq r2, r3, #336 + cmp r1, #10 + ldreq r5, =5 + addeq r2, r3, #48 + cmp r1, #11 + ldreq r5, =0 + addeq r2, r3, #96 + add r6, r3, #144 + add r7, r3, #288 + vld1.8 {d0-d1}, [r6, : 128]! + vld1.8 {d2-d3}, [r6, : 128]! + vld1.8 {d4}, [r6, : 64] + vst1.8 {d0-d1}, [r7, : 128]! + vst1.8 {d2-d3}, [r7, : 128]! + vst1.8 d4, [r7, : 64] + cmp r5, #0 + beq ._skipsquaringloop +._squaringloop: + add r6, r3, #288 + add r7, r3, #288 + add r8, r3, #288 + vmov.i32 q0, #19 + vmov.i32 q1, #0 + vmov.i32 q2, #1 + vzip.i32 q1, q2 + vld1.8 {d4-d5}, [r7, : 128]! + vld1.8 {d6-d7}, [r7, : 128]! + vld1.8 {d9}, [r7, : 64] + vld1.8 {d10-d11}, [r6, : 128]! + add r7, sp, #416 + vld1.8 {d12-d13}, [r6, : 128]! + vmul.i32 q7, q2, q0 + vld1.8 {d8}, [r6, : 64] + vext.32 d17, d11, d10, #1 + vmul.i32 q9, q3, q0 + vext.32 d16, d10, d8, #1 + vshl.u32 q10, q5, q1 + vext.32 d22, d14, d4, #1 + vext.32 d24, d18, d6, #1 + vshl.u32 q13, q6, q1 + vshl.u32 d28, d8, d2 + vrev64.i32 d22, d22 + vmul.i32 d1, d9, d1 + vrev64.i32 d24, d24 + vext.32 d29, d8, d13, #1 + vext.32 d0, d1, d9, #1 + vrev64.i32 d0, d0 + vext.32 d2, d9, d1, #1 + vext.32 d23, d15, d5, #1 + vmull.s32 q4, d20, d4 + vrev64.i32 d23, d23 + vmlal.s32 q4, d21, d1 + vrev64.i32 d2, d2 + vmlal.s32 q4, d26, d19 + vext.32 d3, d5, d15, #1 + vmlal.s32 q4, d27, d18 + vrev64.i32 d3, d3 + vmlal.s32 q4, d28, d15 + vext.32 d14, d12, d11, #1 + vmull.s32 q5, d16, d23 + vext.32 d15, d13, d12, #1 + vmlal.s32 q5, d17, d4 + vst1.8 d8, [r7, : 64]! + vmlal.s32 q5, d14, d1 + vext.32 d12, d9, d8, #0 + vmlal.s32 q5, d15, d19 + vmov.i64 d13, #0 + vmlal.s32 q5, d29, d18 + vext.32 d25, d19, d7, #1 + vmlal.s32 q6, d20, d5 + vrev64.i32 d25, d25 + vmlal.s32 q6, d21, d4 + vst1.8 d11, [r7, : 64]! + vmlal.s32 q6, d26, d1 + vext.32 d9, d10, d10, #0 + vmlal.s32 q6, d27, d19 + vmov.i64 d8, #0 + vmlal.s32 q6, d28, d18 + vmlal.s32 q4, d16, d24 + vmlal.s32 q4, d17, d5 + vmlal.s32 q4, d14, d4 + vst1.8 d12, [r7, : 64]! + vmlal.s32 q4, d15, d1 + vext.32 d10, d13, d12, #0 + vmlal.s32 q4, d29, d19 + vmov.i64 d11, #0 + vmlal.s32 q5, d20, d6 + vmlal.s32 q5, d21, d5 + vmlal.s32 q5, d26, d4 + vext.32 d13, d8, d8, #0 + vmlal.s32 q5, d27, d1 + vmov.i64 d12, #0 + vmlal.s32 q5, d28, d19 + vst1.8 d9, [r7, : 64]! + vmlal.s32 q6, d16, d25 + vmlal.s32 q6, d17, d6 + vst1.8 d10, [r7, : 64] + vmlal.s32 q6, d14, d5 + vext.32 d8, d11, d10, #0 + vmlal.s32 q6, d15, d4 + vmov.i64 d9, #0 + vmlal.s32 q6, d29, d1 + vmlal.s32 q4, d20, d7 + vmlal.s32 q4, d21, d6 + vmlal.s32 q4, d26, d5 + vext.32 d11, d12, d12, #0 + vmlal.s32 q4, d27, d4 + vmov.i64 d10, #0 + vmlal.s32 q4, d28, d1 + vmlal.s32 q5, d16, d0 + sub r6, r7, #32 + vmlal.s32 q5, d17, d7 + vmlal.s32 q5, d14, d6 + vext.32 d30, d9, d8, #0 + vmlal.s32 q5, d15, d5 + vld1.8 {d31}, [r6, : 64]! + vmlal.s32 q5, d29, d4 + vmlal.s32 q15, d20, d0 + vext.32 d0, d6, d18, #1 + vmlal.s32 q15, d21, d25 + vrev64.i32 d0, d0 + vmlal.s32 q15, d26, d24 + vext.32 d1, d7, d19, #1 + vext.32 d7, d10, d10, #0 + vmlal.s32 q15, d27, d23 + vrev64.i32 d1, d1 + vld1.8 {d6}, [r6, : 64] + vmlal.s32 q15, d28, d22 + vmlal.s32 q3, d16, d4 + add r6, r6, #24 + vmlal.s32 q3, d17, d2 + vext.32 d4, d31, d30, #0 + vmov d17, d11 + vmlal.s32 q3, d14, d1 + vext.32 d11, d13, d13, #0 + vext.32 d13, d30, d30, #0 + vmlal.s32 q3, d15, d0 + vext.32 d1, d8, d8, #0 + vmlal.s32 q3, d29, d3 + vld1.8 {d5}, [r6, : 64] + sub r6, r6, #16 + vext.32 d10, d6, d6, #0 + vmov.i32 q1, #0xffffffff + vshl.i64 q4, q1, #25 + add r7, sp, #512 + vld1.8 {d14-d15}, [r7, : 128] + vadd.i64 q9, q2, q7 + vshl.i64 q1, q1, #26 + vshr.s64 q10, q9, #26 + vld1.8 {d0}, [r6, : 64]! + vadd.i64 q5, q5, q10 + vand q9, q9, q1 + vld1.8 {d16}, [r6, : 64]! + add r6, sp, #528 + vld1.8 {d20-d21}, [r6, : 128] + vadd.i64 q11, q5, q10 + vsub.i64 q2, q2, q9 + vshr.s64 q9, q11, #25 + vext.32 d12, d5, d4, #0 + vand q11, q11, q4 + vadd.i64 q0, q0, q9 + vmov d19, d7 + vadd.i64 q3, q0, q7 + vsub.i64 q5, q5, q11 + vshr.s64 q11, q3, #26 + vext.32 d18, d11, d10, #0 + vand q3, q3, q1 + vadd.i64 q8, q8, q11 + vadd.i64 q11, q8, q10 + vsub.i64 q0, q0, q3 + vshr.s64 q3, q11, #25 + vand q11, q11, q4 + vadd.i64 q3, q6, q3 + vadd.i64 q6, q3, q7 + vsub.i64 q8, q8, q11 + vshr.s64 q11, q6, #26 + vand q6, q6, q1 + vadd.i64 q9, q9, q11 + vadd.i64 d25, d19, d21 + vsub.i64 q3, q3, q6 + vshr.s64 d23, d25, #25 + vand q4, q12, q4 + vadd.i64 d21, d23, d23 + vshl.i64 d25, d23, #4 + vadd.i64 d21, d21, d23 + vadd.i64 d25, d25, d21 + vadd.i64 d4, d4, d25 + vzip.i32 q0, q8 + vadd.i64 d12, d4, d14 + add r6, r8, #8 + vst1.8 d0, [r6, : 64] + vsub.i64 d19, d19, d9 + add r6, r6, #16 + vst1.8 d16, [r6, : 64] + vshr.s64 d22, d12, #26 + vand q0, q6, q1 + vadd.i64 d10, d10, d22 + vzip.i32 q3, q9 + vsub.i64 d4, d4, d0 + sub r6, r6, #8 + vst1.8 d6, [r6, : 64] + add r6, r6, #16 + vst1.8 d18, [r6, : 64] + vzip.i32 q2, q5 + sub r6, r6, #32 + vst1.8 d4, [r6, : 64] + subs r5, r5, #1 + bhi ._squaringloop +._skipsquaringloop: + mov r2, r2 + add r5, r3, #288 + add r6, r3, #144 + vmov.i32 q0, #19 + vmov.i32 q1, #0 + vmov.i32 q2, #1 + vzip.i32 q1, q2 + vld1.8 {d4-d5}, [r5, : 128]! + vld1.8 {d6-d7}, [r5, : 128]! + vld1.8 {d9}, [r5, : 64] + vld1.8 {d10-d11}, [r2, : 128]! + add r5, sp, #416 + vld1.8 {d12-d13}, [r2, : 128]! + vmul.i32 q7, q2, q0 + vld1.8 {d8}, [r2, : 64] + vext.32 d17, d11, d10, #1 + vmul.i32 q9, q3, q0 + vext.32 d16, d10, d8, #1 + vshl.u32 q10, q5, q1 + vext.32 d22, d14, d4, #1 + vext.32 d24, d18, d6, #1 + vshl.u32 q13, q6, q1 + vshl.u32 d28, d8, d2 + vrev64.i32 d22, d22 + vmul.i32 d1, d9, d1 + vrev64.i32 d24, d24 + vext.32 d29, d8, d13, #1 + vext.32 d0, d1, d9, #1 + vrev64.i32 d0, d0 + vext.32 d2, d9, d1, #1 + vext.32 d23, d15, d5, #1 + vmull.s32 q4, d20, d4 + vrev64.i32 d23, d23 + vmlal.s32 q4, d21, d1 + vrev64.i32 d2, d2 + vmlal.s32 q4, d26, d19 + vext.32 d3, d5, d15, #1 + vmlal.s32 q4, d27, d18 + vrev64.i32 d3, d3 + vmlal.s32 q4, d28, d15 + vext.32 d14, d12, d11, #1 + vmull.s32 q5, d16, d23 + vext.32 d15, d13, d12, #1 + vmlal.s32 q5, d17, d4 + vst1.8 d8, [r5, : 64]! + vmlal.s32 q5, d14, d1 + vext.32 d12, d9, d8, #0 + vmlal.s32 q5, d15, d19 + vmov.i64 d13, #0 + vmlal.s32 q5, d29, d18 + vext.32 d25, d19, d7, #1 + vmlal.s32 q6, d20, d5 + vrev64.i32 d25, d25 + vmlal.s32 q6, d21, d4 + vst1.8 d11, [r5, : 64]! + vmlal.s32 q6, d26, d1 + vext.32 d9, d10, d10, #0 + vmlal.s32 q6, d27, d19 + vmov.i64 d8, #0 + vmlal.s32 q6, d28, d18 + vmlal.s32 q4, d16, d24 + vmlal.s32 q4, d17, d5 + vmlal.s32 q4, d14, d4 + vst1.8 d12, [r5, : 64]! + vmlal.s32 q4, d15, d1 + vext.32 d10, d13, d12, #0 + vmlal.s32 q4, d29, d19 + vmov.i64 d11, #0 + vmlal.s32 q5, d20, d6 + vmlal.s32 q5, d21, d5 + vmlal.s32 q5, d26, d4 + vext.32 d13, d8, d8, #0 + vmlal.s32 q5, d27, d1 + vmov.i64 d12, #0 + vmlal.s32 q5, d28, d19 + vst1.8 d9, [r5, : 64]! + vmlal.s32 q6, d16, d25 + vmlal.s32 q6, d17, d6 + vst1.8 d10, [r5, : 64] + vmlal.s32 q6, d14, d5 + vext.32 d8, d11, d10, #0 + vmlal.s32 q6, d15, d4 + vmov.i64 d9, #0 + vmlal.s32 q6, d29, d1 + vmlal.s32 q4, d20, d7 + vmlal.s32 q4, d21, d6 + vmlal.s32 q4, d26, d5 + vext.32 d11, d12, d12, #0 + vmlal.s32 q4, d27, d4 + vmov.i64 d10, #0 + vmlal.s32 q4, d28, d1 + vmlal.s32 q5, d16, d0 + sub r2, r5, #32 + vmlal.s32 q5, d17, d7 + vmlal.s32 q5, d14, d6 + vext.32 d30, d9, d8, #0 + vmlal.s32 q5, d15, d5 + vld1.8 {d31}, [r2, : 64]! + vmlal.s32 q5, d29, d4 + vmlal.s32 q15, d20, d0 + vext.32 d0, d6, d18, #1 + vmlal.s32 q15, d21, d25 + vrev64.i32 d0, d0 + vmlal.s32 q15, d26, d24 + vext.32 d1, d7, d19, #1 + vext.32 d7, d10, d10, #0 + vmlal.s32 q15, d27, d23 + vrev64.i32 d1, d1 + vld1.8 {d6}, [r2, : 64] + vmlal.s32 q15, d28, d22 + vmlal.s32 q3, d16, d4 + add r2, r2, #24 + vmlal.s32 q3, d17, d2 + vext.32 d4, d31, d30, #0 + vmov d17, d11 + vmlal.s32 q3, d14, d1 + vext.32 d11, d13, d13, #0 + vext.32 d13, d30, d30, #0 + vmlal.s32 q3, d15, d0 + vext.32 d1, d8, d8, #0 + vmlal.s32 q3, d29, d3 + vld1.8 {d5}, [r2, : 64] + sub r2, r2, #16 + vext.32 d10, d6, d6, #0 + vmov.i32 q1, #0xffffffff + vshl.i64 q4, q1, #25 + add r5, sp, #512 + vld1.8 {d14-d15}, [r5, : 128] + vadd.i64 q9, q2, q7 + vshl.i64 q1, q1, #26 + vshr.s64 q10, q9, #26 + vld1.8 {d0}, [r2, : 64]! + vadd.i64 q5, q5, q10 + vand q9, q9, q1 + vld1.8 {d16}, [r2, : 64]! + add r2, sp, #528 + vld1.8 {d20-d21}, [r2, : 128] + vadd.i64 q11, q5, q10 + vsub.i64 q2, q2, q9 + vshr.s64 q9, q11, #25 + vext.32 d12, d5, d4, #0 + vand q11, q11, q4 + vadd.i64 q0, q0, q9 + vmov d19, d7 + vadd.i64 q3, q0, q7 + vsub.i64 q5, q5, q11 + vshr.s64 q11, q3, #26 + vext.32 d18, d11, d10, #0 + vand q3, q3, q1 + vadd.i64 q8, q8, q11 + vadd.i64 q11, q8, q10 + vsub.i64 q0, q0, q3 + vshr.s64 q3, q11, #25 + vand q11, q11, q4 + vadd.i64 q3, q6, q3 + vadd.i64 q6, q3, q7 + vsub.i64 q8, q8, q11 + vshr.s64 q11, q6, #26 + vand q6, q6, q1 + vadd.i64 q9, q9, q11 + vadd.i64 d25, d19, d21 + vsub.i64 q3, q3, q6 + vshr.s64 d23, d25, #25 + vand q4, q12, q4 + vadd.i64 d21, d23, d23 + vshl.i64 d25, d23, #4 + vadd.i64 d21, d21, d23 + vadd.i64 d25, d25, d21 + vadd.i64 d4, d4, d25 + vzip.i32 q0, q8 + vadd.i64 d12, d4, d14 + add r2, r6, #8 + vst1.8 d0, [r2, : 64] + vsub.i64 d19, d19, d9 + add r2, r2, #16 + vst1.8 d16, [r2, : 64] + vshr.s64 d22, d12, #26 + vand q0, q6, q1 + vadd.i64 d10, d10, d22 + vzip.i32 q3, q9 + vsub.i64 d4, d4, d0 + sub r2, r2, #8 + vst1.8 d6, [r2, : 64] + add r2, r2, #16 + vst1.8 d18, [r2, : 64] + vzip.i32 q2, q5 + sub r2, r2, #32 + vst1.8 d4, [r2, : 64] + cmp r4, #0 + beq ._skippostcopy + add r2, r3, #144 + mov r4, r4 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d4}, [r2, : 64] + vst1.8 {d0-d1}, [r4, : 128]! + vst1.8 {d2-d3}, [r4, : 128]! + vst1.8 d4, [r4, : 64] +._skippostcopy: + cmp r1, #1 + bne ._skipfinalcopy + add r2, r3, #288 + add r4, r3, #144 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d4}, [r2, : 64] + vst1.8 {d0-d1}, [r4, : 128]! + vst1.8 {d2-d3}, [r4, : 128]! + vst1.8 d4, [r4, : 64] +._skipfinalcopy: + add r1, r1, #1 + cmp r1, #12 + blo ._invertloop + add r1, r3, #144 + ldr r2, [r1], #4 + ldr r3, [r1], #4 + ldr r4, [r1], #4 + ldr r5, [r1], #4 + ldr r6, [r1], #4 + ldr r7, [r1], #4 + ldr r8, [r1], #4 + ldr r9, [r1], #4 + ldr r10, [r1], #4 + ldr r1, [r1] + add r11, r1, r1, LSL #4 + add r11, r11, r1, LSL #1 + add r11, r11, #16777216 + mov r11, r11, ASR #25 + add r11, r11, r2 + mov r11, r11, ASR #26 + add r11, r11, r3 + mov r11, r11, ASR #25 + add r11, r11, r4 + mov r11, r11, ASR #26 + add r11, r11, r5 + mov r11, r11, ASR #25 + add r11, r11, r6 + mov r11, r11, ASR #26 + add r11, r11, r7 + mov r11, r11, ASR #25 + add r11, r11, r8 + mov r11, r11, ASR #26 + add r11, r11, r9 + mov r11, r11, ASR #25 + add r11, r11, r10 + mov r11, r11, ASR #26 + add r11, r11, r1 + mov r11, r11, ASR #25 + add r2, r2, r11 + add r2, r2, r11, LSL #1 + add r2, r2, r11, LSL #4 + mov r11, r2, ASR #26 + add r3, r3, r11 + sub r2, r2, r11, LSL #26 + mov r11, r3, ASR #25 + add r4, r4, r11 + sub r3, r3, r11, LSL #25 + mov r11, r4, ASR #26 + add r5, r5, r11 + sub r4, r4, r11, LSL #26 + mov r11, r5, ASR #25 + add r6, r6, r11 + sub r5, r5, r11, LSL #25 + mov r11, r6, ASR #26 + add r7, r7, r11 + sub r6, r6, r11, LSL #26 + mov r11, r7, ASR #25 + add r8, r8, r11 + sub r7, r7, r11, LSL #25 + mov r11, r8, ASR #26 + add r9, r9, r11 + sub r8, r8, r11, LSL #26 + mov r11, r9, ASR #25 + add r10, r10, r11 + sub r9, r9, r11, LSL #25 + mov r11, r10, ASR #26 + add r1, r1, r11 + sub r10, r10, r11, LSL #26 + mov r11, r1, ASR #25 + sub r1, r1, r11, LSL #25 + add r2, r2, r3, LSL #26 + mov r3, r3, LSR #6 + add r3, r3, r4, LSL #19 + mov r4, r4, LSR #13 + add r4, r4, r5, LSL #13 + mov r5, r5, LSR #19 + add r5, r5, r6, LSL #6 + add r6, r7, r8, LSL #25 + mov r7, r8, LSR #7 + add r7, r7, r9, LSL #19 + mov r8, r9, LSR #13 + add r8, r8, r10, LSL #12 + mov r9, r10, LSR #20 + add r1, r9, r1, LSL #6 + str r2, [r0], #4 + str r3, [r0], #4 + str r4, [r0], #4 + str r5, [r0], #4 + str r6, [r0], #4 + str r7, [r0], #4 + str r8, [r0], #4 + str r1, [r0] + ldrd r4, [sp, #0] + ldrd r6, [sp, #8] + ldrd r8, [sp, #16] + ldrd r10, [sp, #24] + ldr r12, [sp, #480] + ldr r14, [sp, #484] + ldr r0, =0 + mov sp, r12 + vpop {q4, q5, q6, q7} + bx lr
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 32/34 crypto: arm/Curve25519 - wire up NEON implementation
In reply to Ard BiesheuvelFrom: "Jason A. Donenfeld" <[email protected]>
This ports the SUPERCOP implementation for usage in kernel space. In addition to the usual header, macro, and style changes required for kernel space, it makes a few small changes to the code:
- The stack alignment is relaxed to 16 bytes.
- Superfluous mov statements have been removed.
- ldr for constants has been replaced with movw.
- ldreq has been replaced with moveq.
- The str epilogue has been made more idiomatic.
- SIMD registers are not pushed and popped at the beginning and end.
- The prologue and epilogue have been made idiomatic.
- A hole has been removed from the stack, saving 32 bytes.
- We write-back the base register whenever possible for vld1.8.
- Some multiplications have been reordered for better A7 performance.
There are more opportunities for cleanup, since this code is from qhasm, which doesn't always do the most opportune thing. But even prior to extensive hand optimizations, this code delivers significant performance improvements (given in get_cycles() per call):
----------- ------------- | generic C | this commit | ------------ ----------- ------------- | Cortex-A7 | 49136 | 22395 | ------------ ----------- ------------- | Cortex-A17 | 17326 | 4983 | ------------ ----------- -------------
[ardb: - move to arch/arm/crypto
- wire into lib/crypto framework
- implement crypto API KPP hooks ]
--- arch/arm/crypto/Kconfig | 6 + arch/arm/crypto/Makefile | 2 + arch/arm/crypto/curve25519-core.S | 347 +++++++++----------- arch/arm/crypto/curve25519-glue.c | 127 +++++++ 4 files changed, 287 insertions(+), 195 deletions(-)
arch/arm/crypto/Kconfig +6 -0
@@ -142,4 +142,10 @@ config CRYPTO_NHPOLY1305_NEON depends on KERNEL_MODE_NEON select CRYPTO_NHPOLY1305 +config CRYPTO_CURVE25519_NEON + tristate "NEON accelerated Curve25519 scalar multiplication library" + depends on KERNEL_MODE_NEON + select CRYPTO_LIB_CURVE25519_GENERIC + select CRYPTO_ARCH_HAVE_LIB_CURVE25519 + endif
arch/arm/crypto/Makefile +2 -0
@@ -12,6 +12,7 @@ obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha-neon.o obj-$(CONFIG_CRYPTO_POLY1305_ARM) += poly1305-arm.o obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) += nhpoly1305-neon.o +obj-$(CONFIG_CRYPTO_CURVE25519_NEON) += curve25519-neon.o obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o obj-$(CONFIG_CRYPTO_SHA1_ARM_CE) += sha1-arm-ce.o
@@ -38,6 +39,7 @@ chacha-neon-y := chacha-scalar-core.o chacha-glue.o chacha-neon-$(CONFIG_KERNEL_MODE_NEON) += chacha-neon-core.o poly1305-arm-y := poly1305-core.o poly1305-glue.o nhpoly1305-neon-y := nh-neon-core.o nhpoly1305-neon-glue.o +curve25519-neon-y := curve25519-core.o curve25519-glue.o ifdef REGENERATE_ARM_CRYPTO quiet_cmd_perl = PERL $@
arch/arm/crypto/curve25519-core.S +152 -195
803 lines not on this page · raw message
arch/arm/crypto/curve25519-glue.c new file +127 -0
127 lines not on this page · raw message
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerPATCH v5 33/34 crypto: chacha20poly1305 - import construction and selftest from Zinc
In reply to Ard BiesheuvelThis incorporates the chacha20poly1305 from the Zinc library, retaining the library interface, but replacing the implementation with calls into the code that already existed in the kernel's crypto API.
Note that this library API does not implement RFC7539 fully, given that it is limited to 64-bit nonces. (The 96-bit nonce version that was part of the selftest only has been removed, along with the 96-bit nonce test vectors that only tested the selftest but not the actual library itself)
--- include/crypto/chacha20poly1305.h | 37 + lib/crypto/Kconfig | 7 + lib/crypto/Makefile | 4 + lib/crypto/chacha20poly1305-selftest.c | 7348 ++++++++++++++++++++ lib/crypto/chacha20poly1305.c | 219 + 5 files changed, 7615 insertions(+)
include/crypto/chacha20poly1305.h new file +37 -0
@@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef __CHACHA20POLY1305_H +#define __CHACHA20POLY1305_H + +#include <linux/types.h> + +enum chacha20poly1305_lengths { +XCHACHA20POLY1305_NONCE_SIZE =24 , +CHACHA20POLY1305_KEY_SIZE =32 , +CHACHA20POLY1305_AUTHTAG_SIZE =16 +}; + +void chacha20poly1305_encrypt (u8 *dst,const u8 *src,const size_t src_len, +const u8 *ad,const size_t ad_len, +const u64 nonce, +const u8 key[CHACHA20POLY1305_KEY_SIZE ]); + +bool __must_check +chacha20poly1305_decrypt (u8 *dst,const u8 *src,const size_t src_len, +const u8 *ad,const size_t ad_len,const u64 nonce, +const u8 key[CHACHA20POLY1305_KEY_SIZE ]); + +void xchacha20poly1305_encrypt (u8 *dst,const u8 *src,const size_t src_len, +const u8 *ad,const size_t ad_len, +const u8 nonce[XCHACHA20POLY1305_NONCE_SIZE ], +const u8 key[CHACHA20POLY1305_KEY_SIZE ]); + +bool __must_check xchacha20poly1305_decrypt ( +u8 *dst,const u8 *src,const size_t src_len,const u8 *ad, +const size_t ad_len,const u8 nonce[XCHACHA20POLY1305_NONCE_SIZE ], +const u8 key[CHACHA20POLY1305_KEY_SIZE ]); + +#endif /* __CHACHA20POLY1305_H */
lib/crypto/Kconfig +7 -0
@@ -119,5 +119,12 @@ config CRYPTO_LIB_POLY1305 by either the generic implementation or an arch-specific one, if one is available and enabled. +config CRYPTO_LIB_CHACHA20POLY1305 + tristate "ChaCha20-Poly1305 AEAD support (8-byte nonce library version)" + depends on CRYPTO_ARCH_HAVE_LIB_CHACHA || !CRYPTO_ARCH_HAVE_LIB_CHACHA + depends on CRYPTO_ARCH_HAVE_LIB_POLY1305 || !CRYPTO_ARCH_HAVE_LIB_POLY1305 + select CRYPTO_LIB_CHACHA + select CRYPTO_LIB_POLY1305 + config CRYPTO_LIB_SHA256 tristate
lib/crypto/Makefile +4 -0
@@ -16,6 +16,9 @@ libblake2s-generic-y += blake2s-generic.o obj-$(CONFIG_CRYPTO_LIB_BLAKE2S) += libblake2s.o libblake2s-y += blake2s.o +obj-$(CONFIG_CRYPTO_LIB_CHACHA20POLY1305) += libchacha20poly1305.o +libchacha20poly1305-y += chacha20poly1305.o + obj-$(CONFIG_CRYPTO_LIB_CURVE25519_GENERIC) += libcurve25519.o libcurve25519-y := curve25519-fiat32.o libcurve25519-$(CONFIG_ARCH_SUPPORTS_INT128) := curve25519-hacl64.o
@@ -32,4 +35,5 @@ libsha256-y := sha256.o ifneq ($(CONFIG_CRYPTO_MANAGER_DISABLE_TESTS),y) libblake2s-y += blake2s-selftest.o +libchacha20poly1305-y += chacha20poly1305-selftest.o endif
lib/crypto/chacha20poly1305-selftest.c new file +7348 -0
2,937 lines not on this page · raw message
lib/crypto/chacha20poly1305.c new file +219 -0
219 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.20.1
Part of the diff is not shown here. View the raw message.
Ard Biesheuvel
maintainerPATCH v5 34/34 crypto: lib/chacha20poly1305 - reimplement crypt_from_sg() routine
In reply to Ard BiesheuvelReimplement the library routines to perform chacha20poly1305 en/decryption on scatterlists, without [ab]using the [deprecated] blkcipher interface, which is rather heavyweight and does things we don't really need.
Instead, we use the sg_miter API in a novel and clever way, to iterate over the scatterlist in-place (i.e., source == destination, which is the only way this library is expected to be used). That way, we don't have to iterate over two scatterlists in parallel.
Another optimization is that, instead of relying on the blkcipher walker to present the input in suitable chunks, we recognize that ChaCha is a streamcipher, and so we can simply deal with partial blocks by keeping a block of cipherstream on the stack and use crypto_xor() to mix it with the in/output.
Finally, we omit the scatterwalk_and_copy() call if the last element of the scatterlist covers the MAC as well (which is the common case), avoiding the need to walk the scatterlist and kmap() the page twice.
--- include/crypto/chacha20poly1305.h | 11 ++ lib/crypto/chacha20poly1305-selftest.c | 45 ++++++ lib/crypto/chacha20poly1305.c | 150 ++++++++++++++++++++ 3 files changed, 206 insertions(+)
include/crypto/chacha20poly1305.h +11 -0
@@ -7,6 +7,7 @@#define __CHACHA20POLY1305_H#include <linux/types.h> +#include <linux/scatterlist.h> enum chacha20poly1305_lengths {XCHACHA20POLY1305_NONCE_SIZE =24 ,
@@ -34,4 +35,14 @@ bool __must_check xchacha20poly1305_decrypt( const size_t ad_len, const u8 nonce[XCHACHA20POLY1305_NONCE_SIZE], const u8 key[CHACHA20POLY1305_KEY_SIZE]); +bool chacha20poly1305_encrypt_sg_inplace (struct scatterlist *src,size_t src_len, +const u8 *ad,const size_t ad_len, +const u64 nonce, +const u8 key[CHACHA20POLY1305_KEY_SIZE ]); + +bool chacha20poly1305_decrypt_sg_inplace (struct scatterlist *src,size_t src_len, +const u8 *ad,const size_t ad_len, +const u64 nonce, +const u8 key[CHACHA20POLY1305_KEY_SIZE ]); +#endif /* __CHACHA20POLY1305_H */
lib/crypto/chacha20poly1305-selftest.c +45 -0
63 lines not on this page · raw message
lib/crypto/chacha20poly1305.c +150 -0
162 lines not on this page · raw message
Signature · 1 line
2.20.1
Herbert Xu
maintainerQuoted 235 lines · Ard Biesheuvel
This series implements the crypto library abstractions that are needed to incorporate WireGuard into the mainline kernel.
Changes since v4:
- Address most review feedback from Eric, with the exception of the remark about libraries being selectable by the user - this is something we need to revisit in the context of moving to weak references or static calls to make accelerated versions of libraries loadable at any time. (Currently, loading an accelerated version at runtime will not supersede calls to the generic routines in the kernel proper, which is counterintuitive, and this is currently being addressed by making the generic library versions only selectable as modules if the accelerated ones are selected as modules as well)
- Align the generic blake2sKconfigsymbols, filenames etc with the recently added blake2b driver.
- Rewrote the blake2s selftest for better coverage of key length and input length combinations, and added a HMAC selftest as well.
- Renameblake2s_hmac()toblake2s256_hmac(), and drop the digest length argument, which was not implemented correctly, and never deviates from the full length in practice anyway.
- Update to more recent version of the blake2s x86 Zinc codeChanges since v3:
- Unify the way the generic vs arch libraries are organized between ChaCha20 and Poly1305 on the one hand and Curve25519 and Blake2s on the other. All are now made up of a generic library, a generic crypto API driver (skcipher for [X]ChaCha, shash for Poly1305 and Blake2s and kpp for Curve25519) and optional per-arch versions providing both the library and the crypto API interfaces while potentially relying on the generic *library* only as a fallback (and not on the generic crypto API driver). Implementations of the libary interface that don't require the fallback don't pull in the generic code at all, but the generic crypto API drivers are tied to the generic implementations directly (this is necessary since we fuzz test the accelerated implementations against the generic implementations)
- Provide testmgr test vectors for the Curve25519 and Blake2s crypto API drivers that were added in this revision. This also required some changes to the KPP test routines so we can test for failures as well.
- Update to the latest version of Andy Polyakov's Poly1305 implementation for MIPS that incorporates Rene's improvements for 32r2
- Remove logic in the x86 and ARM implementations of ChaCha and Poly1305 to prefer the non-SIMD path for short inputs. This is no longer necessary, and even undesirable since it forced ChaCha20Poly1305's ChaCha pass generating the Poly1305 nonce to always take the slower scalar path.Changes since v2:
- Reduce the cc: audience a bit, since I assumed that not everyone is interested in discussing the details of this.
- Incorporate scalar ARM code for ChaCha, and the 64-bit MIPS code for Poly1305. NOTE: the Cryptogams MIPS code now supports 32-bit MIPS as well, and not just 32r2, so I omitted Rene's Poly1305 implementation for now, and used Andy's code for everything.
- Incorporate NEON opt-out for Cortex-A5/A7. Note that the code is still exposed via the crypto API, but with a low prioririty, so it is still available and still gets test coverage, but is not used by default.
- Use static keys (*not* static calls) in the SIMD and bmi2/adx drivers to keep track of which implementation is being used, to avoid the memory load on each call.
- Defer using weak references or static calls until the dust around this has settled. Instead, rely onKconfigconstraints and symbol dependencies to ensure that the arch code is always used when it is loaded. This means you can only opt out of using the arch code if you disable it inKconfigbut this is something I can live with for now.
- Refactor the Curve25519 glue code slightly so that the call sites branch to the arch or generic code directly.
- Split up the Poly1305 refactoring patches so they can be reviewed more easily.Changes since RFC/v1:
- dropped the WireGuard patch itself, and the followup patches - since the purpose was to illustrate the extent of the required changes, there is no reason to keep including them.
- import the MIPS 32r2 versions of ChaCha and Poly1305, but expose both the crypto API and library interfaces so that not only WireGuard but also IPsec and Adiantum can benefit immediately. (The latter required adding support for the reduced round version of ChaCha to the MIPS asm code)
- fix up various minor kconfig/build issues found in randconfig testing (thanks Arnd!)Patches can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=wireguard-crypto-library-api-v5Cc Herbert Xu <[email protected]>Cc David Miller <[email protected]>Cc Jason A. Donenfeld <[email protected]>Cc Samuel Neves <[email protected]>Cc Arnd Bergmann <[email protected]>Cc Eric Biggers <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Martin Willi <[email protected]>Cc Rene van Dorst <[email protected]>Cc David Sterba <[email protected]>Ard Biesheuvel (27):
crypto: tidy up lib/cryptoKconfigandMakefile
crypto: chacha - move existing library code into lib/crypto
crypto: x86/chacha - depend on generic chacha library instead of
crypto driver
crypto: x86/chacha - expose SIMD ChaCha routine as library function
crypto: arm64/chacha - depend on generic chacha library instead of
crypto driver
crypto: arm64/chacha - expose arm64 ChaCha routine as library function
crypto: arm/chacha - import Eric Biggers's scalar accelerated ChaCha
code
crypto: arm/chacha - remove dependency on generic ChaCha driver
crypto: arm/chacha - expose ARM ChaCha routine as library function
crypto: mips/chacha - wire up accelerated 32r2 code from Zinc
crypto: chacha - unexportchacha_genericroutines
crypto: poly1305 - move core routines into a separate library
crypto: x86/poly1305 - unify Poly1305 state struct with generic code
crypto: poly1305 - exposeinit/update/finallibrary interface
crypto: x86/poly1305 - depend on generic library not generic shash
crypto: x86/poly1305 - expose existing driver as poly1305 library
crypto: arm64/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON
implementation
crypto: arm/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON
implementation
crypto: mips/poly1305 - incorporate OpenSSL/CRYPTOGAMS optimized
implementation
int128: move__uint128_tcompiler test toKconfig
crypto: testmgr - add test cases for Blake2s
crypto: blake2s - implement generic shash driver
crypto: curve25519 - add kpp selftest
crypto: curve25519 - implement generic KPP driver
crypto: lib/curve25519 - work around Clang stack spilling issue
crypto: chacha20poly1305 - import construction and selftest from Zinc
crypto: lib/chacha20poly1305 - reimplementcrypt_from_sg()routineJason A. Donenfeld (7):
crypto: mips/chacha - import 32r2 ChaCha code from Zinc
crypto: BLAKE2s - generic C library implementation and selftest
crypto: BLAKE2s -x86_64SIMD implementation
crypto: Curve25519 - generic C library implementations
crypto: Curve25519 -x86_64library and KPP implementations
crypto: arm - import Bernstein and Schwabe's Curve25519 ARM
implementation
crypto: arm/Curve25519 - wire up NEON implementationarch/arm/crypto/Kconfig | 16 +- arch/arm/crypto/Makefile | 17 +- arch/arm/crypto/chacha-glue.c | 343 + arch/arm/crypto/chacha-neon-glue.c | 202 - arch/arm/crypto/chacha-scalar-core.S | 460 ++ arch/arm/crypto/curve25519-core.S | 2062 ++++++ arch/arm/crypto/curve25519-glue.c | 127 + arch/arm/crypto/poly1305-armv4.pl | 1236 ++++ arch/arm/crypto/poly1305-core.S_shipped | 1158 +++ arch/arm/crypto/poly1305-glue.c | 276 + arch/arm64/Kconfig | 2 +- arch/arm64/crypto/Kconfig | 9 +- arch/arm64/crypto/Makefile | 10 +- arch/arm64/crypto/chacha-neon-glue.c | 81 +- arch/arm64/crypto/poly1305-armv8.pl | 913 +++ arch/arm64/crypto/poly1305-core.S_shipped | 835 +++ arch/arm64/crypto/poly1305-glue.c | 237 + arch/mips/Makefile | 2 +- arch/mips/crypto/Makefile | 18 + arch/mips/crypto/chacha-core.S | 497 ++ arch/mips/crypto/chacha-glue.c | 150 + arch/mips/crypto/poly1305-glue.c | 203 + arch/mips/crypto/poly1305-mips.pl | 1273 ++++ arch/riscv/Kconfig | 2 +- arch/x86/Kconfig | 2 +- arch/x86/crypto/Makefile | 3 + arch/x86/crypto/blake2s-core.S | 258 + arch/x86/crypto/blake2s-glue.c | 233 + arch/x86/crypto/chacha_glue.c | 181 +- arch/x86/crypto/curve25519-x86_64.c | 2475 +++++++ arch/x86/crypto/poly1305_glue.c | 199 +- crypto/Kconfig | 71 +- crypto/Makefile | 2 + crypto/adiantum.c | 5 +- crypto/blake2s_generic.c | 171 + crypto/chacha_generic.c | 84 +- crypto/curve25519-generic.c | 90 + crypto/ecc.c | 2 +- crypto/nhpoly1305.c | 3 +- crypto/poly1305_generic.c | 228 +- crypto/testmgr.c | 30 + crypto/testmgr.h | 1520 +++- include/crypto/blake2s.h | 106 + include/crypto/chacha.h | 83 +- include/crypto/chacha20poly1305.h | 48 + include/crypto/curve25519.h | 71 + include/crypto/internal/blake2s.h | 24 + include/crypto/internal/chacha.h | 43 + include/crypto/internal/poly1305.h | 58 + include/crypto/poly1305.h | 69 +- init/Kconfig | 4 + lib/Makefile | 3 +- lib/crypto/Kconfig | 130 + lib/crypto/Makefile | 42 +- lib/crypto/blake2s-generic.c | 111 + lib/crypto/blake2s-selftest.c | 622 ++ lib/crypto/blake2s.c | 126 + lib/{ => crypto}/chacha.c | 20 +- lib/crypto/chacha20poly1305-selftest.c | 7393 ++++++++++++++++++++ lib/crypto/chacha20poly1305.c | 369 + lib/crypto/curve25519-fiat32.c | 864 +++ lib/crypto/curve25519-hacl64.c | 788 +++ lib/crypto/curve25519.c | 25 + lib/crypto/libchacha.c | 35 + lib/crypto/poly1305.c | 232 + lib/ubsan.c | 2 +- lib/ubsan.h | 2 +- 67 files changed, 26148 insertions(+), 808 deletions(-) create mode 100644 arch/arm/crypto/chacha-glue.c delete mode 100644 arch/arm/crypto/chacha-neon-glue.c create mode 100644 arch/arm/crypto/chacha-scalar-core.S create mode 100644 arch/arm/crypto/curve25519-core.S create mode 100644 arch/arm/crypto/curve25519-glue.c create mode 100644 arch/arm/crypto/poly1305-armv4.pl create mode 100644 arch/arm/crypto/poly1305-core.S_shipped create mode 100644 arch/arm/crypto/poly1305-glue.c create mode 100644 arch/arm64/crypto/poly1305-armv8.pl create mode 100644 arch/arm64/crypto/poly1305-core.S_shipped create mode 100644 arch/arm64/crypto/poly1305-glue.c create mode 100644 arch/mips/crypto/chacha-core.S create mode 100644 arch/mips/crypto/chacha-glue.c create mode 100644 arch/mips/crypto/poly1305-glue.c create mode 100644 arch/mips/crypto/poly1305-mips.pl create mode 100644 arch/x86/crypto/blake2s-core.S create mode 100644 arch/x86/crypto/blake2s-glue.c create mode 100644 arch/x86/crypto/curve25519-x86_64.c create mode 100644 crypto/blake2s_generic.c create mode 100644 crypto/curve25519-generic.c create mode 100644 include/crypto/blake2s.h create mode 100644 include/crypto/chacha20poly1305.h create mode 100644 include/crypto/curve25519.h create mode 100644 include/crypto/internal/blake2s.h create mode 100644 include/crypto/internal/chacha.h create mode 100644 include/crypto/internal/poly1305.h create mode 100644 lib/crypto/Kconfig create mode 100644 lib/crypto/blake2s-generic.c create mode 100644 lib/crypto/blake2s-selftest.c create mode 100644 lib/crypto/blake2s.c
Quoted 8 lines
rename lib/{ => crypto}/chacha.c (88%)
create mode 100644 lib/crypto/chacha20poly1305-selftest.c
create mode 100644 lib/crypto/chacha20poly1305.c
create mode 100644 lib/crypto/curve25519-fiat32.c
create mode 100644 lib/crypto/curve25519-hacl64.c
create mode 100644 lib/crypto/curve25519.c
create mode 100644 lib/crypto/libchacha.c
create mode 100644 lib/crypto/poly1305.cAll applied. Thanks.
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
A message that is not in the archive (sent off-list or not yet received): CAHmME9oOfhv6RN00m1c6c5qELC5dzFKS=mgDBQ-stVEWu00p_A@mail.gmail.com
Herbert Xu
maintainerQuoted 4 lines · Jason A. Donenfeld
Hey Herbert,
I've been traveling and got back home yesterday. I was going to review these this afternoon if there's still opportunity.
Hi Jason:
Last I checked there weren't any major objections to this patch-set. If there are any remaining issues we can resolve them with new patches on top of this series.
Thanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Hey Ard, Herbert, Dave,
The series looks fine. Ard -- thanks so much for picking up the work and making this happen. As far as I'm concerned, this is "most" of Zinc, simply without calling it "Zinc", and minus a few other things that I think constitutes an okay compromise and good base for moving forward.
The TODO list for me remains the same, and now I can get moving with that:
- Zinc's generic C implementation of poly1305, which is faster and has separate implementations for u64 and u128.
- x86_64 ChaCha20 from Zinc. Will be fun to discuss with Martin and Andy.
- x86_64 Poly1305 from Zinc.
- Resurrecting the big_keys patch and receiving DavidH's review on that.
- WireGuard! Hurrah!
If you have any feedback on how you'd like this prioritized, please pipe up. For example Dave - would you like WireGuard *now* or sometime later? I can probably get that cooking this week, though I do have some testing and fuzzing of it to do on top of the patches that just landed in cryptodev.
Jason
Hey Jason,
Quoted 11 lines · Jason A. Donenfeld
Hey Ard, Herbert, Dave,
The series looks fine. Ard -- thanks so much for picking up the work and making this happen. As far as I'm concerned, this is "most" of Zinc, simply without calling it "Zinc", and minus a few other things that I think constitutes an okay compromise and good base for moving forward.
Reviewed-by Jason A. Donenfeld <[email protected]>
Thanks!
Quoted 6 lines
The TODO list for me remains the same, and now I can get moving with that:
- Zinc's generic C implementation of poly1305, which is faster and has separate implementations for u64 and u128.
-x86_64ChaCha20 from Zinc. Will be fun to discuss with Martin and Andy.
-x86_64Poly1305 from Zinc.
As I pointed out in the private discussions we had, there are two aspects two AndyP's benchmarking that don't carry over 100% to the Linux kernel:
- Every microarchitecture is given equal weight, regardless of the likelihood that the code will actually run on it. This makes some sense for OpenSSL, I guess, but not for the kernel.
- Benchmarks are typically based on the performance of the core cryptographics transformation rather than a representative workload. This is especially relevant for network use cases, where packet sizes are not necessarily fixed and usually not a multiple of the block size (as opposed to disk encryption, where every single call is the same size and a power of 2)
So for future changes, could we please include performance numbers based on realistic workloads?
Quoted 1 line
- Resurrecting the
big_keyspatch and receiving DavidH's review on that.
My concern here (but we can discuss it in the context of a repost) is that this will pull the accelerated chacha20 and poly1305 code (if enabled) into the core kernel, given that big_keys is not a tristate option. So perhaps we can park this one until we know how to proceed with static calls or alternative approaches?
Quoted 2 lines
- WireGuard! Hurrah!
I'm a bit surprised that this only appears at the end of your list :-)
Quoted 6 lines
If you have any feedback on how you'd like this prioritized, please pipe up. For example Dave - would you like WireGuard *now* or sometime later? I can probably get that cooking this week, though I do have some testing and fuzzing of it to do on top of the patches that just landed in cryptodev.
We're at -rc8, and wireguard itself will not go via the crypto tree so you should wait until after the merge window, rebase it onto -rc1 and repost it.
Jason A. Donenfeld
maintainerQuoted 2 lines · Ard Biesheuvel
Quoted 2 lines
- Zinc's generic C implementation of poly1305, which is faster and has separate implementations for u64 and u128.
I assume your AndyP comment below didn't apply to this top item here. This one should be fairly uncontroversial in your opinion, right?
Quoted 16 lines
Quoted 2 lines
-
x86_64ChaCha20 from Zinc. Will be fun to discuss with Martin and Andy.
-x86_64Poly1305 from Zinc.As I pointed out in the private discussions we had, there are two aspects two AndyP's benchmarking that don't carry over 100% to the Linux kernel:
- Every microarchitecture is given equal weight, regardless of the likelihood that the code will actually run on it. This makes some sense for OpenSSL, I guess, but not for the kernel.
- Benchmarks are typically based on the performance of the core cryptographics transformation rather than a representative workload. This is especially relevant for network use cases, where packet sizes are not necessarily fixed and usually not a multiple of the block size (as opposed to disk encryption, where every single call is the same size and a power of 2)
Quoted 2 lines
So for future changes, could we please include performance numbers based on realistic workloads?
Yea I share your concerns here. From preliminary results, I think the Poly1305 code will be globally better, and I don't think we'll need an abundance of discussion about it.
The ChaCha case is more interesting. I'll submit this with lots of packet-sized microbenchmarks, as well as on-the-wire WireGuard testing. Eric - I'm guessing you don't care too much about Adamantium performance on x86 where people are probably better off with AES-XTS, right? Are there other specific real world cases we care about? IPsec is another one, but those concerns, packet-size wise, are more or less the same as for WireGuard. But anyway, we can cross this bridge when we come to it.
Quoted 3 lines
Quoted 2 lines
- WireGuard! Hurrah!
I'm a bit surprised that this only appears at the end of your list :-)
Haha, "last but not least" :)
Quoted 11 lines
Quoted 7 lines
If you have any feedback on how you'd like this prioritized, please pipe up. For example Dave - would you like WireGuard *now* or sometime later? I can probably get that cooking this week, though I do have some testing and fuzzing of it to do on top of the patches that just landed in cryptodev.
We're at -rc8, and wireguard itself will not go via the crypto tree so you should wait until after the merge window, rebase it onto -rc1 and repost it.
Thanks, yea, that makes sense. Netdev also has its own merge window schedule that I should aim to meet. I guess, based on this if I'm understanding correctly, we're looking at WireGuard for 5.5?
Jason
Quoted 9 lines · Jason A. Donenfeld
Quoted 2 lines · Ard Biesheuvel
Quoted 2 lines
- Zinc's generic C implementation of poly1305, which is faster and has separate implementations for u64 and u128.
I assume your AndyP comment below didn't apply to this top item here. This one should be fairly uncontroversial in your opinion, right?
Yes.
Quoted 16 lines
Quoted 16 lines
Quoted 2 lines
-
x86_64ChaCha20 from Zinc. Will be fun to discuss with Martin and Andy.
-x86_64Poly1305 from Zinc.As I pointed out in the private discussions we had, there are two aspects two AndyP's benchmarking that don't carry over 100% to the Linux kernel:
- Every microarchitecture is given equal weight, regardless of the likelihood that the code will actually run on it. This makes some sense for OpenSSL, I guess, but not for the kernel.
- Benchmarks are typically based on the performance of the core cryptographics transformation rather than a representative workload. This is especially relevant for network use cases, where packet sizes are not necessarily fixed and usually not a multiple of the block size (as opposed to disk encryption, where every single call is the same size and a power of 2)
Quoted 7 lines
Quoted 2 lines
So for future changes, could we please include performance numbers based on realistic workloads?
Yea I share your concerns here. From preliminary results, I think the Poly1305 code will be globally better, and I don't think we'll need an abundance of discussion about it.
Good.
Quoted 9 lines
The ChaCha case is more interesting. I'll submit this with lots of packet-sized microbenchmarks, as well as on-the-wire WireGuard testing. Eric - I'm guessing you don't care too much about Adamantium performance on x86 where people are probably better off with AES-XTS, right? Are there other specific real world cases we care about? IPsec is another one, but those concerns, packet-size wise, are more or less the same as for WireGuard. But anyway, we can cross this bridge when we come to it.
As long as we can show that WireGuard really benefits without regressing other users disproportionately, I think we're fine. There are not /that/ many users to begin with ...
Quoted 15 lines
Quoted 4 lines
Quoted 3 lines
- WireGuard! Hurrah!
I'm a bit surprised that this only appears at the end of your list :-)
Haha, "last but not least" :)
Quoted 8 lines
Quoted 7 lines
If you have any feedback on how you'd like this prioritized, please pipe up. For example Dave - would you like WireGuard *now* or sometime later? I can probably get that cooking this week, though I do have some testing and fuzzing of it to do on top of the patches that just landed in cryptodev.
Quoted 8 lines
Quoted 3 lines
We're at -rc8, and wireguard itself will not go via the crypto tree so you should wait until after the merge window, rebase it onto -rc1 and repost it.
Thanks, yea, that makes sense. Netdev also has its own merge window schedule that I should aim to meet. I guess, based on this if I'm understanding correctly, we're looking at WireGuard for 5.5?
No, v5.6. The merge window for v5.5 will open this Sunday, so you'll need to rebase on v5.5-rc1 once it is released, which will [hopefully] have all the crypto pieces you need.
Note that this applies equally to the other changes: we can queue performance tweaks in the crypto tree in parallel, and they will all hit v5.6 at the same time.
Eric Biggers
maintainerQuoted 8 lines · Jason A. Donenfeld
Quoted 3 lines
So for future changes, could we please include performance numbers based on realistic workloads?
Yea I share your concerns here. From preliminary results, I think the Poly1305 code will be globally better, and I don't think we'll need an abundance of discussion about it.
Quoted 8 lines
The ChaCha case is more interesting. I'll submit this with lots of packet-sized microbenchmarks, as well as on-the-wire WireGuard testing. Eric - I'm guessing you don't care too much about Adamantium performance on x86 where people are probably better off with AES-XTS, right? Are there other specific real world cases we care about? IPsec is another one, but those concerns, packet-size wise, are more or less the same as for WireGuard. But anyway, we can cross this bridge when we come to it.
I'd like for Adiantum to continue to be accelerated on x86, but it doesn't have to squeeze out all performance possible on x86, given that hardware AES support is available there so most people will use that instead. So if e.g. the ChaCha implementation is still AVX2 accelerated, but it's primarily optimized for networking packets rather than disk encryption, that would probably be fine.
- Eric
Quoted 17 lines · Eric Biggers
Quoted 14 lines · Jason A. Donenfeld
Quoted 2 lines
So for future changes, could we please include performance numbers based on realistic workloads?
Yea I share your concerns here. From preliminary results, I think the Poly1305 code will be globally better, and I don't think we'll need an abundance of discussion about it.
The ChaCha case is more interesting. I'll submit this with lots of packet-sized microbenchmarks, as well as on-the-wire WireGuard testing. Eric - I'm guessing you don't care too much about Adamantium performance on x86 where people are probably better off with AES-XTS, right? Are there other specific real world cases we care about? IPsec is another one, but those concerns, packet-size wise, are more or less the same as for WireGuard. But anyway, we can cross this bridge when we come to it.
Quoted 7 lines
I'd like for Adiantum to continue to be accelerated on x86, but it doesn't have to squeeze out all performance possible on x86, given that hardware AES support is available there so most people will use that instead. So if e.g. the ChaCha implementation is still AVX2 accelerated, but it's primarily optimized for networking packets rather than disk encryption, that would probably be fine.
- Eric
I'm interested in using Adamantium on x86 and I hope that you folks won't cripple it :(
Jordan