Architecture-optimized SHA-256 library API
Subsystems (16)
Eric Biggers
maintainerv4 cover Architecture-optimized SHA-256 library API
This is based on cryptodev commit 2dfc7cd74a5e062a. It can also be retrieved from:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git sha256-lib-v4
Following the example of several other algorithms (e.g. CRC32, ChaCha, Poly1305, BLAKE2s), this series refactors the kernel's existing architecture-optimized SHA-256 code to be available via the library API, instead of just via the crypto_shash API as it was before. It also reimplements the SHA-256 crypto_shash API on top of the library API.
This makes it possible to use the SHA-256 library in performance-critical cases. The new design is also much simpler, with a negative diffstat of almost 1200 lines. Finally, this also fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default, so people often forgot to enable it.
For now the SHA-256 library is well-covered by the crypto_shash self-tests, but I plan to add a test for the library directly later. I've fully tested this series on arm, arm64, riscv, and x86. On mips, powerpc, s390, and sparc I've only been able to partially test it, since QEMU does not support the SHA-256 instructions on those platforms. If anyone with access to a mips, powerpc, s390, or sparc system that has SHA-256 instructions can verify that the crypto self-tests still pass,
Changed v1 => v4:
- Moved sha256_generic_blocks() into its own module to avoid a circular module dependency.
- Added Ard's Reviewed-by tags.
- Rebased onto cryptodev.
Eric Biggers (13):
crypto: sha256 - support arch-optimized lib and expose through shash
crypto: arm/sha256 - implement library instead of shash
crypto: arm64/sha256 - remove obsolete chunking logic
crypto: arm64/sha256 - implement library instead of shash
crypto: mips/sha256 - implement library instead of shash
crypto: powerpc/sha256 - implement library instead of shash
crypto: riscv/sha256 - implement library instead of shash
crypto: s390/sha256 - implement library instead of shash
crypto: sparc - move opcodes.h into asm directory
crypto: sparc/sha256 - implement library instead of shash
crypto: x86/sha256 - implement library instead of shash
crypto: sha256 - remove sha256_base.h
crypto: lib/sha256 - improve function prototypes
arch/arm/configs/exynos_defconfig | 1 -
arch/arm/configs/milbeaut_m10v_defconfig | 1 -
arch/arm/configs/multi_v7_defconfig | 1 -
arch/arm/configs/omap2plus_defconfig | 1 -
arch/arm/configs/pxa_defconfig | 1 -
arch/arm/crypto/Kconfig | 21 -
arch/arm/crypto/Makefile | 8 +-
arch/arm/crypto/sha2-ce-glue.c | 87 ----
arch/arm/crypto/sha256_glue.c | 107 -----
arch/arm/crypto/sha256_glue.h | 9 -
arch/arm/crypto/sha256_neon_glue.c | 75 ---
arch/arm/lib/crypto/.gitignore | 1 +
arch/arm/lib/crypto/Kconfig | 6 +
arch/arm/lib/crypto/Makefile | 8 +-
arch/arm/{ => lib}/crypto/sha256-armv4.pl | 0
.../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 10 +-
arch/arm/lib/crypto/sha256.c | 64 +++
arch/arm64/configs/defconfig | 1 -
arch/arm64/crypto/Kconfig | 19 -
arch/arm64/crypto/Makefile | 13 +-
arch/arm64/crypto/sha2-ce-glue.c | 138 ------
arch/arm64/crypto/sha256-glue.c | 171 -------
arch/arm64/lib/crypto/.gitignore | 1 +
arch/arm64/lib/crypto/Kconfig | 5 +
arch/arm64/lib/crypto/Makefile | 9 +-
.../crypto/sha2-armv8.pl} | 0
.../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 36 +-
arch/arm64/lib/crypto/sha256.c | 75 +++
arch/mips/cavium-octeon/Kconfig | 6 +
.../mips/cavium-octeon/crypto/octeon-sha256.c | 135 ++----
arch/mips/configs/cavium_octeon_defconfig | 1 -
arch/mips/crypto/Kconfig | 10 -
arch/powerpc/crypto/Kconfig | 11 -
arch/powerpc/crypto/Makefile | 2 -
arch/powerpc/crypto/sha256-spe-glue.c | 128 ------
arch/powerpc/lib/crypto/Kconfig | 6 +
arch/powerpc/lib/crypto/Makefile | 3 +
.../powerpc/{ => lib}/crypto/sha256-spe-asm.S | 0
arch/powerpc/lib/crypto/sha256.c | 70 +++
arch/riscv/crypto/Kconfig | 11 -
arch/riscv/crypto/Makefile | 3 -
arch/riscv/crypto/sha256-riscv64-glue.c | 125 -----
arch/riscv/lib/crypto/Kconfig | 7 +
arch/riscv/lib/crypto/Makefile | 3 +
.../sha256-riscv64-zvknha_or_zvknhb-zvkb.S | 4 +-
arch/riscv/lib/crypto/sha256.c | 62 +++
arch/s390/configs/debug_defconfig | 1 -
arch/s390/configs/defconfig | 1 -
arch/s390/crypto/Kconfig | 10 -
arch/s390/crypto/Makefile | 1 -
arch/s390/crypto/sha256_s390.c | 144 ------
arch/s390/lib/crypto/Kconfig | 6 +
arch/s390/lib/crypto/Makefile | 2 +
arch/s390/lib/crypto/sha256.c | 47 ++
arch/sparc/crypto/Kconfig | 10 -
arch/sparc/crypto/Makefile | 2 -
arch/sparc/crypto/aes_asm.S | 3 +-
arch/sparc/crypto/aes_glue.c | 3 +-
arch/sparc/crypto/camellia_asm.S | 3 +-
arch/sparc/crypto/camellia_glue.c | 3 +-
arch/sparc/crypto/des_asm.S | 3 +-
arch/sparc/crypto/des_glue.c | 3 +-
arch/sparc/crypto/md5_asm.S | 3 +-
arch/sparc/crypto/md5_glue.c | 3 +-
arch/sparc/crypto/sha1_asm.S | 3 +-
arch/sparc/crypto/sha1_glue.c | 3 +-
arch/sparc/crypto/sha256_glue.c | 129 ------
arch/sparc/crypto/sha512_asm.S | 3 +-
arch/sparc/crypto/sha512_glue.c | 3 +-
arch/sparc/{crypto => include/asm}/opcodes.h | 6 +-
arch/sparc/lib/Makefile | 1 +
arch/sparc/lib/crc32c_asm.S | 3 +-
arch/sparc/lib/crypto/Kconfig | 8 +
arch/sparc/lib/crypto/Makefile | 4 +
arch/sparc/lib/crypto/sha256.c | 64 +++
arch/sparc/{ => lib}/crypto/sha256_asm.S | 5 +-
arch/x86/crypto/Kconfig | 14 -
arch/x86/crypto/Makefile | 3 -
arch/x86/crypto/sha256_ssse3_glue.c | 432 ------------------
arch/x86/lib/crypto/Kconfig | 7 +
arch/x86/lib/crypto/Makefile | 3 +
arch/x86/{ => lib}/crypto/sha256-avx-asm.S | 12 +-
arch/x86/{ => lib}/crypto/sha256-avx2-asm.S | 12 +-
.../crypto/sha256-ni-asm.S} | 36 +-
arch/x86/{ => lib}/crypto/sha256-ssse3-asm.S | 14 +-
arch/x86/lib/crypto/sha256.c | 74 +++
crypto/Kconfig | 1 +
crypto/Makefile | 3 +-
crypto/sha256.c | 201 ++++++++
crypto/sha256_generic.c | 102 -----
include/crypto/internal/sha2.h | 28 ++
include/crypto/sha2.h | 23 +-
include/crypto/sha256_base.h | 180 --------
lib/crypto/Kconfig | 22 +
lib/crypto/Makefile | 3 +
lib/crypto/sha256-generic.c | 137 ++++++
lib/crypto/sha256.c | 204 ++++-----
97 files changed, 1128 insertions(+), 2319 deletions(-)
delete mode 100644 arch/arm/crypto/sha2-ce-glue.c
delete mode 100644 arch/arm/crypto/sha256_glue.c
delete mode 100644 arch/arm/crypto/sha256_glue.h
delete mode 100644 arch/arm/crypto/sha256_neon_glue.c
rename arch/arm/{ => lib}/crypto/sha256-armv4.pl (100%)
rename arch/arm/{crypto/sha2-ce-core.S => lib/crypto/sha256-ce.S} (91%)
create mode 100644 arch/arm/lib/crypto/sha256.c
delete mode 100644 arch/arm64/crypto/sha2-ce-glue.c
delete mode 100644 arch/arm64/crypto/sha256-glue.c
rename arch/arm64/{crypto/sha512-armv8.pl => lib/crypto/sha2-armv8.pl} (100%)
rename arch/arm64/{crypto/sha2-ce-core.S => lib/crypto/sha256-ce.S} (80%)
create mode 100644 arch/arm64/lib/crypto/sha256.c
delete mode 100644 arch/powerpc/crypto/sha256-spe-glue.c
rename arch/powerpc/{ => lib}/crypto/sha256-spe-asm.S (100%)
create mode 100644 arch/powerpc/lib/crypto/sha256.c
delete mode 100644 arch/riscv/crypto/sha256-riscv64-glue.c
rename arch/riscv/{ => lib}/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S (98%)
create mode 100644 arch/riscv/lib/crypto/sha256.c
delete mode 100644 arch/s390/crypto/sha256_s390.c
create mode 100644 arch/s390/lib/crypto/sha256.c
delete mode 100644 arch/sparc/crypto/sha256_glue.c
rename arch/sparc/{crypto => include/asm}/opcodes.h (96%)
create mode 100644 arch/sparc/lib/crypto/Kconfig
create mode 100644 arch/sparc/lib/crypto/Makefile
create mode 100644 arch/sparc/lib/crypto/sha256.c
rename arch/sparc/{ => lib}/crypto/sha256_asm.S (95%)
delete mode 100644 arch/x86/crypto/sha256_ssse3_glue.c
rename arch/x86/{ => lib}/crypto/sha256-avx-asm.S (98%)
rename arch/x86/{ => lib}/crypto/sha256-avx2-asm.S (98%)
rename arch/x86/{crypto/sha256_ni_asm.S => lib/crypto/sha256-ni-asm.S} (85%)
rename arch/x86/{ => lib}/crypto/sha256-ssse3-asm.S (98%)
create mode 100644 arch/x86/lib/crypto/sha256.c
create mode 100644 crypto/sha256.c
delete mode 100644 crypto/sha256_generic.c
create mode 100644 include/crypto/internal/sha2.h
delete mode 100644 include/crypto/sha256_base.h
create mode 100644 lib/crypto/sha256-generic.cbase-commit: 2dfc7cd74a5e062a5405560447517e7aab1c7341
Signature · 1 line
2.49.0
Eric Biggers
maintainerPATCH v4 1/13 crypto: sha256 - support arch-optimized lib and expose through shash
From: Eric Biggers <[email protected]>
As has been done for various other algorithms, rework the design of the SHA-256 library to support arch-optimized implementations, and make crypto/sha256.c expose both generic and arch-optimized shash algorithms that wrap the library functions.
This allows users of the SHA-256 library functions to take advantage of the arch-optimized code, and this makes it much simpler to integrate SHA-256 for each architecture.
Note that sha256_base.h is not used in the new design. It will be removed once all the architecture-specific code has been updated.
Move the generic block function into its own module to avoid a circular dependency from libsha256.ko => sha256-$ARCH.ko => libsha256.ko.
--- crypto/Kconfig | 1 + crypto/Makefile | 3 +- crypto/sha256.c | 201 +++++++++++++++++++++++++++++++++ crypto/sha256_generic.c | 102 ----------------- include/crypto/internal/sha2.h | 28 +++++ include/crypto/sha2.h | 15 +-- include/crypto/sha256_base.h | 9 +- lib/crypto/Kconfig | 19 ++++ lib/crypto/Makefile | 3 + lib/crypto/sha256-generic.c | 137 ++++++++++++++++++++++ lib/crypto/sha256.c | 196 ++++++++++++++------------------ 11 files changed, 487 insertions(+), 227 deletions(-) create mode 100644 crypto/sha256.c delete mode 100644 crypto/sha256_generic.c create mode 100644 include/crypto/internal/sha2.h create mode 100644 lib/crypto/sha256-generic.c
crypto/Kconfig +1 -0
@@ -992,10 +992,11 @@ config CRYPTO_SHA1 config CRYPTO_SHA256 tristate "SHA-224 and SHA-256" select CRYPTO_HASH select CRYPTO_LIB_SHA256 + select CRYPTO_LIB_SHA256_GENERIC help SHA-224 and SHA-256 secure hash algorithms (FIPS 180, ISO/IEC 10118-3) This is required for IPsec AH (XFRM_AH) and IPsec ESP (XFRM_ESP). Used by the btrfs filesystem, Ceph, NFS, and SMB.
crypto/Makefile +2 -1
@@ -74,11 +74,12 @@ obj-$(CONFIG_CRYPTO_XCBC) += xcbc.o obj-$(CONFIG_CRYPTO_NULL2) += crypto_null.o obj-$(CONFIG_CRYPTO_MD4) += md4.o obj-$(CONFIG_CRYPTO_MD5) += md5.o obj-$(CONFIG_CRYPTO_RMD160) += rmd160.o obj-$(CONFIG_CRYPTO_SHA1) += sha1_generic.o -obj-$(CONFIG_CRYPTO_SHA256) += sha256_generic.o +obj-$(CONFIG_CRYPTO_SHA256) += sha256.o +CFLAGS_sha256.o += -DARCH=$(ARCH) obj-$(CONFIG_CRYPTO_SHA512) += sha512_generic.o obj-$(CONFIG_CRYPTO_SHA3) += sha3_generic.o obj-$(CONFIG_CRYPTO_SM3_GENERIC) += sm3_generic.o obj-$(CONFIG_CRYPTO_STREEBOG) += streebog_generic.o obj-$(CONFIG_CRYPTO_WP512) += wp512.o
crypto/sha256.c new file +201 -0
201 lines not on this page · raw message
crypto/sha256_generic.c deleted +0 -102
102 lines not on this page · raw message
include/crypto/internal/sha2.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef _CRYPTO_INTERNAL_SHA2_H +#define _CRYPTO_INTERNAL_SHA2_H + +#include <crypto/sha2.h> + +void sha256_update_generic (struct sha256_state *sctx, +const u8 *data,size_t len); +void sha256_final_generic (struct sha256_state *sctx, +u8 out[SHA256_DIGEST_SIZE ]); +void sha224_final_generic (struct sha256_state *sctx, +u8 out[SHA224_DIGEST_SIZE ]); + +#if IS_ENABLED (CONFIG_CRYPTO_ARCH_HAVE_LIB_SHA256 ) +bool sha256_is_arch_optimized (void ); +#else +static inline bool sha256_is_arch_optimized (void ) +{ +return false; +} +#endif +void sha256_blocks_generic (u32 state[SHA256_STATE_WORDS ], +const u8 *data,size_t nblocks); +void sha256_blocks_arch (u32 state[SHA256_STATE_WORDS ], +const u8 *data,size_t nblocks); + +#endif /* _CRYPTO_INTERNAL_SHA2_H */
include/crypto/sha2.h +3 -12
@@ -11,10 +11,11 @@#define SHA224_DIGEST_SIZE 28#define SHA224_BLOCK_SIZE 64#define SHA256_DIGEST_SIZE 32#define SHA256_BLOCK_SIZE 64 +#define SHA256_STATE_WORDS 8#define SHA384_DIGEST_SIZE 48#define SHA384_BLOCK_SIZE 128#define SHA512_DIGEST_SIZE 64
@@ -64,36 +65,26 @@ extern const u8 sha256_zero_message_hash[SHA256_DIGEST_SIZE];extern const u8 sha384_zero_message_hash[SHA384_DIGEST_SIZE ];extern const u8 sha512_zero_message_hash[SHA512_DIGEST_SIZE ];struct crypto_sha256_state { -u32 state[SHA256_DIGEST_SIZE /4 ]; +u32 state[SHA256_STATE_WORDS ];u64 count; };struct sha256_state { -u32 state[SHA256_DIGEST_SIZE /4 ]; +u32 state[SHA256_STATE_WORDS ];u64 count;u8 buf[SHA256_BLOCK_SIZE ]; };struct sha512_state {u64 state[SHA512_DIGEST_SIZE /8 ];u64 count[2 ];u8 buf[SHA512_BLOCK_SIZE ]; }; -/* -* Stand-alone implementation of the SHA256 algorithm. It is designed to -* have as little dependencies as possible so it can be used in the -* kexec_file purgatory. In other cases you should generally use the -* hash APIs from include/crypto/hash.h. Especially when hashing large -* amounts of data as those APIs may be hw-accelerated. -* -* For details see lib/crypto/sha256.c -*/ -static inline void sha256_init (struct sha256_state *sctx) { sctx->state[0 ] =SHA256_H0 ; sctx->state[1 ] =SHA256_H1 ; sctx->state[2 ] =SHA256_H2 ;
include/crypto/sha256_base.h +6 -3
@@ -7,11 +7,11 @@#ifndef _CRYPTO_SHA256_BASE_H#define _CRYPTO_SHA256_BASE_H#include <crypto/internal/hash.h> -#include <crypto/sha2.h> +#include <crypto/internal/sha2.h> #include <linux/math.h> #include <linux/string.h> #include <linux/types.h> #include <linux/unaligned.h>
@@ -172,9 +172,12 @@ static inline int sha256_base_finish(struct shash_desc *desc, u8 *out)struct crypto_sha256_state *sctx =shash_desc_ctx (desc); return __sha256_base_finish(sctx->state, out, digest_size); } -void sha256_transform_blocks (struct crypto_sha256_state *sst, -const u8 *input,int blocks); +static inline void sha256_transform_blocks (struct crypto_sha256_state *sst, +const u8 *input,int blocks) +{ +sha256_blocks_generic (sst->state, input, blocks); +}#endif /* _CRYPTO_SHA256_BASE_H */
lib/crypto/Kconfig +19 -0
@@ -137,10 +137,29 @@ config CRYPTO_LIB_CHACHA20POLY1305 config CRYPTO_LIB_SHA1 tristate config CRYPTO_LIB_SHA256 tristate + help + Enable the SHA-256 library interface. This interface may be fulfilled + by either the generic implementation or an arch-specific one, if one + is available and enabled. + +config CRYPTO_ARCH_HAVE_LIB_SHA256 + bool + help + Declares whether the architecture provides an arch-specific + accelerated implementation of the SHA-256 library interface. + +config CRYPTO_LIB_SHA256_GENERIC + tristate + default CRYPTO_LIB_SHA256 if !CRYPTO_ARCH_HAVE_LIB_SHA256 + help + This symbol can be selected by arch implementations of the SHA-256 + library interface that require the generic code as a fallback, e.g., + for SIMD implementations. If no arch specific implementation is + enabled, this implementation serves the users of CRYPTO_LIB_SHA256. config CRYPTO_LIB_SM3 tristate if !KMSAN # avoid false positives from assembly
lib/crypto/Makefile +3 -0
@@ -49,10 +49,13 @@ obj-$(CONFIG_CRYPTO_LIB_SHA1) += libsha1.o libsha1-y := sha1.o obj-$(CONFIG_CRYPTO_LIB_SHA256) += libsha256.o libsha256-y := sha256.o +obj-$(CONFIG_CRYPTO_LIB_SHA256_GENERIC) += libsha256-generic.o +libsha256-generic-y := sha256-generic.o + ifneq ($(CONFIG_CRYPTO_MANAGER_DISABLE_TESTS),y) libblake2s-y += blake2s-selftest.o libchacha20poly1305-y += chacha20poly1305-selftest.o libcurve25519-y += curve25519-selftest.o endif
lib/crypto/sha256-generic.c new file +137 -0
137 lines not on this page · raw message
lib/crypto/sha256.c +87 -109
246 lines not on this page · raw message
Signature · 1 line
2.49.0
Herbert Xu
maintainerRe: [PATCH v4 01/13] crypto: sha256 - support arch-optimized lib and expose through shash
Quoted 27 lines · Eric Biggers
From: Eric Biggers <[email protected]>
As has been done for various other algorithms, rework the design of the SHA-256 library to support arch-optimized implementations, and make
crypto/sha256.cexpose both generic and arch-optimized shash algorithms that wrap the library functions.This allows users of the SHA-256 library functions to take advantage of the arch-optimized code, and this makes it much simpler to integrate SHA-256 for each architecture.
Note that
sha256_base.his not used in the new design. It will be removed once all the architecture-specific code has been updated.Move the generic block function into its own module to avoid a circular dependency from libsha256.ko => sha256-$ARCH.ko => libsha256.ko.
Signed-off-by Eric Biggers <[email protected]>--- crypto/Kconfig | 1 + crypto/Makefile | 3 +- crypto/sha256.c | 201 +++++++++++++++++++++++++++++++++ crypto/sha256_generic.c | 102 ----------------- include/crypto/internal/sha2.h | 28 +++++ include/crypto/sha2.h | 15 +-- include/crypto/sha256_base.h | 9 +- lib/crypto/Kconfig | 19 ++++
Quoted 8 lines
lib/crypto/Makefile | 3 + lib/crypto/sha256-generic.c | 137 ++++++++++++++++++++++ lib/crypto/sha256.c | 196 ++++++++++++++------------------ 11 files changed, 487 insertions(+), 227 deletions(-) create mode 100644 crypto/sha256.c delete mode 100644 crypto/sha256_generic.c create mode 100644 include/crypto/internal/sha2.h create mode 100644 lib/crypto/sha256-generic.c
This is the patch that I will fold in here to maintain the existing export format:
crypto/sha256.c +42 -0
@@ -116,6 +116,32 @@ static int crypto_sha224_final_arch(struct shash_desc *desc, u8 *out)return 0 ; } +static int crypto_sha256_import_lib (struct shash_desc *desc,const void *in) +{ +struct sha256_state *sctx =shash_desc_ctx (desc); +const u8 *p = in; + +memcpy (sctx, p,sizeof (*sctx)); + p +=sizeof (*sctx); + sctx->count += *p; +return 0 ; +} + +static int crypto_sha256_export_lib (struct shash_desc *desc,void *out) +{ +struct sha256_state *sctx0 =shash_desc_ctx (desc); +struct sha256_state sctx = *sctx0; +unsigned int partial; +u8 *p = out; + + partial = sctx.count %SHA256_BLOCK_SIZE ; + sctx.count -= partial; +memcpy (p, &sctx,sizeof (sctx)); + p +=sizeof (sctx); + *p = partial; +return 0 ; +} +static struct shash_alg algs[] = { { .base.cra_name ="sha256" ,
@@ -130,6 +156,10 @@ static struct shash_alg algs[] = { .finup = crypto_sha256_finup_generic, .digest = crypto_sha256_digest_generic, .descsize = sizeof(struct sha256_state), + .statesize =sizeof (struct crypto_sha256_state ) + + SHA256_BLOCK_SIZE + 1, + .import = crypto_sha256_import_lib, + .export = crypto_sha256_export_lib, }, { .base.cra_name = "sha224",
@@ -142,6 +172,10 @@ static struct shash_alg algs[] = { .update = crypto_sha256_update_generic, .final = crypto_sha224_final_generic, .descsize = sizeof(struct sha256_state), + .statesize =sizeof (struct crypto_sha256_state ) + + SHA256_BLOCK_SIZE + 1, + .import = crypto_sha256_import_lib, + .export = crypto_sha256_export_lib, }, { .base.cra_name = "sha256",
@@ -156,6 +190,10 @@ static struct shash_alg algs[] = { .finup = crypto_sha256_finup_arch, .digest = crypto_sha256_digest_arch, .descsize = sizeof(struct sha256_state), + .statesize =sizeof (struct crypto_sha256_state ) + + SHA256_BLOCK_SIZE + 1, + .import = crypto_sha256_import_lib, + .export = crypto_sha256_export_lib, }, { .base.cra_name = "sha224",
@@ -168,6 +206,10 @@ static struct shash_alg algs[] = { .update = crypto_sha256_update_arch, .final = crypto_sha224_final_arch, .descsize = sizeof(struct sha256_state), + .statesize = sizeof(struct crypto_sha256_state) + + SHA256_BLOCK_SIZE + 1, + .import = crypto_sha256_import_lib, + .export = crypto_sha256_export_lib, }, };
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Eric Biggers
maintainerPATCH v4 2/13 crypto: arm/sha256 - implement library instead of shash
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Instead of providing crypto_shash algorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available through crypto_shash, but individual architectures no longer need to handle it.
To merge the scalar, NEON, and CE code all into one module cleanly, add !CPU_V7M as a direct dependency of the CE code. Previously, !CPU_V7M was only a direct dependency of the scalar and NEON code. The result is still the same because CPU_V7M implies !KERNEL_MODE_NEON, so !CPU_V7M was already an indirect dependency of the CE code.
To match sha256_blocks_arch(), change the type of the nblocks parameter of the assembly functions from int to size_t. The assembly functions actually already treated it as size_t.
While renaming the assembly files, also fix the naming quirk where "sha2" meant sha256. (SHA-512 is also part of SHA-2.)
---
arch/arm/configs/exynos_defconfig | 1 -
arch/arm/configs/milbeaut_m10v_defconfig | 1 -
arch/arm/configs/multi_v7_defconfig | 1 -
arch/arm/configs/omap2plus_defconfig | 1 -
arch/arm/configs/pxa_defconfig | 1 -
arch/arm/crypto/Kconfig | 21 ----
arch/arm/crypto/Makefile | 8 +-
arch/arm/crypto/sha2-ce-glue.c | 87 --------------
arch/arm/crypto/sha256_glue.c | 107 ------------------
arch/arm/crypto/sha256_glue.h | 9 --
arch/arm/crypto/sha256_neon_glue.c | 75 ------------
arch/arm/lib/crypto/.gitignore | 1 +
arch/arm/lib/crypto/Kconfig | 6 +
arch/arm/lib/crypto/Makefile | 8 +-
arch/arm/{ => lib}/crypto/sha256-armv4.pl | 0
.../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 10 +-
arch/arm/lib/crypto/sha256.c | 64 +++++++++++
17 files changed, 84 insertions(+), 317 deletions(-)
delete mode 100644 arch/arm/crypto/sha2-ce-glue.c
delete mode 100644 arch/arm/crypto/sha256_glue.c
delete mode 100644 arch/arm/crypto/sha256_glue.h
delete mode 100644 arch/arm/crypto/sha256_neon_glue.c
rename arch/arm/{ => lib}/crypto/sha256-armv4.pl (100%)
rename arch/arm/{crypto/sha2-ce-core.S => lib/crypto/sha256-ce.S} (91%)
create mode 100644 arch/arm/lib/crypto/sha256.carch/arm/configs/exynos_defconfig +0 -1
@@ -362,11 +362,10 @@ CONFIG_CRYPTO_LZ4=m CONFIG_CRYPTO_USER_API_HASH=m CONFIG_CRYPTO_USER_API_SKCIPHER=m CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_USER_API_AEAD=m CONFIG_CRYPTO_SHA1_ARM_NEON=m -CONFIG_CRYPTO_SHA256_ARM=m CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRYPTO_DEV_EXYNOS_RNG=y CONFIG_CRYPTO_DEV_S5P=y
arch/arm/configs/milbeaut_m10v_defconfig +0 -1
@@ -99,11 +99,10 @@ CONFIG_CRYPTO_MANAGER=y CONFIG_CRYPTO_AES=y CONFIG_CRYPTO_SEQIV=m CONFIG_CRYPTO_GHASH_ARM_CE=m CONFIG_CRYPTO_SHA1_ARM_NEON=m CONFIG_CRYPTO_SHA1_ARM_CE=m -CONFIG_CRYPTO_SHA2_ARM_CE=m CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_AES_ARM_CE=m CONFIG_CRYPTO_CHACHA20_NEON=m
arch/arm/configs/multi_v7_defconfig +0 -1
@@ -1299,11 +1299,10 @@ CONFIG_CRYPTO_USER_API_SKCIPHER=m CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_USER_API_AEAD=m CONFIG_CRYPTO_GHASH_ARM_CE=m CONFIG_CRYPTO_SHA1_ARM_NEON=m CONFIG_CRYPTO_SHA1_ARM_CE=m -CONFIG_CRYPTO_SHA2_ARM_CE=m CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_AES_ARM_CE=m CONFIG_CRYPTO_CHACHA20_NEON=m
arch/arm/configs/omap2plus_defconfig +0 -1
@@ -695,11 +695,10 @@ CONFIG_NLS_CODEPAGE_437=y CONFIG_NLS_ISO8859_1=y CONFIG_SECURITY=y CONFIG_CRYPTO_MICHAEL_MIC=y CONFIG_CRYPTO_GHASH_ARM_CE=m CONFIG_CRYPTO_SHA1_ARM_NEON=m -CONFIG_CRYPTO_SHA256_ARM=m CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRYPTO_DEV_OMAP=m
arch/arm/configs/pxa_defconfig +0 -1
@@ -658,11 +658,10 @@ CONFIG_CRYPTO_WP512=m CONFIG_CRYPTO_ANUBIS=m CONFIG_CRYPTO_XCBC=m CONFIG_CRYPTO_DEFLATE=y CONFIG_CRYPTO_LZO=y CONFIG_CRYPTO_SHA1_ARM=m -CONFIG_CRYPTO_SHA256_ARM=m CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRC_CCITT=y CONFIG_CRC_T10DIF=m CONFIG_FONTS=y
arch/arm/crypto/Kconfig +0 -21
@@ -91,31 +91,10 @@ config CRYPTO_SHA1_ARM_CE help SHA-1 secure hash algorithm (FIPS 180) Architecture: arm using ARMv8 Crypto Extensions -config CRYPTO_SHA2_ARM_CE - tristate "Hash functions: SHA-224 and SHA-256 (ARMv8 Crypto Extensions)" - depends on KERNEL_MODE_NEON - select CRYPTO_SHA256_ARM - select CRYPTO_HASH - help - SHA-224 and SHA-256 secure hash algorithms (FIPS 180) - - Architecture: arm using - - ARMv8 Crypto Extensions - -config CRYPTO_SHA256_ARM - tristate "Hash functions: SHA-224 and SHA-256 (NEON)" - select CRYPTO_HASH - depends on !CPU_V7M - help - SHA-224 and SHA-256 secure hash algorithms (FIPS 180) - - Architecture: arm using - - NEON (Advanced SIMD) extensions - config CRYPTO_SHA512_ARM tristate "Hash functions: SHA-384 and SHA-512 (NEON)" select CRYPTO_HASH depends on !CPU_V7M help
arch/arm/crypto/Makefile +1 -7
@@ -5,32 +5,27 @@ obj-$(CONFIG_CRYPTO_AES_ARM) += aes-arm.o obj-$(CONFIG_CRYPTO_AES_ARM_BS) += aes-arm-bs.o obj-$(CONFIG_CRYPTO_SHA1_ARM) += sha1-arm.o obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o -obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o obj-$(CONFIG_CRYPTO_BLAKE2B_NEON) += blake2b-neon.o obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) += nhpoly1305-neon.o obj-$(CONFIG_CRYPTO_CURVE25519_NEON) += curve25519-neon.o obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o obj-$(CONFIG_CRYPTO_SHA1_ARM_CE) += sha1-arm-ce.o -obj-$(CONFIG_CRYPTO_SHA2_ARM_CE) += sha2-arm-ce.o obj-$(CONFIG_CRYPTO_GHASH_ARM_CE) += ghash-arm-ce.o aes-arm-y := aes-cipher-core.o aes-cipher-glue.o aes-arm-bs-y := aes-neonbs-core.o aes-neonbs-glue.o sha1-arm-y := sha1-armv4-large.o sha1_glue.o sha1-arm-neon-y := sha1-armv7-neon.o sha1_neon_glue.o -sha256-arm-neon-$(CONFIG_KERNEL_MODE_NEON) := sha256_neon_glue.o -sha256-arm-y := sha256-core.o sha256_glue.o $(sha256-arm-neon-y) sha512-arm-neon-$(CONFIG_KERNEL_MODE_NEON) := sha512-neon-glue.o sha512-arm-y := sha512-core.o sha512-glue.o $(sha512-arm-neon-y) blake2b-neon-y := blake2b-neon-core.o blake2b-neon-glue.o sha1-arm-ce-y := sha1-ce-core.o sha1-ce-glue.o -sha2-arm-ce-y := sha2-ce-core.o sha2-ce-glue.o aes-arm-ce-y := aes-ce-core.o aes-ce-glue.o ghash-arm-ce-y := ghash-ce-core.o ghash-ce-glue.o nhpoly1305-neon-y := nh-neon-core.o nhpoly1305-neon-glue.o curve25519-neon-y := curve25519-core.o curve25519-glue.o
@@ -38,11 +33,10 @@ quiet_cmd_perl = PERL $@ cmd_perl = $(PERL) $(<) > $(@) $(obj)/%-core.S: $(src)/%-armv4.pl $(call cmd,perl) -clean-files += sha256-core.S sha512-core.S +clean-files += sha512-core.S aflags-thumb2-$(CONFIG_THUMB2_KERNEL) := -U__thumb2__ -D__thumb2__=1 -AFLAGS_sha256-core.o += $(aflags-thumb2-y) AFLAGS_sha512-core.o += $(aflags-thumb2-y)
arch/arm/crypto/sha2-ce-glue.c deleted +0 -87
87 lines not on this page · raw message
arch/arm/crypto/sha256_glue.c deleted +0 -107
107 lines not on this page · raw message
arch/arm/crypto/sha256_glue.h deleted +0 -9
@@ -1,9 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _CRYPTO_SHA256_GLUE_H -#define _CRYPTO_SHA256_GLUE_H - -#include <crypto/hash.h> - -extern struct shash_alg sha256_neon_algs[2 ]; - -#endif /* _CRYPTO_SHA256_GLUE_H */
arch/arm/crypto/sha256_neon_glue.c deleted +0 -75
75 lines not on this page · raw message
arch/arm/lib/crypto/.gitignore +1 -0
@@ -1,2 +1,3 @@ # SPDX-License-Identifier: GPL-2.0-only poly1305-core.S +sha256-core.S
arch/arm/lib/crypto/Kconfig +6 -0
@@ -20,5 +20,11 @@ config CRYPTO_CHACHA20_NEON config CRYPTO_POLY1305_ARM tristate default CRYPTO_LIB_POLY1305 select CRYPTO_ARCH_HAVE_LIB_POLY1305 + +config CRYPTO_SHA256_ARM + tristate + depends on !CPU_V7M + default CRYPTO_LIB_SHA256 + select CRYPTO_ARCH_HAVE_LIB_SHA256
arch/arm/lib/crypto/Makefile +7 -1
@@ -8,19 +8,25 @@ chacha-neon-y := chacha-scalar-core.o chacha-glue.o chacha-neon-$(CONFIG_KERNEL_MODE_NEON) += chacha-neon-core.o obj-$(CONFIG_CRYPTO_POLY1305_ARM) += poly1305-arm.o poly1305-arm-y := poly1305-core.o poly1305-glue.o +obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o +sha256-arm-y := sha256.o sha256-core.o +sha256-arm-$(CONFIG_KERNEL_MODE_NEON) += sha256-ce.o + quiet_cmd_perl = PERL $@ cmd_perl = $(PERL) $(<) > $(@) $(obj)/%-core.S: $(src)/%-armv4.pl $(call cmd,perl) -clean-files += poly1305-core.S +clean-files += poly1305-core.S sha256-core.S aflags-thumb2-$(CONFIG_THUMB2_KERNEL) := -U__thumb2__ -D__thumb2__=1 # massage the perlasm code a bit so we only get the NEON routine if we need it poly1305-aflags-$(CONFIG_CPU_V7) := -U__LINUX_ARM_ARCH__ -D__LINUX_ARM_ARCH__=5 poly1305-aflags-$(CONFIG_KERNEL_MODE_NEON) := -U__LINUX_ARM_ARCH__ -D__LINUX_ARM_ARCH__=7 AFLAGS_poly1305-core.o += $(poly1305-aflags-y) $(aflags-thumb2-y) + +AFLAGS_sha256-core.o += $(aflags-thumb2-y)
arch/arm/lib/crypto/sha256-armv4.pl renamed from arch/arm/crypto/sha256-armv4.pl +0 -0
arch/arm/lib/crypto/sha256-ce.S renamed from arch/arm/crypto/sha2-ce-core.S +5 -5
@@ -1,8 +1,8 @@ /* SPDX-License-Identifier: GPL-2.0-only */ /* - * sha2-ce-core.S - SHA-224/256 secure hash using ARMv8 Crypto Extensions + * sha256-ce.S - SHA-224/256 secure hash using ARMv8 Crypto Extensions * * Copyright (C) 2015 Linaro Ltd. * Author: Ard Biesheuvel <[email protected]> */
@@ -65,14 +65,14 @@ .word 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3 .word 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208 .word 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2 /* - * void sha2_ce_transform(struct sha256_state *sst, u8 const *src, - int blocks); + * void sha256_ce_transform(u32 state[SHA256_STATE_WORDS], + * const u8 *data, size_t nblocks); */ -ENTRY(sha2_ce_transform) +ENTRY(sha256_ce_transform) /* load state */ vld1.32 {dga-dgb}, [r0] /* load input */ 0: vld1.32 {q0-q1}, [r1]!
@@ -118,6 +118,6 @@ ENTRY(sha2_ce_transform) bne 0b /* store new state */ vst1.32 {dga-dgb}, [r0] bx lr -ENDPROC(sha2_ce_transform) +ENDPROC(sha256_ce_transform)
arch/arm/lib/crypto/sha256.c new file +64 -0
64 lines not on this page · raw message
Signature · 1 line
2.49.0
Eric Biggers
maintainerPATCH v4 3/13 crypto: arm64/sha256 - remove obsolete chunking logic
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Since kernel-mode NEON sections are now preemptible on arm64, there is no longer any need to limit the length of them.
--- arch/arm64/crypto/sha256-glue.c | 19 ++----------------- 1 file changed, 2 insertions(+), 17 deletions(-)
arch/arm64/crypto/sha256-glue.c +2 -17
@@ -84,27 +84,12 @@ static struct shash_alg algs[] = { { } };static int sha256_update_neon (struct shash_desc *desc,const u8 *data,unsigned int len) { -do { -unsigned int chunk = len; - -/* -* Don't hog the CPU for the entire time it takes to process all -* input when running on a preemptible kernel, but process the -* data block by block instead. -*/ -if (IS_ENABLED (CONFIG_PREEMPTION )) - chunk =SHA256_BLOCK_SIZE ; - - chunk -=sha256_base_do_update_blocks (desc, data, chunk, - sha256_neon_transform); - data += chunk; - len -= chunk; - }while (len >=SHA256_BLOCK_SIZE ); -return len; +return sha256_base_do_update_blocks (desc, data, len, + sha256_neon_transform); }static int sha256_finup_neon (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) {
Signature · 1 line
2.49.0
Eric Biggers
maintainerPATCH v4 4/13 crypto: arm64/sha256 - implement library instead of shash
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Instead of providing crypto_shash algorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available through crypto_shash, but individual architectures no longer need to handle it.
Remove support for SHA-256 finalization from the ARMv8 CE assembly code, since the library does not yet support architecture-specific overrides of the finalization. (Support for that has been omitted for now, for simplicity and because usually it isn't performance-critical.)
To match sha256_blocks_arch(), change the type of the nblocks parameter of the assembly functions from int or 'unsigned int' to size_t. Update the ARMv8 CE assembly function accordingly. The scalar and NEON assembly functions actually already treated it as size_t.
While renaming the assembly files, also fix the naming quirks where "sha2" meant sha256, and "sha512" meant both sha256 and sha512.
---
arch/arm64/configs/defconfig | 1 -
arch/arm64/crypto/Kconfig | 19 ---
arch/arm64/crypto/Makefile | 13 +-
arch/arm64/crypto/sha2-ce-glue.c | 138 ----------------
arch/arm64/crypto/sha256-glue.c | 156 ------------------
arch/arm64/lib/crypto/.gitignore | 1 +
arch/arm64/lib/crypto/Kconfig | 5 +
arch/arm64/lib/crypto/Makefile | 9 +-
.../crypto/sha2-armv8.pl} | 0
.../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 36 +---
arch/arm64/lib/crypto/sha256.c | 75 +++++++++
11 files changed, 98 insertions(+), 355 deletions(-)
delete mode 100644 arch/arm64/crypto/sha2-ce-glue.c
delete mode 100644 arch/arm64/crypto/sha256-glue.c
rename arch/arm64/{crypto/sha512-armv8.pl => lib/crypto/sha2-armv8.pl} (100%)
rename arch/arm64/{crypto/sha2-ce-core.S => lib/crypto/sha256-ce.S} (80%)
create mode 100644 arch/arm64/lib/crypto/sha256.carch/arm64/configs/defconfig +0 -1
@@ -1735,11 +1735,10 @@ CONFIG_CRYPTO_MICHAEL_MIC=m CONFIG_CRYPTO_ANSI_CPRNG=y CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRYPTO_GHASH_ARM64_CE=y CONFIG_CRYPTO_SHA1_ARM64_CE=y -CONFIG_CRYPTO_SHA2_ARM64_CE=y CONFIG_CRYPTO_SHA512_ARM64_CE=m CONFIG_CRYPTO_SHA3_ARM64=m CONFIG_CRYPTO_SM3_ARM64_CE=m CONFIG_CRYPTO_AES_ARM64_CE_BLK=y CONFIG_CRYPTO_AES_ARM64_BS=m
arch/arm64/crypto/Kconfig +0 -19
@@ -34,29 +34,10 @@ config CRYPTO_SHA1_ARM64_CE SHA-1 secure hash algorithm (FIPS 180) Architecture: arm64 using: - ARMv8 Crypto Extensions -config CRYPTO_SHA256_ARM64 - tristate "Hash functions: SHA-224 and SHA-256" - select CRYPTO_HASH - help - SHA-224 and SHA-256 secure hash algorithms (FIPS 180) - - Architecture: arm64 - -config CRYPTO_SHA2_ARM64_CE - tristate "Hash functions: SHA-224 and SHA-256 (ARMv8 Crypto Extensions)" - depends on KERNEL_MODE_NEON - select CRYPTO_HASH - select CRYPTO_SHA256_ARM64 - help - SHA-224 and SHA-256 secure hash algorithms (FIPS 180) - - Architecture: arm64 using: - - ARMv8 Crypto Extensions - config CRYPTO_SHA512_ARM64 tristate "Hash functions: SHA-384 and SHA-512" select CRYPTO_HASH help SHA-384 and SHA-512 secure hash algorithms (FIPS 180)
arch/arm64/crypto/Makefile +2 -11
@@ -6,13 +6,10 @@ # obj-$(CONFIG_CRYPTO_SHA1_ARM64_CE) += sha1-ce.o sha1-ce-y := sha1-ce-glue.o sha1-ce-core.o -obj-$(CONFIG_CRYPTO_SHA2_ARM64_CE) += sha2-ce.o -sha2-ce-y := sha2-ce-glue.o sha2-ce-core.o - obj-$(CONFIG_CRYPTO_SHA512_ARM64_CE) += sha512-ce.o sha512-ce-y := sha512-ce-glue.o sha512-ce-core.o obj-$(CONFIG_CRYPTO_SHA3_ARM64) += sha3-ce.o sha3-ce-y := sha3-ce-glue.o sha3-ce-core.o
@@ -54,13 +51,10 @@ obj-$(CONFIG_CRYPTO_AES_ARM64_CE_BLK) += aes-ce-blk.o aes-ce-blk-y := aes-glue-ce.o aes-ce.o obj-$(CONFIG_CRYPTO_AES_ARM64_NEON_BLK) += aes-neon-blk.o aes-neon-blk-y := aes-glue-neon.o aes-neon.o -obj-$(CONFIG_CRYPTO_SHA256_ARM64) += sha256-arm64.o -sha256-arm64-y := sha256-glue.o sha256-core.o - obj-$(CONFIG_CRYPTO_SHA512_ARM64) += sha512-arm64.o sha512-arm64-y := sha512-glue.o sha512-core.o obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) += nhpoly1305-neon.o nhpoly1305-neon-y := nh-neon-core.o nhpoly1305-neon-glue.o
@@ -72,12 +66,9 @@ obj-$(CONFIG_CRYPTO_AES_ARM64_BS) += aes-neon-bs.o aes-neon-bs-y := aes-neonbs-core.o aes-neonbs-glue.o quiet_cmd_perlasm = PERLASM $@ cmd_perlasm = $(PERL) $(<) void $(@) -$(obj)/%-core.S: $(src)/%-armv8.pl - $(call cmd,perlasm) - -$(obj)/sha256-core.S: $(src)/sha512-armv8.pl +$(obj)/sha512-core.S: $(src)/../lib/crypto/sha2-armv8.pl $(call cmd,perlasm) -clean-files += sha256-core.S sha512-core.S +clean-files += sha512-core.S
arch/arm64/crypto/sha2-ce-glue.c deleted +0 -138
138 lines not on this page · raw message
arch/arm64/crypto/sha256-glue.c deleted +0 -156
156 lines not on this page · raw message
arch/arm64/lib/crypto/.gitignore +1 -0
@@ -1,2 +1,3 @@ # SPDX-License-Identifier: GPL-2.0-only poly1305-core.S +sha256-core.S
arch/arm64/lib/crypto/Kconfig +5 -0
@@ -10,5 +10,10 @@ config CRYPTO_CHACHA20_NEON config CRYPTO_POLY1305_NEON tristate depends on KERNEL_MODE_NEON default CRYPTO_LIB_POLY1305 select CRYPTO_ARCH_HAVE_LIB_POLY1305 + +config CRYPTO_SHA256_ARM64 + tristate + default CRYPTO_LIB_SHA256 + select CRYPTO_ARCH_HAVE_LIB_SHA256
arch/arm64/lib/crypto/Makefile +8 -1
@@ -5,12 +5,19 @@ chacha-neon-y := chacha-neon-core.o chacha-neon-glue.o obj-$(CONFIG_CRYPTO_POLY1305_NEON) += poly1305-neon.o poly1305-neon-y := poly1305-core.o poly1305-glue.o AFLAGS_poly1305-core.o += -Dpoly1305_init=poly1305_init_arm64 +obj-$(CONFIG_CRYPTO_SHA256_ARM64) += sha256-arm64.o +sha256-arm64-y := sha256.o sha256-core.o +sha256-arm64-$(CONFIG_KERNEL_MODE_NEON) += sha256-ce.o + quiet_cmd_perlasm = PERLASM $@ cmd_perlasm = $(PERL) $(<) void $(@) $(obj)/%-core.S: $(src)/%-armv8.pl $(call cmd,perlasm) -clean-files += poly1305-core.S +$(obj)/sha256-core.S: $(src)/sha2-armv8.pl + $(call cmd,perlasm) + +clean-files += poly1305-core.S sha256-core.S
arch/arm64/lib/crypto/sha2-armv8.pl renamed from arch/arm64/crypto/sha512-armv8.pl +0 -0
arch/arm64/lib/crypto/sha256-ce.S renamed from arch/arm64/crypto/sha2-ce-core.S +7 -29
@@ -69,12 +69,12 @@ .word 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3 .word 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208 .word 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2 /* - * int __sha256_ce_transform(struct sha256_ce_state *sst, u8 const *src, - * int blocks) + * size_t __sha256_ce_transform(u32 state[SHA256_STATE_WORDS], + * const u8 *data, size_t nblocks); */ .text SYM_FUNC_START(__sha256_ce_transform) /* load round constants */ adr_l x8, .Lsha2_rcon
@@ -84,24 +84,20 @@ SYM_FUNC_START(__sha256_ce_transform) ld1 {v12.4s-v15.4s}, [x8] /* load state */ ld1 {dgav.4s, dgbv.4s}, [x0] - /* load sha256_ce_state::finalize */ - ldr_l w4, sha256_ce_offsetof_finalize, x4 - ldr w4, [x0, x4] - /* load input */ 0: ld1 {v16.4s-v19.4s}, [x1], #64 - sub w2, w2, #1 + sub x2, x2, #1 CPU_LE( rev32 v16.16b, v16.16b ) CPU_LE( rev32 v17.16b, v17.16b ) CPU_LE( rev32 v18.16b, v18.16b ) CPU_LE( rev32 v19.16b, v19.16b ) -1: add t0.4s, v16.4s, v0.4s + add t0.4s, v16.4s, v0.4s mov dg0v.16b, dgav.16b mov dg1v.16b, dgbv.16b add_update 0, v1, 16, 17, 18, 19 add_update 1, v2, 17, 18, 19, 16
@@ -126,32 +122,14 @@ CPU_LE( rev32 v19.16b, v19.16b ) /* update state */ add dgav.4s, dgav.4s, dg0v.4s add dgbv.4s, dgbv.4s, dg1v.4s /* handled all input blocks? */ - cbz w2, 2f + cbz x2, 1f cond_yield 3f, x5, x6 b 0b - /* - * Final block: add padding and total bit count. - * Skip if the input size was not a round multiple of the block size, - * the padding is handled by the C code in that case. - */ -2: cbz x4, 3f - ldr_l w4, sha256_ce_offsetof_count, x4 - ldr x4, [x0, x4] - movi v17.2d, #0 - mov x8, #0x80000000 - movi v18.2d, #0 - ror x7, x4, #29 // ror(lsl(x4, 3), 32) - fmov d16, x8 - mov x4, #0 - mov v19.d[0], xzr - mov v19.d[1], x7 - b 1b - /* store new state */ -3: st1 {dgav.4s, dgbv.4s}, [x0] - mov w0, w2 +1: st1 {dgav.4s, dgbv.4s}, [x0] + mov x0, x2 ret SYM_FUNC_END(__sha256_ce_transform)
arch/arm64/lib/crypto/sha256.c new file +75 -0
75 lines not on this page · raw message
Signature · 1 line
2.49.0
Eric Biggers
maintainerPATCH v4 5/13 crypto: mips/sha256 - implement library instead of shash
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Instead of providing crypto_shash algorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available through crypto_shash, but individual architectures no longer need to handle it.
--- arch/mips/cavium-octeon/Kconfig | 6 + .../mips/cavium-octeon/crypto/octeon-sha256.c | 135 ++++-------------- arch/mips/configs/cavium_octeon_defconfig | 1 - arch/mips/crypto/Kconfig | 10 -- 4 files changed, 33 insertions(+), 119 deletions(-)
arch/mips/cavium-octeon/Kconfig +6 -0
@@ -21,10 +21,16 @@ config CAVIUM_OCTEON_CVMSEG_SIZE local memory; the larger CVMSEG is, the smaller the cache is. This selects the size of CVMSEG LM, which is in cache blocks. The legally range is from zero to 54 cache blocks (i.e. CVMSEG LM is between zero and 6192 bytes). +config CRYPTO_SHA256_OCTEON + tristate + default CRYPTO_LIB_SHA256 + select CRYPTO_ARCH_HAVE_LIB_SHA256 + select CRYPTO_LIB_SHA256_GENERIC + endif # CPU_CAVIUM_OCTEON if CAVIUM_OCTEON_SOC config CAVIUM_OCTEON_LOCK_L2
arch/mips/cavium-octeon/crypto/octeon-sha256.c +27 -108
179 lines not on this page · raw message
arch/mips/configs/cavium_octeon_defconfig +0 -1
@@ -155,11 +155,10 @@ CONFIG_SECURITY=y CONFIG_SECURITY_NETWORK=y CONFIG_CRYPTO_CBC=y CONFIG_CRYPTO_HMAC=y CONFIG_CRYPTO_MD5_OCTEON=y CONFIG_CRYPTO_SHA1_OCTEON=m -CONFIG_CRYPTO_SHA256_OCTEON=m CONFIG_CRYPTO_SHA512_OCTEON=m CONFIG_CRYPTO_DES=y CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y CONFIG_DEBUG_FS=y CONFIG_MAGIC_SYSRQ=y
arch/mips/crypto/Kconfig +0 -10
@@ -20,20 +20,10 @@ config CRYPTO_SHA1_OCTEON help SHA-1 secure hash algorithm (FIPS 180) Architecture: mips OCTEON -config CRYPTO_SHA256_OCTEON - tristate "Hash functions: SHA-224 and SHA-256 (OCTEON)" - depends on CPU_CAVIUM_OCTEON - select CRYPTO_SHA256 - select CRYPTO_HASH - help - SHA-224 and SHA-256 secure hash algorithms (FIPS 180) - - Architecture: mips OCTEON using crypto instructions, when available - config CRYPTO_SHA512_OCTEON tristate "Hash functions: SHA-384 and SHA-512 (OCTEON)" depends on CPU_CAVIUM_OCTEON select CRYPTO_SHA512 select CRYPTO_HASH
Signature · 1 line
2.49.0
Eric Biggers
maintainerPATCH v4 6/13 crypto: powerpc/sha256 - implement library instead of shash
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Instead of providing crypto_shash algorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available through crypto_shash, but individual architectures no longer need to handle it.
---
arch/powerpc/crypto/Kconfig | 11 --
arch/powerpc/crypto/Makefile | 2 -
arch/powerpc/crypto/sha256-spe-glue.c | 128 ------------------
arch/powerpc/lib/crypto/Kconfig | 6 +
arch/powerpc/lib/crypto/Makefile | 3 +
.../powerpc/{ => lib}/crypto/sha256-spe-asm.S | 0
arch/powerpc/lib/crypto/sha256.c | 70 ++++++++++
7 files changed, 79 insertions(+), 141 deletions(-)
delete mode 100644 arch/powerpc/crypto/sha256-spe-glue.c
rename arch/powerpc/{ => lib}/crypto/sha256-spe-asm.S (100%)
create mode 100644 arch/powerpc/lib/crypto/sha256.carch/powerpc/crypto/Kconfig +0 -11
@@ -37,21 +37,10 @@ config CRYPTO_SHA1_PPC_SPE SHA-1 secure hash algorithm (FIPS 180) Architecture: powerpc using - SPE (Signal Processing Engine) extensions -config CRYPTO_SHA256_PPC_SPE - tristate "Hash functions: SHA-224 and SHA-256 (SPE)" - depends on SPE - select CRYPTO_SHA256 - select CRYPTO_HASH - help - SHA-224 and SHA-256 secure hash algorithms (FIPS 180) - - Architecture: powerpc using - - SPE (Signal Processing Engine) extensions - config CRYPTO_AES_PPC_SPE tristate "Ciphers: AES, modes: ECB/CBC/CTR/XTS (SPE)" depends on SPE select CRYPTO_SKCIPHER help
arch/powerpc/crypto/Makefile +0 -2
@@ -7,20 +7,18 @@ obj-$(CONFIG_CRYPTO_AES_PPC_SPE) += aes-ppc-spe.o obj-$(CONFIG_CRYPTO_MD5_PPC) += md5-ppc.o obj-$(CONFIG_CRYPTO_SHA1_PPC) += sha1-powerpc.o obj-$(CONFIG_CRYPTO_SHA1_PPC_SPE) += sha1-ppc-spe.o -obj-$(CONFIG_CRYPTO_SHA256_PPC_SPE) += sha256-ppc-spe.o obj-$(CONFIG_CRYPTO_AES_GCM_P10) += aes-gcm-p10-crypto.o obj-$(CONFIG_CRYPTO_DEV_VMX_ENCRYPT) += vmx-crypto.o obj-$(CONFIG_CRYPTO_CURVE25519_PPC64) += curve25519-ppc64le.o aes-ppc-spe-y := aes-spe-core.o aes-spe-keys.o aes-tab-4k.o aes-spe-modes.o aes-spe-glue.o md5-ppc-y := md5-asm.o md5-glue.o sha1-powerpc-y := sha1-powerpc-asm.o sha1.o sha1-ppc-spe-y := sha1-spe-asm.o sha1-spe-glue.o -sha256-ppc-spe-y := sha256-spe-asm.o sha256-spe-glue.o aes-gcm-p10-crypto-y := aes-gcm-p10-glue.o aes-gcm-p10.o ghashp10-ppc.o aesp10-ppc.o vmx-crypto-objs := vmx.o aesp8-ppc.o ghashp8-ppc.o aes.o aes_cbc.o aes_ctr.o aes_xts.o ghash.o curve25519-ppc64le-y := curve25519-ppc64le-core.o curve25519-ppc64le_asm.o ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y)
arch/powerpc/crypto/sha256-spe-glue.c deleted +0 -128
128 lines not on this page · raw message
arch/powerpc/lib/crypto/Kconfig +6 -0
@@ -11,5 +11,11 @@ config CRYPTO_POLY1305_P10 tristate depends on PPC64 && CPU_LITTLE_ENDIAN && VSX default CRYPTO_LIB_POLY1305 select CRYPTO_ARCH_HAVE_LIB_POLY1305 select CRYPTO_LIB_POLY1305_GENERIC + +config CRYPTO_SHA256_PPC_SPE + tristate + depends on SPE + default CRYPTO_LIB_SHA256 + select CRYPTO_ARCH_HAVE_LIB_SHA256
arch/powerpc/lib/crypto/Makefile +3 -0
@@ -3,5 +3,8 @@ obj-$(CONFIG_CRYPTO_CHACHA20_P10) += chacha-p10-crypto.o chacha-p10-crypto-y := chacha-p10-glue.o chacha-p10le-8x.o obj-$(CONFIG_CRYPTO_POLY1305_P10) += poly1305-p10-crypto.o poly1305-p10-crypto-y := poly1305-p10-glue.o poly1305-p10le_64.o + +obj-$(CONFIG_CRYPTO_SHA256_PPC_SPE) += sha256-ppc-spe.o +sha256-ppc-spe-y := sha256.o sha256-spe-asm.o
arch/powerpc/lib/crypto/sha256-spe-asm.S renamed from arch/powerpc/crypto/sha256-spe-asm.S +0 -0
arch/powerpc/lib/crypto/sha256.c new file +70 -0
70 lines not on this page · raw message
Signature · 1 line
2.49.0
Eric Biggers
maintainerPATCH v4 7/13 crypto: riscv/sha256 - implement library instead of shash
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Instead of providing crypto_shash algorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available through crypto_shash, but individual architectures no longer need to handle it.
To match sha256_blocks_arch(), change the type of the nblocks parameter of the assembly function from int to size_t. The assembly function actually already treated it as size_t.
---
arch/riscv/crypto/Kconfig | 11 --
arch/riscv/crypto/Makefile | 3 -
arch/riscv/crypto/sha256-riscv64-glue.c | 125 ------------------
arch/riscv/lib/crypto/Kconfig | 7 +
arch/riscv/lib/crypto/Makefile | 3 +
.../sha256-riscv64-zvknha_or_zvknhb-zvkb.S | 4 +-
arch/riscv/lib/crypto/sha256.c | 62 +++++++++
7 files changed, 74 insertions(+), 141 deletions(-)
delete mode 100644 arch/riscv/crypto/sha256-riscv64-glue.c
rename arch/riscv/{ => lib}/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S (98%)
create mode 100644 arch/riscv/lib/crypto/sha256.carch/riscv/crypto/Kconfig +0 -11
@@ -26,21 +26,10 @@ config CRYPTO_GHASH_RISCV64 GCM GHASH function (NIST SP 800-38D) Architecture: riscv64 using: - Zvkg vector crypto extension -config CRYPTO_SHA256_RISCV64 - tristate "Hash functions: SHA-224 and SHA-256" - depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO - select CRYPTO_SHA256 - help - SHA-224 and SHA-256 secure hash algorithm (FIPS 180) - - Architecture: riscv64 using: - - Zvknha or Zvknhb vector crypto extensions - - Zvkb vector crypto extension - config CRYPTO_SHA512_RISCV64 tristate "Hash functions: SHA-384 and SHA-512" depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO select CRYPTO_SHA512 help
arch/riscv/crypto/Makefile +0 -3
@@ -5,13 +5,10 @@ aes-riscv64-y := aes-riscv64-glue.o aes-riscv64-zvkned.o \ aes-riscv64-zvkned-zvbb-zvkg.o aes-riscv64-zvkned-zvkb.o obj-$(CONFIG_CRYPTO_GHASH_RISCV64) += ghash-riscv64.o ghash-riscv64-y := ghash-riscv64-glue.o ghash-riscv64-zvkg.o -obj-$(CONFIG_CRYPTO_SHA256_RISCV64) += sha256-riscv64.o -sha256-riscv64-y := sha256-riscv64-glue.o sha256-riscv64-zvknha_or_zvknhb-zvkb.o - obj-$(CONFIG_CRYPTO_SHA512_RISCV64) += sha512-riscv64.o sha512-riscv64-y := sha512-riscv64-glue.o sha512-riscv64-zvknhb-zvkb.o obj-$(CONFIG_CRYPTO_SM3_RISCV64) += sm3-riscv64.o sm3-riscv64-y := sm3-riscv64-glue.o sm3-riscv64-zvksh-zvkb.o
arch/riscv/crypto/sha256-riscv64-glue.c deleted +0 -125
125 lines not on this page · raw message
arch/riscv/lib/crypto/Kconfig +7 -0
@@ -4,5 +4,12 @@ config CRYPTO_CHACHA_RISCV64 tristate depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO default CRYPTO_LIB_CHACHA select CRYPTO_ARCH_HAVE_LIB_CHACHA select CRYPTO_LIB_CHACHA_GENERIC + +config CRYPTO_SHA256_RISCV64 + tristate + depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO + default CRYPTO_LIB_SHA256 + select CRYPTO_ARCH_HAVE_LIB_SHA256 + select CRYPTO_LIB_SHA256_GENERIC
arch/riscv/lib/crypto/Makefile +3 -0
@@ -1,4 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_CRYPTO_CHACHA_RISCV64) += chacha-riscv64.o chacha-riscv64-y := chacha-riscv64-glue.o chacha-riscv64-zvkb.o + +obj-$(CONFIG_CRYPTO_SHA256_RISCV64) += sha256-riscv64.o +sha256-riscv64-y := sha256.o sha256-riscv64-zvknha_or_zvknhb-zvkb.o
arch/riscv/lib/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S renamed from arch/riscv/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S +2 -2
@@ -104,12 +104,12 @@ sha256_4rounds \last, \k1, W1, W2, W3, W0 sha256_4rounds \last, \k2, W2, W3, W0, W1 sha256_4rounds \last, \k3, W3, W0, W1, W2 .endm -// void sha256_transform_zvknha_or_zvknhb_zvkb(u32 state[8], const u8 *data, -// int num_blocks); +// void sha256_transform_zvknha_or_zvknhb_zvkb(u32 state[SHA256_STATE_WORDS], +// const u8 *data, size_t nblocks); SYM_FUNC_START(sha256_transform_zvknha_or_zvknhb_zvkb) // Load the round constants into K0-K15. vsetivli zero, 4, e32, m1, ta, ma la t0, K256
arch/riscv/lib/crypto/sha256.c new file +62 -0
62 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH v4 07/13] crypto: riscv/sha256 - implement library instead of shash
Quoted 311 lines · On Mon, 28 Apr 2025 10:00:32 PDT (-0700), [email protected] wrote:
From: Eric Biggers <[email protected]>
Instead of providing
crypto_shashalgorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available throughcrypto_shash, but individual architectures no longer need to handle it.To match
sha256_blocks_arch(), change the type of the nblocks parameter of the assembly function from int tosize_t. The assembly function actually already treated it assize_t.Signed-off-by Eric Biggers <[email protected]>--- arch/riscv/crypto/Kconfig | 11 -- arch/riscv/crypto/Makefile | 3 - arch/riscv/crypto/sha256-riscv64-glue.c | 125 ------------------ arch/riscv/lib/crypto/Kconfig | 7 + arch/riscv/lib/crypto/Makefile | 3 + .../sha256-riscv64-zvknha_or_zvknhb-zvkb.S | 4 +- arch/riscv/lib/crypto/sha256.c | 62 +++++++++ 7 files changed, 74 insertions(+), 141 deletions(-) delete mode 100644 arch/riscv/crypto/sha256-riscv64-glue.c rename arch/riscv/{ => lib}/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S (98%) create mode 100644 arch/riscv/lib/crypto/sha256.carch/riscv/crypto/Kconfig +0
-11@@ -26,21 +26,10 @@ config CRYPTO_GHASH_RISCV64 GCM GHASH function (NIST SP 800-38D) Architecture: riscv64 using: - Zvkg vector crypto extension -config CRYPTO_SHA256_RISCV64 - tristate "Hash functions: SHA-224 and SHA-256" - depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO - select CRYPTO_SHA256 - help - SHA-224 and SHA-256 secure hash algorithm (FIPS 180) - - Architecture: riscv64 using: - - Zvknha or Zvknhb vector crypto extensions - - Zvkb vector crypto extension - config CRYPTO_SHA512_RISCV64 tristate "Hash functions: SHA-384 and SHA-512" depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO select CRYPTO_SHA512 helparch/riscv/crypto/Makefile +0
-3@@ -5,13 +5,10 @@ aes-riscv64-y := aes-riscv64-glue.o aes-riscv64-zvkned.o \ aes-riscv64-zvkned-zvbb-zvkg.o aes-riscv64-zvkned-zvkb.o obj-$(CONFIG_CRYPTO_GHASH_RISCV64) += ghash-riscv64.o ghash-riscv64-y := ghash-riscv64-glue.o ghash-riscv64-zvkg.o -obj-$(CONFIG_CRYPTO_SHA256_RISCV64) += sha256-riscv64.o -sha256-riscv64-y := sha256-riscv64-glue.o sha256-riscv64-zvknha_or_zvknhb-zvkb.o - obj-$(CONFIG_CRYPTO_SHA512_RISCV64) += sha512-riscv64.o sha512-riscv64-y := sha512-riscv64-glue.o sha512-riscv64-zvknhb-zvkb.o obj-$(CONFIG_CRYPTO_SM3_RISCV64) += sm3-riscv64.o sm3-riscv64-y := sm3-riscv64-glue.o sm3-riscv64-zvksh-zvkb.oarch/riscv/crypto/sha256-riscv64-glue.c deleted +0
-125@@ -1,125 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* -* SHA-256 and SHA-224 using the RISC-V vector crypto extensions -* -* Copyright (C) 2022 VRULL GmbH -* Author: Heiko Stuebner <[email protected]> -* -* Copyright (C) 2023 SiFive, Inc. -* Author: Jerry Shih <[email protected]> -*/ - -#include <asm/simd.h> -#include <asm/vector.h> -#include <crypto/internal/hash.h> -#include <crypto/internal/simd.h> -#include <crypto/sha256_base.h> -#include <linux/kernel.h> -#include <linux/module.h> - -/* -* Note: the asm function only uses the 'state' field of struct sha256_state. -* It is assumed to be the first field. -*/ -asmlinkage void sha256_transform_zvknha_or_zvknhb_zvkb( -struct crypto_sha256_state *state,const u8 *data,int num_blocks); - -static void sha256_block (struct crypto_sha256_state *state,const u8 *data, -int num_blocks) -{ -/* -* Ensure struct crypto_sha256_state begins directly with the SHA-256 -* 256-bit internal state, as this is what the asm function expects. -*/ -BUILD_BUG_ON (offsetof(struct crypto_sha256_state , state) !=0 ); - -if (crypto_simd_usable ()) { -kernel_vector_begin (); -sha256_transform_zvknha_or_zvknhb_zvkb (state, data, num_blocks); -kernel_vector_end (); - }else -sha256_transform_blocks (state, data, num_blocks); -} - -static int riscv64_sha256_update (struct shash_desc *desc,const u8 *data, -unsigned int len) -{ -return sha256_base_do_update_blocks (desc, data, len, sha256_block); -} - -static int riscv64_sha256_finup (struct shash_desc *desc,const u8 *data, -unsigned int len,u8 *out) -{ -sha256_base_do_finup (desc, data, len, sha256_block); -return sha256_base_finish (desc, out); -} - -static int riscv64_sha256_digest (struct shash_desc *desc,const u8 *data, -unsigned int len,u8 *out) -{ -return sha256_base_init (desc) ?: -riscv64_sha256_finup (desc, data, len, out); -} - -static struct shash_alg riscv64_sha256_algs[] = { - { - .init = sha256_base_init, - .update = riscv64_sha256_update, - .finup = riscv64_sha256_finup, - .digest = riscv64_sha256_digest, - .descsize =sizeof (struct crypto_sha256_state ), - .digestsize =SHA256_DIGEST_SIZE , - .base = { - .cra_blocksize =SHA256_BLOCK_SIZE , - .cra_flags =CRYPTO_AHASH_ALG_BLOCK_ONLY | -CRYPTO_AHASH_ALG_FINUP_MAX , - .cra_priority =300 , - .cra_name ="sha256" , - .cra_driver_name ="sha256-riscv64-zvknha_or_zvknhb-zvkb" , - .cra_module =THIS_MODULE , - }, - }, { - .init = sha224_base_init, - .update = riscv64_sha256_update, - .finup = riscv64_sha256_finup, - .descsize =sizeof (struct crypto_sha256_state ), - .digestsize =SHA224_DIGEST_SIZE , - .base = { - .cra_blocksize =SHA224_BLOCK_SIZE , - .cra_flags =CRYPTO_AHASH_ALG_BLOCK_ONLY | -CRYPTO_AHASH_ALG_FINUP_MAX , - .cra_priority =300 , - .cra_name ="sha224" , - .cra_driver_name ="sha224-riscv64-zvknha_or_zvknhb-zvkb" , - .cra_module =THIS_MODULE , - }, - }, -}; - -static int __init riscv64_sha256_mod_init (void ) -{ -/* Both zvknha and zvknhb provide the SHA-256 instructions. */ -if ((riscv_isa_extension_available (NULL ,ZVKNHA ) || -riscv_isa_extension_available (NULL ,ZVKNHB )) && -riscv_isa_extension_available (NULL ,ZVKB ) && -riscv_vector_vlen () >=128 ) -return crypto_register_shashes (riscv64_sha256_algs, -ARRAY_SIZE (riscv64_sha256_algs)); - -return -ENODEV ; -} - -static void __exit riscv64_sha256_mod_exit(void) -{ -crypto_unregister_shashes (riscv64_sha256_algs, -ARRAY_SIZE (riscv64_sha256_algs)); -} - -module_init (riscv64_sha256_mod_init); -module_exit (riscv64_sha256_mod_exit); - -MODULE_DESCRIPTION ("SHA-256 (RISC-V accelerated)" ); -MODULE_AUTHOR ("Heiko Stuebner <[email protected]>" ); -MODULE_LICENSE ("GPL" ); -MODULE_ALIAS_CRYPTO ("sha256" ); -MODULE_ALIAS_CRYPTO ("sha224" );arch/riscv/lib/crypto/Kconfig +7
-0@@ -4,5 +4,12 @@ config CRYPTO_CHACHA_RISCV64 tristate depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO default CRYPTO_LIB_CHACHA select CRYPTO_ARCH_HAVE_LIB_CHACHA select CRYPTO_LIB_CHACHA_GENERIC + +config CRYPTO_SHA256_RISCV64 + tristate + depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO + default CRYPTO_LIB_SHA256 + select CRYPTO_ARCH_HAVE_LIB_SHA256 + select CRYPTO_LIB_SHA256_GENERICarch/riscv/lib/crypto/Makefile +3
-0@@ -1,4 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_CRYPTO_CHACHA_RISCV64) += chacha-riscv64.o chacha-riscv64-y := chacha-riscv64-glue.o chacha-riscv64-zvkb.o + +obj-$(CONFIG_CRYPTO_SHA256_RISCV64) += sha256-riscv64.o +sha256-riscv64-y := sha256.o sha256-riscv64-zvknha_or_zvknhb-zvkb.oarch/riscv/lib/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S renamed from arch/riscv/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S +2
-2@@ -104,12 +104,12 @@ sha256_4rounds \last, \k1, W1, W2, W3, W0 sha256_4rounds \last, \k2, W2, W3, W0, W1 sha256_4rounds \last, \k3, W3, W0, W1, W2 .endm -// void sha256_transform_zvknha_or_zvknhb_zvkb(u32 state[8], const u8 *data, -// int num_blocks); +// void sha256_transform_zvknha_or_zvknhb_zvkb(u32 state[SHA256_STATE_WORDS], +// const u8 *data, size_t nblocks); SYM_FUNC_START(sha256_transform_zvknha_or_zvknhb_zvkb) // Load the round constants into K0-K15. vsetivli zero, 4, e32, m1, ta, ma la t0, K256arch/riscv/lib/crypto/sha256.c new file +52
-0@@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* SHA-256 (RISC-V accelerated) +* +* Copyright (C) 2022 VRULL GmbH +* Author: Heiko Stuebner <[email protected]> +* +* Copyright (C) 2023 SiFive, Inc. +* Author: Jerry Shih <[email protected]> +*/ + +#include <asm/simd.h> +#include <asm/vector.h> +#include <crypto/internal/sha2.h> +#include <crypto/internal/simd.h> +#include <linux/kernel.h> +#include <linux/module.h> + +asmlinkage void sha256_transform_zvknha_or_zvknhb_zvkb( +u32 state[SHA256_STATE_WORDS ],const u8 *data,size_t nblocks); + +static __ro_after_init DEFINE_STATIC_KEY_FALSE (have_extensions); + +void sha256_blocks_arch (u32 state[SHA256_STATE_WORDS ], +const u8 *data,size_t nblocks) +{ +if (static_branch_likely (&have_extensions) &&crypto_simd_usable ()) { +kernel_vector_begin (); +sha256_transform_zvknha_or_zvknhb_zvkb (state, data, nblocks); +kernel_vector_end (); + }else { +sha256_blocks_generic (state, data, nblocks); + } +} +EXPORT_SYMBOL (sha256_blocks_arch); + +bool sha256_is_arch_optimized (void ) +{ +return static_key_enabled (&have_extensions); +} +EXPORT_SYMBOL (sha256_is_arch_optimized); + +static int __init riscv64_sha256_mod_init (void ) +{ +/* Both zvknha and zvknhb provide the SHA-256 instructions. */ +if ((riscv_isa_extension_available (NULL ,ZVKNHA ) || +riscv_isa_extension_available (NULL ,ZVKNHB )) && +riscv_isa_extension_available (NULL ,ZVKB ) && +riscv_vector_vlen () >=128 ) +static_branch_enable (&have_extensions); +return 0 ; +}
arch/riscv/lib/crypto/sha256.c line 62
Quoted 10 lines
arch/riscv/lib/crypto/sha256.c new file +10
-0+arch_initcall (riscv64_sha256_mod_init); + +static void __exit riscv64_sha256_mod_exit(void) +{ +} +module_exit (riscv64_sha256_mod_exit); + +MODULE_DESCRIPTION ("SHA-256 (RISC-V accelerated)" ); +MODULE_AUTHOR ("Heiko Stuebner <[email protected]>" ); +MODULE_LICENSE ("GPL" );
I assume you want to keep these all together somewhere, so I'm going to drop it from the RISC-V patchwork.
Eric Biggers
maintainerRe: [PATCH v4 07/13] crypto: riscv/sha256 - implement library instead of shash
Quoted 5 lines · Palmer Dabbelt
Acked-by Palmer Dabbelt <[email protected]>I assume you want to keep these all together somewhere, so I'm going to drop it from the RISC-V patchwork.
Thanks! Yes, this series was already applied to "cryptodev"
(https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git/log/).
- Eric
Eric Biggers
maintainerPATCH v4 8/13 crypto: s390/sha256 - implement library instead of shash
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Instead of providing crypto_shash algorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available through crypto_shash, but individual architectures no longer need to handle it.
--- arch/s390/configs/debug_defconfig | 1 - arch/s390/configs/defconfig | 1 - arch/s390/crypto/Kconfig | 10 --- arch/s390/crypto/Makefile | 1 - arch/s390/crypto/sha256_s390.c | 144 ------------------------------ arch/s390/lib/crypto/Kconfig | 6 ++ arch/s390/lib/crypto/Makefile | 2 + arch/s390/lib/crypto/sha256.c | 47 ++++++++++ 8 files changed, 55 insertions(+), 157 deletions(-) delete mode 100644 arch/s390/crypto/sha256_s390.c create mode 100644 arch/s390/lib/crypto/sha256.c
arch/s390/configs/debug_defconfig +0 -1
@@ -793,11 +793,10 @@ CONFIG_CRYPTO_USER_API_HASH=m CONFIG_CRYPTO_USER_API_SKCIPHER=m CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_USER_API_AEAD=m CONFIG_CRYPTO_SHA512_S390=m CONFIG_CRYPTO_SHA1_S390=m -CONFIG_CRYPTO_SHA256_S390=m CONFIG_CRYPTO_SHA3_256_S390=m CONFIG_CRYPTO_SHA3_512_S390=m CONFIG_CRYPTO_GHASH_S390=m CONFIG_CRYPTO_AES_S390=m CONFIG_CRYPTO_DES_S390=m
arch/s390/configs/defconfig +0 -1
@@ -780,11 +780,10 @@ CONFIG_CRYPTO_USER_API_HASH=m CONFIG_CRYPTO_USER_API_SKCIPHER=m CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_USER_API_AEAD=m CONFIG_CRYPTO_SHA512_S390=m CONFIG_CRYPTO_SHA1_S390=m -CONFIG_CRYPTO_SHA256_S390=m CONFIG_CRYPTO_SHA3_256_S390=m CONFIG_CRYPTO_SHA3_512_S390=m CONFIG_CRYPTO_GHASH_S390=m CONFIG_CRYPTO_AES_S390=m CONFIG_CRYPTO_DES_S390=m
arch/s390/crypto/Kconfig +0 -10
@@ -20,20 +20,10 @@ config CRYPTO_SHA1_S390 Architecture: s390 It is available as of z990. -config CRYPTO_SHA256_S390 - tristate "Hash functions: SHA-224 and SHA-256" - select CRYPTO_HASH - help - SHA-224 and SHA-256 secure hash algorithms (FIPS 180) - - Architecture: s390 - - It is available as of z9. - config CRYPTO_SHA3_256_S390 tristate "Hash functions: SHA3-224 and SHA3-256" select CRYPTO_HASH help SHA3-224 and SHA3-256 secure hash algorithms (FIPS 202)
arch/s390/crypto/Makefile +0 -1
@@ -2,11 +2,10 @@ # # Cryptographic API # obj-$(CONFIG_CRYPTO_SHA1_S390) += sha1_s390.o sha_common.o -obj-$(CONFIG_CRYPTO_SHA256_S390) += sha256_s390.o sha_common.o obj-$(CONFIG_CRYPTO_SHA512_S390) += sha512_s390.o sha_common.o obj-$(CONFIG_CRYPTO_SHA3_256_S390) += sha3_256_s390.o sha_common.o obj-$(CONFIG_CRYPTO_SHA3_512_S390) += sha3_512_s390.o sha_common.o obj-$(CONFIG_CRYPTO_DES_S390) += des_s390.o obj-$(CONFIG_CRYPTO_AES_S390) += aes_s390.o
arch/s390/crypto/sha256_s390.c deleted +0 -144
144 lines not on this page · raw message
arch/s390/lib/crypto/Kconfig +6 -0
@@ -3,5 +3,11 @@ config CRYPTO_CHACHA_S390 tristate default CRYPTO_LIB_CHACHA select CRYPTO_LIB_CHACHA_GENERIC select CRYPTO_ARCH_HAVE_LIB_CHACHA + +config CRYPTO_SHA256_S390 + tristate + default CRYPTO_LIB_SHA256 + select CRYPTO_ARCH_HAVE_LIB_SHA256 + select CRYPTO_LIB_SHA256_GENERIC
arch/s390/lib/crypto/Makefile +2 -0
@@ -1,4 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_CRYPTO_CHACHA_S390) += chacha_s390.o chacha_s390-y := chacha-glue.o chacha-s390.o + +obj-$(CONFIG_CRYPTO_SHA256_S390) += sha256.o
arch/s390/lib/crypto/sha256.c new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
Quoted 8 lines · On Mon, 28 Apr 2025 10:00:33 -0700 Eric Biggers <[email protected]> wrote:
From: Eric Biggers <[email protected]>
Instead of providing
crypto_shashalgorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available throughcrypto_shash, but individual architectures no longer need to handle it.
I can get to the following error after this patch, now merged as commit b9eac03edcf8 ("crypto: s390/sha256 - implement library instead of shash"):
error: the following would cause module name conflict:
crypto/sha256.ko arch/s390/lib/crypto/sha256.ko
Base config file is generated from:
$ CONFIG=$(mktemp) $ cat << EOF > $CONFIG CONFIG_MODULES=y CONFIG_CRYPTO=y CONFIG_CRYPTO_SHA256=m EOF
Base config applied to allnoconfig:
$ KCONFIG_ALLCONFIG=$CONFIG make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- allnoconfig
Resulting in:
$ grep SHA256 .config CONFIG_CRYPTO_SHA256=m CONFIG_CRYPTO_LIB_SHA256=m CONFIG_CRYPTO_ARCH_HAVE_LIB_SHA256=y CONFIG_CRYPTO_LIB_SHA256_GENERIC=m CONFIG_CRYPTO_SHA256_S390=m
Thanks,
Alex
Eric Biggers
maintainerRe: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
Quoted 12 lines · Alex Williamson
Quoted 8 lines · On Mon, 28 Apr 2025 10:00:33 -0700 Eric Biggers <[email protected]> wrote:
From: Eric Biggers <[email protected]>
Instead of providing
crypto_shashalgorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available throughcrypto_shash, but individual architectures no longer need to handle it.
Quoted 6 lines
I can get to the following error after this patch, now merged as commit
b9eac03edcf8("crypto: s390/sha256 - implement library instead of shash"):error: the following would cause module name conflict:
crypto/sha256.koarch/s390/lib/crypto/sha256.ko
Thanks for reporting this. For now the s390 one should be renamed to sha256-s390, similar to how the other architectures' sha256 modules are named. I'll send a patch.
Long-term, I'd like to find a clean way to consolidate the library code for each algorithm into a single module. So instead of e.g. libsha256.ko, libsha256-generic.ko, and sha256-s390.ko (all of which get loaded when the SHA-256 library is needed), we'd just have libsha256.ko. (Or just sha256.ko, with the old-school crypto API one renamed to sha256-cryptoapi.ko.) A lot of these weird build problems we've been having are caused by the unnecessary separation into multiple modules.
- Eric
Eric Biggers
maintainerRe: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
Quoted 32 lines · Eric Biggers
Quoted 12 lines · Alex Williamson
Quoted 8 lines · On Mon, 28 Apr 2025 10:00:33 -0700 Eric Biggers <[email protected]> wrote:
From: Eric Biggers <[email protected]>
Instead of providing
crypto_shashalgorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available throughcrypto_shash, but individual architectures no longer need to handle it.Quoted 6 lines
I can get to the following error after this patch, now merged as commit
b9eac03edcf8("crypto: s390/sha256 - implement library instead of shash"):error: the following would cause module name conflict:
crypto/sha256.koarch/s390/lib/crypto/sha256.koThanks for reporting this. For now the s390 one should be renamed to sha256-s390, similar to how the other architectures' sha256 modules are named. I'll send a patch.
Long-term, I'd like to find a clean way to consolidate the library code for each algorithm into a single module. So instead of e.g. libsha256.ko, libsha256-generic.ko, and sha256-s390.ko (all of which get loaded when the SHA-256 library is needed), we'd just have libsha256.ko. (Or just sha256.ko, with the old-school crypto API one renamed to sha256-cryptoapi.ko.) A lot of these weird build problems we've been having are caused by the unnecessary separation into multiple modules.
Quoted 2 lines
- Eric
Patch sent: https://lore.kernel.org/r/[email protected]
- Eric
Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
In reply to Eric BiggersQuoted 3 lines · Eric Biggers
Long-term, I'd like to find a clean way to consolidate the library code for each algorithm into a single module.
No, while I think the current situation isn't great, I think the "make it one single module" is even worse.
For most architectures - including s390 - you end up being in the situation that these kinds of hw accelerated crypto things depend on some CPU capability, and aren't necessarily statically always available.
So these things end up having stupid extra overhead due to having some conditional.
That extra overhead is then in turn minimized with tricks like static branches, but that's all just just piling more ugly hacks on top because it picked a bad choice to begin with.
So what's the *right* thing to do?
The right thing to do is to just link the right routine in the first place, and *not* have static branch hackery at all. Because you didn't need it.
And we already do runtime linking at module loading time. So if it's a module, if the hardware acceleration doesn't exist, the module load should just fail, and the loader should go on to load the next option.
Not any silly "one module to rule them all" hackery that only results in worse code. Just a simple "if this module loads successfully, you'll link the optimal hw acceleration".
Now, the problem with this all is the *non*modular case.
For modules, we already have the optimal solution in the form of init-module error handling and runtime linking.
So I think the module case is "solved" (except the solution is not what we actually do).
For the non-module case, the problem is that "I linked this unconditionally, and now it turns out I run on hardware that doesn't have the capability to run this".
And that's when you need to do things like static_call_update() to basically do runtime re-linking of a static decision.
And currently we very much do this wrong. See how s390 and x86-64 (and presumably others) basically have the *exact* same problems, but they then mix static branches and static calls (in the case of x86-64) and just have non-optimal code in general.
What I think the generic code should do (for the built-in case) is just have
DEFINE_STATIC_CALL(sha256_blocks_fn, sha256_blocks_generic);
and do
static_call(sha256_blocks_fn)(args..);
and then architecture code can do the static_call_update() to set their optimal version.
And yeah, we'd presumably need multiple versions, since there's the whole "is simd usable" thing. Although maybe that's going away?
Linus
Eric Biggers
maintainerRe: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
Quoted 22 lines · Linus Torvalds
Quoted 3 lines · Eric Biggers
Long-term, I'd like to find a clean way to consolidate the library code for each algorithm into a single module.
No, while I think the current situation isn't great, I think the "make it one single module" is even worse.
For most architectures - including s390 - you end up being in the situation that these kinds of hw accelerated crypto things depend on some CPU capability, and aren't necessarily statically always available.
So these things end up having stupid extra overhead due to having some conditional.
That extra overhead is then in turn minimized with tricks like static branches, but that's all just just piling more ugly hacks on top because it picked a bad choice to begin with.
So what's the *right* thing to do?
Quoted 7 lines
The right thing to do is to just link the right routine in the first place, and *not* have static branch hackery at all. Because you didn't need it.
And we already do runtime linking at module loading time. So if it's a module, if the hardware acceleration doesn't exist, the module load should just fail, and the loader should go on to load the next option.
So using crc32c() + ext4 + x86 as an example (but SHA-256 would be very similar), the current behavior is that ext4.ko depends on the crc32c_arch() symbol. That causes crc32-x86.ko to be loaded, which then depends on the crc32c_base() symbol as a fallback, which causes crc32.ko to be loaded too. My idea is to consolidate the two crc32 modules into one (they always go together, after all), keeping the same symbols. The main challenge is just the current directory structure.
Your suggestion sounds like: ext4.ko would depend on the crc32c() symbol, which would be defined in *both* crc32-x86.ko and crc32.ko. The module loader would try to load crc32-x86.ko first. If the CPU does not support any of the x86 accelerated CRC32 code, then loading that module would fail. The module loader would then load crc32.ko instead.
Does any of the infrastructure to handle "this symbol is in multiple modules and they must be loaded in this particular order" actually exist, though?
And how do we avoid the issues the crypto API often has where the accelerated modules don't get loaded, causing slow generic code to unnecessarily be used?
IMO this sounds questionable compared to just using static keys and/or branches, which we'd need anyway to support the non-modular case.
Quoted 30 lines
Not any silly "one module to rule them all" hackery that only results in worse code. Just a simple "if this module loads successfully, you'll link the optimal hw acceleration".
Now, the problem with this all is the *non*modular case.
For modules, we already have the optimal solution in the form of init-module error handling and runtime linking.
So I think the module case is "solved" (except the solution is not what we actually do).
For the non-module case, the problem is that "I linked this unconditionally, and now it turns out I run on hardware that doesn't have the capability to run this".
And that's when you need to do things like
static_call_update()to basically do runtime re-linking of a static decision.And currently we very much do this wrong. See how s390 and x86-64 (and presumably others) basically have the *exact* same problems, but they then mix static branches and static calls (in the case of x86-64) and just have non-optimal code in general.
What I think the generic code should do (for the built-in case) is just have
DEFINE_STATIC_CALL(sha256_blocks_fn, sha256_blocks_generic);and do
Quoted 7 lines
static_call(sha256_blocks_fn)(args..);and then architecture code can do the
static_call_update()to set their optimal version.And yeah, we'd presumably need multiple versions, since there's the whole "is simd usable" thing. Although maybe that's going away?
Moving the static_call into the generic code might make sense. I don't think it's a win in all cases currently, though. Only x86 and PPC32 actually have a real static_call implementation; everywhere else it's an indirect call which is slower than a static branch. Also, some arch code is just usable unconditionally without any CPU feature check, e.g. the MIPS ChaCha code. That doesn't use (or need to use) a static call or branch at all.
Also, while the centralized static_call would *allow* for the generic code to be loaded while the arch code is not, in the vast majority of cases that would be a bug, not a feature. The generic crypto infrastructure has that bug, and this has caused a huge amount of pain over the years. People have to go out of the way to ensure that the arch-optimized crypto code gets loaded. And they often forget, resulting in the slow generic code being used unnecessarily...
Making the arch-optimized code be loaded through a direct symbol dependency solves that problem.
- Eric
Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
Quoted 4 lines · Eric Biggers
So using
crc32c()+ ext4 + x86 as an example (but SHA-256 would be very similar), the current behavior is that ext4.ko depends on thecrc32c_arch()symbol.
Yes, I think that's a good example.
I think it's an example of something that "works", but it certainly is a bit hacky.
Wouldn't it be nicer if just plain "crc32c()" did the right thing, instead of users having to do strange hacks just to get the optimized version that they are looking for?
Quoted 2 lines
Does any of the infrastructure to handle "this symbol is in multiple modules and they must be loaded in this particular order" actually exist, though?
Hmm. I was sure we already did that for other things, but looking around, I'm not finding any cases.
Or rather, I _am_ finding cases where we export the same symbol from different code, but all the ones I found were being careful to not be active at the same time.
I really thought we had cases where depending on which module you loaded you got different implementations, but it looks like it either was some historical thing that no longer exists - or that I need to go take my meds.
Quoted 2 lines
IMO this sounds questionable compared to just using static keys and/or branches, which we'd need anyway to support the non-modular case.
I really wish the non-modular case used static calls, not static keys like it does now.
In fact, that should work even for modular users.
Of course, not all architectures actually do the optimized thing, and the generic fallback uses indirect calls through a function pointer, but hey, if an architecture didn't bother with the rewriting code that is fixable - if the architecture maintainer cares.
(On some architectures, indirect calls are not noticeably slower than direct calls - because you have to load the address from some global pointer area anyway - so not having the rewriting can be a "we don't need it" thing)
Linus
Eric Biggers
maintainerRe: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Eric Biggers
So using
crc32c()+ ext4 + x86 as an example (but SHA-256 would be very similar), the current behavior is that ext4.ko depends on thecrc32c_arch()symbol.
Quoted 8 lines
Yes, I think that's a good example.
I think it's an example of something that "works", but it certainly is a bit hacky.
Wouldn't it be nicer if just plain "
crc32c()" did the right thing, instead of users having to do strange hacks just to get the optimized version that they are looking for?
For crc32c() that's exactly how it works (since v6.14, when I implemented it). The users call crc32c() which is an inline function, which then calls crc32c_arch() or crc32c_base() depending on the kconfig. So that's why I said the symbol dependency is currently on crc32c_arch. Sorry if I wasn't clear. The SHA-256, ChaCha, and Poly1305 library code now has a similar design too.
If we merged the arch and generic modules together, then the symbol would become crc32c. But in either case crc32c() is the API that all the users call.
- Eric
Eric Biggers
maintainerRe: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
Quoted 22 lines · Eric Biggers
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Eric Biggers
So using
crc32c()+ ext4 + x86 as an example (but SHA-256 would be very similar), the current behavior is that ext4.ko depends on thecrc32c_arch()symbol.Quoted 8 lines
Yes, I think that's a good example.
I think it's an example of something that "works", but it certainly is a bit hacky.
Wouldn't it be nicer if just plain "
crc32c()" did the right thing, instead of users having to do strange hacks just to get the optimized version that they are looking for?For
crc32c()that's exactly how it works (since v6.14, when I implemented it). The users callcrc32c()which is an inline function, which then callscrc32c_arch()orcrc32c_base()depending on the kconfig. So that's why I said the symbol dependency is currently oncrc32c_arch. Sorry if I wasn't clear. The SHA-256, ChaCha, and Poly1305 library code now has a similar design too.
Quoted 5 lines
If we merged the arch and generic modules together, then the symbol would become crc32c. But in either case
crc32c()is the API that all the users call.- Eric
I implemented my proposal, for lib/crc first, in
https://lore.kernel.org/lkml/[email protected].
I think it's strictly better than the status quo, and once applied to lib/crypto it will solve some of the problems we've been having there too. But let me know if you still have misgivings.
- Eric
Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
Quoted 2 lines · Eric Biggers
I implemented my proposal, for lib/crc first,
Ok, I scanned through that series, and it looks good to me. A clear improvement.
Linus
Eric Biggers
maintainerPATCH v4 9/13 crypto: sparc - move opcodes.h into asm directory
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Since arch/sparc/crypto/opcodes.h is now needed outside the arch/sparc/crypto/ directory, move it into arch/sparc/include/asm/ so that it can be included as <asm/opcodes.h>.
---
arch/sparc/crypto/aes_asm.S | 3 +--
arch/sparc/crypto/aes_glue.c | 3 +--
arch/sparc/crypto/camellia_asm.S | 3 +--
arch/sparc/crypto/camellia_glue.c | 3 +--
arch/sparc/crypto/des_asm.S | 3 +--
arch/sparc/crypto/des_glue.c | 3 +--
arch/sparc/crypto/md5_asm.S | 3 +--
arch/sparc/crypto/md5_glue.c | 3 +--
arch/sparc/crypto/sha1_asm.S | 3 +--
arch/sparc/crypto/sha1_glue.c | 3 +--
arch/sparc/crypto/sha256_asm.S | 3 +--
arch/sparc/crypto/sha256_glue.c | 3 +--
arch/sparc/crypto/sha512_asm.S | 3 +--
arch/sparc/crypto/sha512_glue.c | 3 +--
arch/sparc/{crypto => include/asm}/opcodes.h | 6 +++---
arch/sparc/lib/crc32c_asm.S | 3 +--
16 files changed, 18 insertions(+), 33 deletions(-)
rename arch/sparc/{crypto => include/asm}/opcodes.h (96%)arch/sparc/crypto/aes_asm.S +1 -2
@@ -1,11 +1,10 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include <linux/linkage.h> +#include <asm/opcodes.h> #include <asm/visasm.h> -#include "opcodes.h" - #define ENCRYPT_TWO_ROUNDS(KEY_BASE, I0, I1, T0, T1) \ AES_EROUND01(KEY_BASE + 0, I0, I1, T0) \ AES_EROUND23(KEY_BASE + 2, I0, I1, T1) \ AES_EROUND01(KEY_BASE + 4, T0, T1, I0) \ AES_EROUND23(KEY_BASE + 6, T0, T1, I1)
arch/sparc/crypto/aes_glue.c +1 -2
@@ -25,15 +25,14 @@#include <crypto/algapi.h> #include <crypto/aes.h> #include <crypto/internal/skcipher.h> #include <asm/fpumacro.h> +#include <asm/opcodes.h> #include <asm/pstate.h> #include <asm/elf.h> -#include "opcodes.h" -struct aes_ops {void (*encrypt)(const u64 *key,const u32 *input,u32 *output);void (*decrypt)(const u64 *key,const u32 *input,u32 *output);void (*load_encrypt_keys)(const u64 *key);void (*load_decrypt_keys)(const u64 *key);
arch/sparc/crypto/camellia_asm.S +1 -2
@@ -1,11 +1,10 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include <linux/linkage.h> +#include <asm/opcodes.h> #include <asm/visasm.h> -#include "opcodes.h" - #define CAMELLIA_6ROUNDS(KEY_BASE, I0, I1) \ CAMELLIA_F(KEY_BASE + 0, I1, I0, I1) \ CAMELLIA_F(KEY_BASE + 2, I0, I1, I0) \ CAMELLIA_F(KEY_BASE + 4, I1, I0, I1) \ CAMELLIA_F(KEY_BASE + 6, I0, I1, I0) \
arch/sparc/crypto/camellia_glue.c +1 -2
@@ -13,15 +13,14 @@#include <linux/types.h> #include <crypto/algapi.h> #include <crypto/internal/skcipher.h> #include <asm/fpumacro.h> +#include <asm/opcodes.h> #include <asm/pstate.h> #include <asm/elf.h> -#include "opcodes.h" -#define CAMELLIA_MIN_KEY_SIZE 16#define CAMELLIA_MAX_KEY_SIZE 32#define CAMELLIA_BLOCK_SIZE 16#define CAMELLIA_TABLE_BYTE_LEN 272
arch/sparc/crypto/des_asm.S +1 -2
@@ -1,11 +1,10 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include <linux/linkage.h> +#include <asm/opcodes.h> #include <asm/visasm.h> -#include "opcodes.h" - .align 32 ENTRY(des_sparc64_key_expand) /* %o0=input_key, %o1=output_key */ VISEntryHalf ld [%o0 + 0x00], %f0
arch/sparc/crypto/des_glue.c +1 -2
@@ -14,15 +14,14 @@#include <crypto/algapi.h> #include <crypto/internal/des.h> #include <crypto/internal/skcipher.h> #include <asm/fpumacro.h> +#include <asm/opcodes.h> #include <asm/pstate.h> #include <asm/elf.h> -#include "opcodes.h" -struct des_sparc64_ctx {u64 encrypt_expkey[DES_EXPKEY_WORDS /2 ];u64 decrypt_expkey[DES_EXPKEY_WORDS /2 ]; };
arch/sparc/crypto/md5_asm.S +1 -2
@@ -1,11 +1,10 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include <linux/linkage.h> +#include <asm/opcodes.h> #include <asm/visasm.h> -#include "opcodes.h" - ENTRY(md5_sparc64_transform) /* %o0 = digest, %o1 = data, %o2 = rounds */ VISEntryHalf ld [%o0 + 0x00], %f0 ld [%o0 + 0x04], %f1
arch/sparc/crypto/md5_glue.c +1 -2
@@ -13,21 +13,20 @@*/ #define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <asm/elf.h> +#include <asm/opcodes.h> #include <asm/pstate.h> #include <crypto/internal/hash.h> #include <crypto/md5.h> #include <linux/errno.h> #include <linux/kernel.h> #include <linux/module.h> #include <linux/string.h> #include <linux/unaligned.h> -#include "opcodes.h" -struct sparc_md5_state {__le32 hash[MD5_HASH_WORDS ];u64 byte_count; };
arch/sparc/crypto/sha1_asm.S +1 -2
@@ -1,11 +1,10 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include <linux/linkage.h> +#include <asm/opcodes.h> #include <asm/visasm.h> -#include "opcodes.h" - ENTRY(sha1_sparc64_transform) /* %o0 = digest, %o1 = data, %o2 = rounds */ VISEntryHalf ld [%o0 + 0x00], %f0 ld [%o0 + 0x04], %f1
arch/sparc/crypto/sha1_glue.c +1 -2
@@ -10,19 +10,18 @@*/ #define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <asm/elf.h> +#include <asm/opcodes.h> #include <asm/pstate.h> #include <crypto/internal/hash.h> #include <crypto/sha1.h> #include <crypto/sha1_base.h> #include <linux/kernel.h> #include <linux/module.h> -#include "opcodes.h" - asmlinkage void sha1_sparc64_transform(struct sha1_state *digest,const u8 *data,int rounds);static int sha1_sparc64_update (struct shash_desc *desc,const u8 *data,unsigned int len)
arch/sparc/crypto/sha256_asm.S +1 -2
@@ -1,11 +1,10 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include <linux/linkage.h> +#include <asm/opcodes.h> #include <asm/visasm.h> -#include "opcodes.h" - ENTRY(sha256_sparc64_transform) /* %o0 = digest, %o1 = data, %o2 = rounds */ VISEntryHalf ld [%o0 + 0x00], %f0 ld [%o0 + 0x04], %f1
arch/sparc/crypto/sha256_glue.c +1 -2
@@ -10,19 +10,18 @@*/ #define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <asm/elf.h> +#include <asm/opcodes.h> #include <asm/pstate.h> #include <crypto/internal/hash.h> #include <crypto/sha2.h> #include <crypto/sha256_base.h> #include <linux/kernel.h> #include <linux/module.h> -#include "opcodes.h" - asmlinkage void sha256_sparc64_transform(u32 *digest, const char *data,unsigned int rounds);static void sha256_block (struct crypto_sha256_state *sctx,const u8 *src,int blocks)
arch/sparc/crypto/sha512_asm.S +1 -2
@@ -1,11 +1,10 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include <linux/linkage.h> +#include <asm/opcodes.h> #include <asm/visasm.h> -#include "opcodes.h" - ENTRY(sha512_sparc64_transform) /* %o0 = digest, %o1 = data, %o2 = rounds */ VISEntry ldd [%o0 + 0x00], %f0 ldd [%o0 + 0x08], %f2
arch/sparc/crypto/sha512_glue.c +1 -2
@@ -9,19 +9,18 @@*/ #define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <asm/elf.h> +#include <asm/opcodes.h> #include <asm/pstate.h> #include <crypto/internal/hash.h> #include <crypto/sha2.h> #include <crypto/sha512_base.h> #include <linux/kernel.h> #include <linux/module.h> -#include "opcodes.h" - asmlinkage void sha512_sparc64_transform(u64 *digest, const char *data,unsigned int rounds);static void sha512_block (struct sha512_state *sctx,const u8 *src,int blocks) {
arch/sparc/include/asm/opcodes.h renamed from arch/sparc/crypto/opcodes.h +3 -3
@@ -1,8 +1,8 @@/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _OPCODES_H -#define _OPCODES_H +#ifndef _SPARC_ASM_OPCODES_H +#define _SPARC_ASM_OPCODES_H#define SPARC_CR_OPCODE_PRIORITY 300#define F3F (x,y,z) (((x)<<30)|((y)<<19)|((z)<<5))
@@ -95,6 +95,6 @@#define MOVXTOD_G3_F60 \ .word 0xbbb02303;#define MOVXTOD_G7_F62 \ .word 0xbfb02307; -#endif /* _OPCODES_H */ +#endif /* _SPARC_ASM_OPCODES_H */
arch/sparc/lib/crc32c_asm.S +1 -2
@@ -1,12 +1,11 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include <linux/linkage.h> +#include <asm/opcodes.h> #include <asm/visasm.h> #include <asm/asi.h> -#include "../crypto/opcodes.h" - ENTRY(crc32c_sparc64) /* %o0=crc32p, %o1=data_ptr, %o2=len */ VISEntryHalf lda [%o0] ASI_PL, %f1 1: ldd [%o1], %f2
Signature · 1 line
2.49.0
Eric Biggers
maintainerPATCH v4 10/13 crypto: sparc/sha256 - implement library instead of shash
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Instead of providing crypto_shash algorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available through crypto_shash, but individual architectures no longer need to handle it.
---
arch/sparc/crypto/Kconfig | 10 --
arch/sparc/crypto/Makefile | 2 -
arch/sparc/crypto/sha256_glue.c | 128 -----------------------
arch/sparc/lib/Makefile | 1 +
arch/sparc/lib/crypto/Kconfig | 8 ++
arch/sparc/lib/crypto/Makefile | 4 +
arch/sparc/lib/crypto/sha256.c | 64 ++++++++++++
arch/sparc/{ => lib}/crypto/sha256_asm.S | 2 +-
lib/crypto/Kconfig | 3 +
9 files changed, 81 insertions(+), 141 deletions(-)
delete mode 100644 arch/sparc/crypto/sha256_glue.c
create mode 100644 arch/sparc/lib/crypto/Kconfig
create mode 100644 arch/sparc/lib/crypto/Makefile
create mode 100644 arch/sparc/lib/crypto/sha256.c
rename arch/sparc/{ => lib}/crypto/sha256_asm.S (96%)arch/sparc/crypto/Kconfig +0 -10
@@ -34,20 +34,10 @@ config CRYPTO_SHA1_SPARC64 help SHA-1 secure hash algorithm (FIPS 180) Architecture: sparc64 -config CRYPTO_SHA256_SPARC64 - tristate "Hash functions: SHA-224 and SHA-256" - depends on SPARC64 - select CRYPTO_SHA256 - select CRYPTO_HASH - help - SHA-224 and SHA-256 secure hash algorithms (FIPS 180) - - Architecture: sparc64 using crypto instructions, when available - config CRYPTO_SHA512_SPARC64 tristate "Hash functions: SHA-384 and SHA-512" depends on SPARC64 select CRYPTO_SHA512 select CRYPTO_HASH
arch/sparc/crypto/Makefile +0 -2
@@ -2,20 +2,18 @@ # # Arch-specific CryptoAPI modules. # obj-$(CONFIG_CRYPTO_SHA1_SPARC64) += sha1-sparc64.o -obj-$(CONFIG_CRYPTO_SHA256_SPARC64) += sha256-sparc64.o obj-$(CONFIG_CRYPTO_SHA512_SPARC64) += sha512-sparc64.o obj-$(CONFIG_CRYPTO_MD5_SPARC64) += md5-sparc64.o obj-$(CONFIG_CRYPTO_AES_SPARC64) += aes-sparc64.o obj-$(CONFIG_CRYPTO_DES_SPARC64) += des-sparc64.o obj-$(CONFIG_CRYPTO_CAMELLIA_SPARC64) += camellia-sparc64.o sha1-sparc64-y := sha1_asm.o sha1_glue.o -sha256-sparc64-y := sha256_asm.o sha256_glue.o sha512-sparc64-y := sha512_asm.o sha512_glue.o md5-sparc64-y := md5_asm.o md5_glue.o aes-sparc64-y := aes_asm.o aes_glue.o des-sparc64-y := des_asm.o des_glue.o
arch/sparc/crypto/sha256_glue.c deleted +0 -128
128 lines not on this page · raw message
arch/sparc/lib/Makefile +1 -0
@@ -2,10 +2,11 @@ # Makefile for Sparc library files.. # asflags-y := -ansi -DST_DIV0=0x02 +obj-y += crypto/ lib-$(CONFIG_SPARC32) += ashrdi3.o lib-$(CONFIG_SPARC32) += memcpy.o memset.o lib-y += strlen.o lib-y += checksum_$(BITS).o lib-$(CONFIG_SPARC32) += blockops.o
arch/sparc/lib/crypto/Kconfig new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-2.0-only + +config CRYPTO_SHA256_SPARC64 + tristate + depends on SPARC64 + default CRYPTO_LIB_SHA256 + select CRYPTO_ARCH_HAVE_LIB_SHA256 + select CRYPTO_LIB_SHA256_GENERIC
arch/sparc/lib/crypto/Makefile new file +4 -0
@@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0-only + +obj-$(CONFIG_CRYPTO_SHA256_SPARC64) += sha256-sparc64.o +sha256-sparc64-y := sha256.o sha256_asm.o
arch/sparc/lib/crypto/sha256.c new file +64 -0
64 lines not on this page · raw message
arch/sparc/lib/crypto/sha256_asm.S renamed from arch/sparc/crypto/sha256_asm.S +1 -1
@@ -2,11 +2,11 @@ #include <linux/linkage.h> #include <asm/opcodes.h> #include <asm/visasm.h> ENTRY(sha256_sparc64_transform) - /* %o0 = digest, %o1 = data, %o2 = rounds */ + /* %o0 = state, %o1 = data, %o2 = nblocks */ VISEntryHalf ld [%o0 + 0x00], %f0 ld [%o0 + 0x04], %f1 ld [%o0 + 0x08], %f2 ld [%o0 + 0x0c], %f3
lib/crypto/Kconfig +3 -0
@@ -179,10 +179,13 @@ if RISCV source "arch/riscv/lib/crypto/Kconfig" endif if S390 source "arch/s390/lib/crypto/Kconfig" endif +if SPARC +source "arch/sparc/lib/crypto/Kconfig" +endif if X86 source "arch/x86/lib/crypto/Kconfig" endif endif
Signature · 1 line
2.49.0
Eric Biggers
maintainerPATCH v4 11/13 crypto: x86/sha256 - implement library instead of shash
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Instead of providing crypto_shash algorithms for the arch-optimized SHA-256 code, instead implement the SHA-256 library. This is much simpler, it makes the SHA-256 library functions be arch-optimized, and it fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default. SHA-256 still remains available through crypto_shash, but individual architectures no longer need to handle it.
To match sha256_blocks_arch(), change the type of the nblocks parameter of the assembly functions from int to size_t. The assembly functions actually already treated it as size_t.
---
arch/x86/crypto/Kconfig | 14 -
arch/x86/crypto/Makefile | 3 -
arch/x86/crypto/sha256_ssse3_glue.c | 432 ------------------
arch/x86/lib/crypto/Kconfig | 7 +
arch/x86/lib/crypto/Makefile | 3 +
arch/x86/{ => lib}/crypto/sha256-avx-asm.S | 12 +-
arch/x86/{ => lib}/crypto/sha256-avx2-asm.S | 12 +-
.../crypto/sha256-ni-asm.S} | 36 +-
arch/x86/{ => lib}/crypto/sha256-ssse3-asm.S | 14 +-
arch/x86/lib/crypto/sha256.c | 74 +++
10 files changed, 118 insertions(+), 489 deletions(-)
delete mode 100644 arch/x86/crypto/sha256_ssse3_glue.c
rename arch/x86/{ => lib}/crypto/sha256-avx-asm.S (98%)
rename arch/x86/{ => lib}/crypto/sha256-avx2-asm.S (98%)
rename arch/x86/{crypto/sha256_ni_asm.S => lib/crypto/sha256-ni-asm.S} (85%)
rename arch/x86/{ => lib}/crypto/sha256-ssse3-asm.S (98%)
create mode 100644 arch/x86/lib/crypto/sha256.carch/x86/crypto/Kconfig +0 -14
@@ -388,24 +388,10 @@ config CRYPTO_SHA1_SSSE3 - SSSE3 (Supplemental SSE3) - AVX (Advanced Vector Extensions) - AVX2 (Advanced Vector Extensions 2) - SHA-NI (SHA Extensions New Instructions) -config CRYPTO_SHA256_SSSE3 - tristate "Hash functions: SHA-224 and SHA-256 (SSSE3/AVX/AVX2/SHA-NI)" - depends on 64BIT - select CRYPTO_SHA256 - select CRYPTO_HASH - help - SHA-224 and SHA-256 secure hash algorithms (FIPS 180) - - Architecture: x86_64 using: - - SSSE3 (Supplemental SSE3) - - AVX (Advanced Vector Extensions) - - AVX2 (Advanced Vector Extensions 2) - - SHA-NI (SHA Extensions New Instructions) - config CRYPTO_SHA512_SSSE3 tristate "Hash functions: SHA-384 and SHA-512 (SSSE3/AVX/AVX2)" depends on 64BIT select CRYPTO_SHA512 select CRYPTO_HASH
arch/x86/crypto/Makefile +0 -3
@@ -52,13 +52,10 @@ aesni-intel-$(CONFIG_64BIT) += aes-gcm-avx10-x86_64.o endif obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ni_asm.o sha1_ssse3_glue.o -obj-$(CONFIG_CRYPTO_SHA256_SSSE3) += sha256-ssse3.o -sha256-ssse3-y := sha256-ssse3-asm.o sha256-avx-asm.o sha256-avx2-asm.o sha256_ni_asm.o sha256_ssse3_glue.o - obj-$(CONFIG_CRYPTO_SHA512_SSSE3) += sha512-ssse3.o sha512-ssse3-y := sha512-ssse3-asm.o sha512-avx-asm.o sha512-avx2-asm.o sha512_ssse3_glue.o obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
arch/x86/crypto/sha256_ssse3_glue.c deleted +0 -432
432 lines not on this page · raw message
arch/x86/lib/crypto/Kconfig +7 -0
@@ -22,5 +22,12 @@ config CRYPTO_CHACHA20_X86_64 config CRYPTO_POLY1305_X86_64 tristate depends on 64BIT default CRYPTO_LIB_POLY1305 select CRYPTO_ARCH_HAVE_LIB_POLY1305 + +config CRYPTO_SHA256_X86_64 + tristate + depends on 64BIT + default CRYPTO_LIB_SHA256 + select CRYPTO_ARCH_HAVE_LIB_SHA256 + select CRYPTO_LIB_SHA256_GENERIC
arch/x86/lib/crypto/Makefile +3 -0
@@ -8,10 +8,13 @@ chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha-avx512vl-x8 obj-$(CONFIG_CRYPTO_POLY1305_X86_64) += poly1305-x86_64.o poly1305-x86_64-y := poly1305-x86_64-cryptogams.o poly1305_glue.o targets += poly1305-x86_64-cryptogams.S +obj-$(CONFIG_CRYPTO_SHA256_X86_64) += sha256-x86_64.o +sha256-x86_64-y := sha256.o sha256-ssse3-asm.o sha256-avx-asm.o sha256-avx2-asm.o sha256-ni-asm.o + quiet_cmd_perlasm = PERLASM $@ cmd_perlasm = $(PERL) $< > $@ $(obj)/%.S: $(src)/%.pl FORCE $(call if_changed,perlasm)
arch/x86/lib/crypto/sha256-avx-asm.S renamed from arch/x86/crypto/sha256-avx-asm.S +6 -6
@@ -46,11 +46,11 @@ ######################################################################## # This code schedules 1 block at a time, with 4 lanes per block ######################################################################## #include <linux/linkage.h> -#include <linux/cfi_types.h> +#include <linux/objtool.h> ## assume buffers not aligned #define VMOVDQ vmovdqu ################################ Define Macros
@@ -339,17 +339,17 @@ a = TMP_ add y0, h # h = h + S1 + CH + k + w + S0 + MAJ ROTATE_ARGS .endm ######################################################################## -## void sha256_transform_avx(state sha256_state *state, const u8 *data, int blocks) -## arg 1 : pointer to state -## arg 2 : pointer to input data -## arg 3 : Num blocks +## void sha256_transform_avx(u32 state[SHA256_STATE_WORDS], +## const u8 *data, size_t nblocks); ######################################################################## .text -SYM_TYPED_FUNC_START(sha256_transform_avx) +SYM_FUNC_START(sha256_transform_avx) + ANNOTATE_NOENDBR # since this is called only via static_call + pushq %rbx pushq %r12 pushq %r13 pushq %r14 pushq %r15
arch/x86/lib/crypto/sha256-avx2-asm.S renamed from arch/x86/crypto/sha256-avx2-asm.S +6 -6
@@ -47,11 +47,11 @@ ######################################################################## # This code schedules 2 blocks at a time, with 4 lanes per block ######################################################################## #include <linux/linkage.h> -#include <linux/cfi_types.h> +#include <linux/objtool.h> ## assume buffers not aligned #define VMOVDQ vmovdqu ################################ Define Macros
@@ -516,17 +516,17 @@ STACK_SIZE = _CTX + _CTX_SIZE ROTATE_ARGS .endm ######################################################################## -## void sha256_transform_rorx(struct sha256_state *state, const u8 *data, int blocks) -## arg 1 : pointer to state -## arg 2 : pointer to input data -## arg 3 : Num blocks +## void sha256_transform_rorx(u32 state[SHA256_STATE_WORDS], +## const u8 *data, size_t nblocks); ######################################################################## .text -SYM_TYPED_FUNC_START(sha256_transform_rorx) +SYM_FUNC_START(sha256_transform_rorx) + ANNOTATE_NOENDBR # since this is called only via static_call + pushq %rbx pushq %r12 pushq %r13 pushq %r14 pushq %r15
arch/x86/lib/crypto/sha256-ni-asm.S renamed from arch/x86/crypto/sha256_ni_asm.S +16 -20
@@ -52,13 +52,13 @@ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * */ #include <linux/linkage.h> -#include <linux/cfi_types.h> +#include <linux/objtool.h> -#define DIGEST_PTR %rdi /* 1st arg */ +#define STATE_PTR %rdi /* 1st arg */ #define DATA_PTR %rsi /* 2nd arg */ #define NUM_BLKS %rdx /* 3rd arg */ #define SHA256CONSTANTS %rax
@@ -96,40 +96,36 @@ sha256msg1 \m0, \m3 .endif .endm /* - * Intel SHA Extensions optimized implementation of a SHA-256 update function + * Intel SHA Extensions optimized implementation of a SHA-256 block function * - * The function takes a pointer to the current hash values, a pointer to the - * input data, and a number of 64 byte blocks to process. Once all blocks have - * been processed, the digest pointer is updated with the resulting hash value. - * The function only processes complete blocks, there is no functionality to - * store partial blocks. All message padding and hash value initialization must - * be done outside the update function. + * This function takes a pointer to the current SHA-256 state, a pointer to the + * input data, and the number of 64-byte blocks to process. Once all blocks + * have been processed, the state is updated with the new state. This function + * only processes complete blocks. State initialization, buffering of partial + * blocks, and digest finalization is expected to be handled elsewhere. * - * void sha256_ni_transform(uint32_t *digest, const void *data, - uint32_t numBlocks); - * digest : pointer to digest - * data: pointer to input data - * numBlocks: Number of blocks to process + * void sha256_ni_transform(u32 state[SHA256_STATE_WORDS], + * const u8 *data, size_t nblocks); */ - .text -SYM_TYPED_FUNC_START(sha256_ni_transform) +SYM_FUNC_START(sha256_ni_transform) + ANNOTATE_NOENDBR # since this is called only via static_call shl $6, NUM_BLKS /* convert to bytes */ jz .Ldone_hash add DATA_PTR, NUM_BLKS /* pointer to end of data */ /* * load initial hash values * Need to reorder these appropriately * DCBA, HGFE -> ABEF, CDGH */ - movdqu 0*16(DIGEST_PTR), STATE0 /* DCBA */ - movdqu 1*16(DIGEST_PTR), STATE1 /* HGFE */ + movdqu 0*16(STATE_PTR), STATE0 /* DCBA */ + movdqu 1*16(STATE_PTR), STATE1 /* HGFE */ movdqa STATE0, TMP punpcklqdq STATE1, STATE0 /* FEBA */ punpckhqdq TMP, STATE1 /* DCHG */ pshufd $0x1B, STATE0, STATE0 /* ABEF */
@@ -164,12 +160,12 @@ SYM_TYPED_FUNC_START(sha256_ni_transform) punpcklqdq STATE1, STATE0 /* GHEF */ punpckhqdq TMP, STATE1 /* ABCD */ pshufd $0xB1, STATE0, STATE0 /* HGFE */ pshufd $0x1B, STATE1, STATE1 /* DCBA */ - movdqu STATE1, 0*16(DIGEST_PTR) - movdqu STATE0, 1*16(DIGEST_PTR) + movdqu STATE1, 0*16(STATE_PTR) + movdqu STATE0, 1*16(STATE_PTR) .Ldone_hash: RET SYM_FUNC_END(sha256_ni_transform)
arch/x86/lib/crypto/sha256-ssse3-asm.S renamed from arch/x86/crypto/sha256-ssse3-asm.S +6 -8
@@ -45,11 +45,11 @@ # and search for that title. # ######################################################################## #include <linux/linkage.h> -#include <linux/cfi_types.h> +#include <linux/objtool.h> ## assume buffers not aligned #define MOVDQ movdqu ################################ Define Macros
@@ -346,19 +346,17 @@ a = TMP_ add y0, h # h = h + S1 + CH + k + w + S0 + MAJ ROTATE_ARGS .endm ######################################################################## -## void sha256_transform_ssse3(struct sha256_state *state, const u8 *data, -## int blocks); -## arg 1 : pointer to state -## (struct sha256_state is assumed to begin with u32 state[8]) -## arg 2 : pointer to input data -## arg 3 : Num blocks +## void sha256_transform_ssse3(u32 state[SHA256_STATE_WORDS], +## const u8 *data, size_t nblocks); ######################################################################## .text -SYM_TYPED_FUNC_START(sha256_transform_ssse3) +SYM_FUNC_START(sha256_transform_ssse3) + ANNOTATE_NOENDBR # since this is called only via static_call + pushq %rbx pushq %r12 pushq %r13 pushq %r14 pushq %r15
arch/x86/lib/crypto/sha256.c new file +74 -0
74 lines not on this page · raw message
Signature · 1 line
2.49.0
Eric Biggers
maintainerPATCH v4 12/13 crypto: sha256 - remove sha256_base.h
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
sha256_base.h is no longer used, so remove it.
--- include/crypto/sha256_base.h | 183 ----------------------------------- 1 file changed, 183 deletions(-) delete mode 100644 include/crypto/sha256_base.h
include/crypto/sha256_base.h deleted +0 -183
@@ -1,183 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* -* sha256_base.h - core logic for SHA-256 implementations -* -* Copyright (C) 2015 Linaro Ltd <[email protected]> -*/ - -#ifndef _CRYPTO_SHA256_BASE_H -#define _CRYPTO_SHA256_BASE_H - -#include <crypto/internal/hash.h> -#include <crypto/internal/sha2.h> -#include <linux/math.h> -#include <linux/string.h> -#include <linux/types.h> -#include <linux/unaligned.h> - -typedef void (sha256_block_fn )(struct crypto_sha256_state *sst,u8 const *src, -int blocks); - -static inline int sha224_base_init (struct shash_desc *desc) -{ -struct sha256_state *sctx =shash_desc_ctx (desc); - -sha224_init (sctx); -return 0 ; -} - -static inline int sha256_base_init (struct shash_desc *desc) -{ -struct sha256_state *sctx =shash_desc_ctx (desc); - -sha256_init (sctx); -return 0 ; -} - -static inline int lib_sha256_base_do_update (struct sha256_state *sctx, -const u8 *data, -unsigned int len, -sha256_block_fn *block_fn) -{ -unsigned int partial = sctx->count %SHA256_BLOCK_SIZE ; -struct crypto_sha256_state *state = (void *)sctx; - - sctx->count += len; - -if (unlikely ((partial + len) >=SHA256_BLOCK_SIZE )) { -int blocks; - -if (partial) { -int p =SHA256_BLOCK_SIZE - partial; - -memcpy (sctx->buf + partial, data, p); - data += p; - len -= p; - -block_fn (state, sctx->buf,1 ); - } - - blocks = len /SHA256_BLOCK_SIZE ; - len %=SHA256_BLOCK_SIZE ; - -if (blocks) { -block_fn (state, data, blocks); - data += blocks *SHA256_BLOCK_SIZE ; - } - partial =0 ; - } -if (len) -memcpy (sctx->buf + partial, data, len); - -return 0 ; -} - -static inline int lib_sha256_base_do_update_blocks ( -struct crypto_sha256_state *sctx,const u8 *data,unsigned int len, -sha256_block_fn *block_fn) -{ -unsigned int remain = len -round_down (len,SHA256_BLOCK_SIZE ); - - sctx->count += len - remain; -block_fn (sctx, data, len /SHA256_BLOCK_SIZE ); -return remain; -} - -static inline int sha256_base_do_update_blocks ( -struct shash_desc *desc,const u8 *data,unsigned int len, -sha256_block_fn *block_fn) -{ -return lib_sha256_base_do_update_blocks (shash_desc_ctx (desc), data, - len, block_fn); -} - -static inline int lib_sha256_base_do_finup (struct crypto_sha256_state *sctx, -const u8 *src,unsigned int len, -sha256_block_fn *block_fn) -{ -unsigned int bit_offset =SHA256_BLOCK_SIZE /8 -1 ; -union { -__be64 b64[SHA256_BLOCK_SIZE /4 ]; -u8 u8[SHA256_BLOCK_SIZE *2 ]; - } block = {}; - -if (len >= bit_offset *8 ) - bit_offset +=SHA256_BLOCK_SIZE /8 ; -memcpy (&block, src, len); - block.u8[len] =0x80 ; - sctx->count += len; - block.b64[bit_offset] =cpu_to_be64 (sctx->count <<3 ); -block_fn (sctx, block.u8, (bit_offset +1 ) *8 /SHA256_BLOCK_SIZE ); -memzero_explicit (&block,sizeof (block)); - -return 0 ; -} - -static inline int sha256_base_do_finup (struct shash_desc *desc, -const u8 *src,unsigned int len, -sha256_block_fn *block_fn) -{ -struct crypto_sha256_state *sctx =shash_desc_ctx (desc); - -if (len >=SHA256_BLOCK_SIZE ) { -int remain; - - remain =lib_sha256_base_do_update_blocks (sctx, src, len, - block_fn); - src += len - remain; - len = remain; - } -return lib_sha256_base_do_finup (sctx, src, len, block_fn); -} - -static inline int lib_sha256_base_do_finalize (struct sha256_state *sctx, -sha256_block_fn *block_fn) -{ -unsigned int partial = sctx->count %SHA256_BLOCK_SIZE ; -struct crypto_sha256_state *state = (void *)sctx; - - sctx->count -= partial; -return lib_sha256_base_do_finup (state, sctx->buf, partial, block_fn); -} - -static inline int sha256_base_do_finalize (struct shash_desc *desc, -sha256_block_fn *block_fn) -{ -struct sha256_state *sctx =shash_desc_ctx (desc); - -return lib_sha256_base_do_finalize (sctx, block_fn); -} - -static inline int __sha256_base_finish (u32 state[SHA256_DIGEST_SIZE /4 ], -u8 *out,unsigned int digest_size) -{ -__be32 *digest = (__be32 *)out; -int i; - -for (i =0 ; digest_size >0 ; i++, digest_size -=sizeof (__be32)) -put_unaligned_be32 (state[i], digest++); -return 0 ; -} - -static inline void lib_sha256_base_finish (struct sha256_state *sctx,u8 *out, -unsigned int digest_size) -{ -__sha256_base_finish (sctx->state, out, digest_size); -memzero_explicit (sctx,sizeof (*sctx)); -} - -static inline int sha256_base_finish (struct shash_desc *desc,u8 *out) -{ -unsigned int digest_size =crypto_shash_digestsize (desc->tfm); -struct crypto_sha256_state *sctx =shash_desc_ctx (desc); - -return __sha256_base_finish (sctx->state, out, digest_size); -} - -static inline void sha256_transform_blocks (struct crypto_sha256_state *sst, -const u8 *input,int blocks) -{ -sha256_blocks_generic (sst->state, input, blocks); -} - -#endif /* _CRYPTO_SHA256_BASE_H */
Signature · 1 line
2.49.0
Eric Biggers
maintainerPATCH v4 13/13 crypto: lib/sha256 - improve function prototypes
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Follow best practices by changing the length parameters to size_t and explicitly specifying the length of the output digest arrays.
--- include/crypto/sha2.h | 8 ++++---- lib/crypto/sha256.c | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-)
include/crypto/sha2.h +4 -4
@@ -93,13 +93,13 @@ static inline void sha256_init(struct sha256_state *sctx) sctx->state[5 ] =SHA256_H5 ; sctx->state[6 ] =SHA256_H6 ; sctx->state[7 ] =SHA256_H7 ; sctx->count =0 ; } -void sha256_update (struct sha256_state *sctx,const u8 *data,unsigned int len); -void sha256_final (struct sha256_state *sctx,u8 *out); -void sha256 (const u8 *data,unsigned int len,u8 *out); +void sha256_update (struct sha256_state *sctx,const u8 *data,size_t len); +void sha256_final (struct sha256_state *sctx,u8 out[SHA256_DIGEST_SIZE ]); +void sha256 (const u8 *data,size_t len,u8 out[SHA256_DIGEST_SIZE ]);static inline void sha224_init (struct sha256_state *sctx) { sctx->state[0 ] =SHA224_H0 ; sctx->state[1 ] =SHA224_H1 ;
@@ -110,8 +110,8 @@ static inline void sha224_init(struct sha256_state *sctx) sctx->state[6 ] =SHA224_H6 ; sctx->state[7 ] =SHA224_H7 ; sctx->count = 0; }/* Simply use sha256_update as it is equivalent to sha224_update. */ -void sha224_final (struct sha256_state *sctx,u8 *out); +void sha224_final (struct sha256_state *sctx,u8 out[SHA224_DIGEST_SIZE ]);#endif /* _CRYPTO_SHA2_H */
lib/crypto/sha256.c +4 -4
@@ -68,11 +68,11 @@ static inline void __sha256_update(struct sha256_state *sctx, const u8 *data, }if (len)memcpy (&sctx->buf[partial], data, len); } -void sha256_update (struct sha256_state *sctx,const u8 *data,unsigned int len) +void sha256_update (struct sha256_state *sctx,const u8 *data,size_t len) {__sha256_update (sctx, data, len, false); }EXPORT_SYMBOL (sha256_update);
@@ -99,23 +99,23 @@ static inline void __sha256_final(struct sha256_state *sctx, u8 *out,put_unaligned_be32 (sctx->state[i /4 ], out + i);memzero_explicit (sctx,sizeof (*sctx)); } -void sha256_final (struct sha256_state *sctx,u8 *out) +void sha256_final (struct sha256_state *sctx,u8 out[SHA256_DIGEST_SIZE ]) {__sha256_final (sctx, out,SHA256_DIGEST_SIZE , false); }EXPORT_SYMBOL (sha256_final); -void sha224_final (struct sha256_state *sctx,u8 *out) +void sha224_final (struct sha256_state *sctx,u8 out[SHA224_DIGEST_SIZE ]) {__sha256_final (sctx, out,SHA224_DIGEST_SIZE , false); }EXPORT_SYMBOL (sha224_final); -void sha256 (const u8 *data,unsigned int len,u8 *out) +void sha256 (const u8 *data,size_t len,u8 out[SHA256_DIGEST_SIZE ]) {struct sha256_state sctx;sha256_init (&sctx);sha256_update (&sctx, data, len);
Signature · 1 line
2.49.0
Herbert Xu
maintainerQuoted 183 lines · Eric Biggers
This is based on cryptodev commit
2dfc7cd74a5e062a. It can also be retrieved from:git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git sha256-lib-v4
Following the example of several other algorithms (e.g. CRC32, ChaCha, Poly1305, BLAKE2s), this series refactors the kernel's existing architecture-optimized SHA-256 code to be available via the library API, instead of just via the
crypto_shashAPI as it was before. It also reimplements the SHA-256crypto_shashAPI on top of the library API.This makes it possible to use the SHA-256 library in performance-critical cases. The new design is also much simpler, with a negative diffstat of almost 1200 lines. Finally, this also fixes the longstanding issue where the arch-optimized SHA-256 was disabled by default, so people often forgot to enable it.
For now the SHA-256 library is well-covered by the
crypto_shashself-tests, but I plan to add a test for the library directly later. I've fully tested this series on arm, arm64, riscv, and x86. On mips, powerpc, s390, and sparc I've only been able to partially test it, since QEMU does not support the SHA-256 instructions on those platforms. If anyone with access to a mips, powerpc, s390, or sparc system that has SHA-256 instructions can verify that the crypto self-tests still pass,Changed v1 => v4:
- Movedsha256_generic_blocks()into its own module to avoid a circular module dependency.
- Added Ard's Reviewed-by tags.
- Rebased onto cryptodev.Eric Biggers (13):
crypto: sha256 - support arch-optimized lib and expose through shash
crypto: arm/sha256 - implement library instead of shash
crypto: arm64/sha256 - remove obsolete chunking logic
crypto: arm64/sha256 - implement library instead of shash
crypto: mips/sha256 - implement library instead of shash
crypto: powerpc/sha256 - implement library instead of shash
crypto: riscv/sha256 - implement library instead of shash
crypto: s390/sha256 - implement library instead of shash
crypto: sparc - moveopcodes.hinto asm directory
crypto: sparc/sha256 - implement library instead of shash
crypto: x86/sha256 - implement library instead of shash
crypto: sha256 - removesha256_base.h
crypto: lib/sha256 - improve function prototypesarch/arm/configs/exynos_defconfig | 1 - arch/arm/configs/milbeaut_m10v_defconfig | 1 - arch/arm/configs/multi_v7_defconfig | 1 - arch/arm/configs/omap2plus_defconfig | 1 - arch/arm/configs/pxa_defconfig | 1 - arch/arm/crypto/Kconfig | 21 - arch/arm/crypto/Makefile | 8 +- arch/arm/crypto/sha2-ce-glue.c | 87 ---- arch/arm/crypto/sha256_glue.c | 107 ----- arch/arm/crypto/sha256_glue.h | 9 - arch/arm/crypto/sha256_neon_glue.c | 75 --- arch/arm/lib/crypto/.gitignore | 1 + arch/arm/lib/crypto/Kconfig | 6 + arch/arm/lib/crypto/Makefile | 8 +- arch/arm/{ => lib}/crypto/sha256-armv4.pl | 0 .../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 10 +- arch/arm/lib/crypto/sha256.c | 64 +++ arch/arm64/configs/defconfig | 1 - arch/arm64/crypto/Kconfig | 19 - arch/arm64/crypto/Makefile | 13 +- arch/arm64/crypto/sha2-ce-glue.c | 138 ------ arch/arm64/crypto/sha256-glue.c | 171 ------- arch/arm64/lib/crypto/.gitignore | 1 + arch/arm64/lib/crypto/Kconfig | 5 + arch/arm64/lib/crypto/Makefile | 9 +- .../crypto/sha2-armv8.pl} | 0 .../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 36 +- arch/arm64/lib/crypto/sha256.c | 75 +++ arch/mips/cavium-octeon/Kconfig | 6 + .../mips/cavium-octeon/crypto/octeon-sha256.c | 135 ++---- arch/mips/configs/cavium_octeon_defconfig | 1 - arch/mips/crypto/Kconfig | 10 - arch/powerpc/crypto/Kconfig | 11 - arch/powerpc/crypto/Makefile | 2 - arch/powerpc/crypto/sha256-spe-glue.c | 128 ------ arch/powerpc/lib/crypto/Kconfig | 6 + arch/powerpc/lib/crypto/Makefile | 3 + .../powerpc/{ => lib}/crypto/sha256-spe-asm.S | 0 arch/powerpc/lib/crypto/sha256.c | 70 +++ arch/riscv/crypto/Kconfig | 11 - arch/riscv/crypto/Makefile | 3 - arch/riscv/crypto/sha256-riscv64-glue.c | 125 ----- arch/riscv/lib/crypto/Kconfig | 7 + arch/riscv/lib/crypto/Makefile | 3 + .../sha256-riscv64-zvknha_or_zvknhb-zvkb.S | 4 +- arch/riscv/lib/crypto/sha256.c | 62 +++ arch/s390/configs/debug_defconfig | 1 - arch/s390/configs/defconfig | 1 - arch/s390/crypto/Kconfig | 10 - arch/s390/crypto/Makefile | 1 - arch/s390/crypto/sha256_s390.c | 144 ------ arch/s390/lib/crypto/Kconfig | 6 + arch/s390/lib/crypto/Makefile | 2 + arch/s390/lib/crypto/sha256.c | 47 ++ arch/sparc/crypto/Kconfig | 10 - arch/sparc/crypto/Makefile | 2 - arch/sparc/crypto/aes_asm.S | 3 +- arch/sparc/crypto/aes_glue.c | 3 +- arch/sparc/crypto/camellia_asm.S | 3 +- arch/sparc/crypto/camellia_glue.c | 3 +- arch/sparc/crypto/des_asm.S | 3 +- arch/sparc/crypto/des_glue.c | 3 +- arch/sparc/crypto/md5_asm.S | 3 +- arch/sparc/crypto/md5_glue.c | 3 +- arch/sparc/crypto/sha1_asm.S | 3 +- arch/sparc/crypto/sha1_glue.c | 3 +- arch/sparc/crypto/sha256_glue.c | 129 ------ arch/sparc/crypto/sha512_asm.S | 3 +- arch/sparc/crypto/sha512_glue.c | 3 +- arch/sparc/{crypto => include/asm}/opcodes.h | 6 +- arch/sparc/lib/Makefile | 1 + arch/sparc/lib/crc32c_asm.S | 3 +- arch/sparc/lib/crypto/Kconfig | 8 + arch/sparc/lib/crypto/Makefile | 4 + arch/sparc/lib/crypto/sha256.c | 64 +++ arch/sparc/{ => lib}/crypto/sha256_asm.S | 5 +- arch/x86/crypto/Kconfig | 14 - arch/x86/crypto/Makefile | 3 - arch/x86/crypto/sha256_ssse3_glue.c | 432 ------------------ arch/x86/lib/crypto/Kconfig | 7 + arch/x86/lib/crypto/Makefile | 3 + arch/x86/{ => lib}/crypto/sha256-avx-asm.S | 12 +- arch/x86/{ => lib}/crypto/sha256-avx2-asm.S | 12 +- .../crypto/sha256-ni-asm.S} | 36 +- arch/x86/{ => lib}/crypto/sha256-ssse3-asm.S | 14 +- arch/x86/lib/crypto/sha256.c | 74 +++ crypto/Kconfig | 1 + crypto/Makefile | 3 +- crypto/sha256.c | 201 ++++++++ crypto/sha256_generic.c | 102 ----- include/crypto/internal/sha2.h | 28 ++ include/crypto/sha2.h | 23 +- include/crypto/sha256_base.h | 180 -------- lib/crypto/Kconfig | 22 + lib/crypto/Makefile | 3 + lib/crypto/sha256-generic.c | 137 ++++++ lib/crypto/sha256.c | 204 ++++----- 97 files changed, 1128 insertions(+), 2319 deletions(-)delete mode 100644
arch/arm/crypto/sha2-ce-glue.c
delete mode 100644arch/arm/crypto/sha256_glue.c
delete mode 100644arch/arm/crypto/sha256_glue.h
delete mode 100644arch/arm/crypto/sha256_neon_glue.c
renamearch/arm/{=>lib}/crypto/sha256-armv4.pl(100%)
renamearch/arm/{crypto/sha2-ce-core.S=>lib/crypto/sha256-ce.S} (91%) create mode 100644arch/arm/lib/crypto/sha256.c
delete mode 100644arch/arm64/crypto/sha2-ce-glue.c
delete mode 100644arch/arm64/crypto/sha256-glue.c
renamearch/arm64/{crypto/sha512-armv8.pl=>lib/crypto/sha2-armv8.pl} (100%) renamearch/arm64/{crypto/sha2-ce-core.S=>lib/crypto/sha256-ce.S} (80%) create mode 100644arch/arm64/lib/crypto/sha256.c
delete mode 100644arch/powerpc/crypto/sha256-spe-glue.c
renamearch/powerpc/{=>lib}/crypto/sha256-spe-asm.S(100%) create mode 100644arch/powerpc/lib/crypto/sha256.c
delete mode 100644arch/riscv/crypto/sha256-riscv64-glue.c
renamearch/riscv/{=>lib}/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S(98%) create mode 100644arch/riscv/lib/crypto/sha256.c
delete mode 100644arch/s390/crypto/sha256_s390.c
create mode 100644arch/s390/lib/crypto/sha256.c
delete mode 100644arch/sparc/crypto/sha256_glue.c
renamearch/sparc/{crypto=>include/asm}/opcodes.h(96%)
create mode 100644arch/sparc/lib/crypto/Kconfig
create mode 100644arch/sparc/lib/crypto/Makefile
create mode 100644arch/sparc/lib/crypto/sha256.c
renamearch/sparc/{=>lib}/crypto/sha256_asm.S(95%)
delete mode 100644arch/x86/crypto/sha256_ssse3_glue.c
renamearch/x86/{=>lib}/crypto/sha256-avx-asm.S(98%)
renamearch/x86/{=>lib}/crypto/sha256-avx2-asm.S(98%)
renamearch/x86/{crypto/sha256_ni_asm.S=>lib/crypto/sha256-ni-asm.S} (85%) renamearch/x86/{=>lib}/crypto/sha256-ssse3-asm.S(98%)
create mode 100644arch/x86/lib/crypto/sha256.c
create mode 100644crypto/sha256.c
delete mode 100644crypto/sha256_generic.c
create mode 100644include/crypto/internal/sha2.h
delete mode 100644include/crypto/sha256_base.h
create mode 100644lib/crypto/sha256-generic.c
Quoted 1 line
base-commit:
2dfc7cd74a5e062a5405560447517e7aab1c7341
All applied with export/import addition to patch 1. Thanks.
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt