SHA-512 library functions
Subsystems (15)
Eric Biggers
maintainerv3 cover SHA-512 library functions
[ I had applied v2 of this earlier, but since it's been accumulating
some minor fixups and it's still -rc4, I figured I'd just refresh it ]
This series is also available at:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git sha512-lib-v3
This series adds support for SHA-384, SHA-512, HMAC-SHA384, and HMAC-SHA512 to lib/crypto/. The new functions take advantage of the kernel's existing architecture-optimized implementations of the SHA-512 compression function. The new functions are fully tested using KUnit.
To avoid duplicating all arch-optimized implementations of the SHA-512 compression function (~3000 lines of code total), they are moved into lib/crypto/ rather than copied. To make the "sha384", "sha512", "hmac(sha384)", and "hmac(sha512)" crypto_shash algorithms in the old-school crypto API continue to be properly optimized after that, they are reimplemented on top of lib/crypto/, which is straightforward.
The following lists some of the design choices and conventions that I've followed in more detail. Where these differ from the code or APIs for other algorithms (e.g., SHA-256 in some cases), I'd like to do it this way going forward and plan to fix up the other algorithms accordingly:
- APIs are fully documented with kerneldoc comments.
- APIs cannot fail, and return void.
- APIs work in all contexts. This doesn't mean that they *should* be called in all contexts, but rather they always just work as expected.
- Tests are KUnit tests, and they are fairly thorough (more thorough than crypto/testmgr.c) and also optionally include benchmarks.
- Architecture-optimized code is integrated the same way I'm doing it for lib/crc/: it's in subdirectories lib/crypto/$(SRCARCH), it's enabled by default, and it's inlined into the same module as the generic code. This solves a number of problems; for more details, see
https://lore.kernel.org/r/[email protected]
- HMAC support is a first-class citizen.
- APIs handle zeroization, when applicable.
- Message contexts are *_ctx instead of *_state. It's shorter, avoids ambiguity with the compression function state, and matches OpenSSL.
- Length arguments are size_t, are in bytes, are named len or *_len, and immediately follow the corresponding buffer. "Object" being operated on is first argument; outputs otherwise follow inputs.
- The structures for different algorithms use different types, which prevents usage errors where functions are mixed up between algorithms.
- The compression function state is strongly typed, not a plain array.
Changed in v3:
- Dropped KUnit tests from this series, since they'll be separate and will need to go last in the branch (as per request from Linus)
- Folded in the following fixups I previously sent:
"crypto: testmgr/sha512 - Omit generic_driver for no-op tests", "crypto: sha512 - Use the correct legacy export format", "lib/crypto: sha512: Reorder some code in sha512.c", "lib/crypto: sha512: Do not include <crypto/internal/sha2.h>", "lib/crypto: sha512: Fix a grammatical error in kerneldoc comments", "lib/crypto: x86/sha512: Drop unnecessary ANNOTATE_NOENDBR"
- Updated cra_priority values to be consistent with sha256.c and not decreased from what the arch-optimized code used before
- Added MODULE_ALIAS_CRYPTO() lines for the *-lib cra_driver_names
- Updated commit titles to align with the conventions I'm adopting
Changed in v2:
- Added "crypto: sha512 - use same state format as legacy drivers"
- Fixed build on user-mode Linux
- Fixed W=1 build warning by adding <linux/export.h>
- Optimized __sha512_final() and __hmac_sha512_final() slightly
Eric Biggers (16):
crypto: sha512 - Rename conflicting symbols
lib/crypto: sha512: Add support for SHA-384 and SHA-512 lib/crypto: sha512: Add HMAC-SHA384 and HMAC-SHA512 support
crypto: riscv/sha512 - Stop depending on sha512_generic_block_fn
crypto: sha512 - Replace sha512_generic with wrapper around SHA-512
library
crypto: sha512 - Use same state format as legacy drivers lib/crypto: arm/sha512: Migrate optimized SHA-512 code to library lib/crypto: arm64/sha512: Migrate optimized SHA-512 code to library
mips: cavium-octeon: Move octeon-crypto.h into asm directory lib/crypto: mips/sha512: Migrate optimized SHA-512 code to library lib/crypto: riscv/sha512: Migrate optimized SHA-512 code to library lib/crypto: s390/sha512: Migrate optimized SHA-512 code to library lib/crypto: sparc/sha512: Migrate optimized SHA-512 code to library lib/crypto: x86/sha512: Migrate optimized SHA-512 code to library lib/crypto: x86/sha512: Remove unnecessary checks for nblocks==0
crypto: sha512 - Remove sha512_base.h
arch/arm/configs/exynos_defconfig | 1 -
arch/arm/configs/milbeaut_m10v_defconfig | 1 -
arch/arm/configs/multi_v7_defconfig | 1 -
arch/arm/configs/omap2plus_defconfig | 1 -
arch/arm/configs/pxa_defconfig | 1 -
arch/arm/crypto/Kconfig | 10 -
arch/arm/crypto/Makefile | 15 -
arch/arm/crypto/sha512-glue.c | 110 -----
arch/arm/crypto/sha512-neon-glue.c | 75 ----
arch/arm/crypto/sha512.h | 3 -
arch/arm64/configs/defconfig | 1 -
arch/arm64/crypto/Kconfig | 19 -
arch/arm64/crypto/Makefile | 14 -
arch/arm64/crypto/sha512-ce-glue.c | 96 -----
arch/arm64/crypto/sha512-glue.c | 83 ----
arch/mips/cavium-octeon/crypto/Makefile | 1 -
.../mips/cavium-octeon/crypto/octeon-crypto.c | 3 +-
arch/mips/cavium-octeon/crypto/octeon-md5.c | 3 +-
arch/mips/cavium-octeon/crypto/octeon-sha1.c | 3 +-
.../mips/cavium-octeon/crypto/octeon-sha256.c | 3 +-
.../mips/cavium-octeon/crypto/octeon-sha512.c | 167 --------
arch/mips/configs/cavium_octeon_defconfig | 1 -
arch/mips/crypto/Kconfig | 10 -
.../asm/octeon/crypto.h} | 0
arch/riscv/crypto/Kconfig | 11 -
arch/riscv/crypto/Makefile | 3 -
arch/riscv/crypto/sha512-riscv64-glue.c | 124 ------
arch/s390/configs/debug_defconfig | 1 -
arch/s390/configs/defconfig | 1 -
arch/s390/crypto/Kconfig | 10 -
arch/s390/crypto/Makefile | 1 -
arch/s390/crypto/sha512_s390.c | 151 -------
arch/sparc/crypto/Kconfig | 10 -
arch/sparc/crypto/Makefile | 2 -
arch/sparc/crypto/sha512_glue.c | 122 ------
arch/x86/crypto/Kconfig | 13 -
arch/x86/crypto/Makefile | 3 -
arch/x86/crypto/sha512_ssse3_glue.c | 322 --------------
crypto/Kconfig | 4 +-
crypto/Makefile | 2 +-
crypto/sha512.c | 354 +++++++++++++++
crypto/sha512_generic.c | 217 ----------
crypto/testmgr.c | 10 +
drivers/crypto/starfive/jh7110-hash.c | 8 +-
include/crypto/sha2.h | 350 +++++++++++++++
include/crypto/sha512_base.h | 120 ------
lib/crypto/Kconfig | 18 +
lib/crypto/Makefile | 36 ++
lib/crypto/arm/.gitignore | 2 +
.../crypto => lib/crypto/arm}/sha512-armv4.pl | 0
lib/crypto/arm/sha512.h | 38 ++
lib/crypto/arm64/.gitignore | 2 +
.../crypto/arm64}/sha512-ce-core.S | 10 +-
lib/crypto/arm64/sha512.h | 46 ++
lib/crypto/mips/sha512.h | 74 ++++
.../riscv}/sha512-riscv64-zvknhb-zvkb.S | 4 +-
lib/crypto/riscv/sha512.h | 41 ++
lib/crypto/s390/sha512.h | 28 ++
lib/crypto/sha512.c | 402 ++++++++++++++++++
lib/crypto/sparc/sha512.h | 42 ++
.../crypto => lib/crypto/sparc}/sha512_asm.S | 0
.../crypto/x86}/sha512-avx-asm.S | 11 +-
.../crypto/x86}/sha512-avx2-asm.S | 12 +-
.../crypto/x86}/sha512-ssse3-asm.S | 14 +-
lib/crypto/x86/sha512.h | 54 +++
65 files changed, 1529 insertions(+), 1766 deletions(-)
delete mode 100644 arch/arm/crypto/sha512-glue.c
delete mode 100644 arch/arm/crypto/sha512-neon-glue.c
delete mode 100644 arch/arm/crypto/sha512.h
delete mode 100644 arch/arm64/crypto/sha512-ce-glue.c
delete mode 100644 arch/arm64/crypto/sha512-glue.c
delete mode 100644 arch/mips/cavium-octeon/crypto/octeon-sha512.c
rename arch/mips/{cavium-octeon/crypto/octeon-crypto.h => include/asm/octeon/crypto.h} (100%)
delete mode 100644 arch/riscv/crypto/sha512-riscv64-glue.c
delete mode 100644 arch/s390/crypto/sha512_s390.c
delete mode 100644 arch/sparc/crypto/sha512_glue.c
delete mode 100644 arch/x86/crypto/sha512_ssse3_glue.c
create mode 100644 crypto/sha512.c
delete mode 100644 crypto/sha512_generic.c
delete mode 100644 include/crypto/sha512_base.h
create mode 100644 lib/crypto/arm/.gitignore
rename {arch/arm/crypto => lib/crypto/arm}/sha512-armv4.pl (100%)
create mode 100644 lib/crypto/arm/sha512.h
create mode 100644 lib/crypto/arm64/.gitignore
rename {arch/arm64/crypto => lib/crypto/arm64}/sha512-ce-core.S (97%)
create mode 100644 lib/crypto/arm64/sha512.h
create mode 100644 lib/crypto/mips/sha512.h
rename {arch/riscv/crypto => lib/crypto/riscv}/sha512-riscv64-zvknhb-zvkb.S (98%)
create mode 100644 lib/crypto/riscv/sha512.h
create mode 100644 lib/crypto/s390/sha512.h
create mode 100644 lib/crypto/sha512.c
create mode 100644 lib/crypto/sparc/sha512.h
rename {arch/sparc/crypto => lib/crypto/sparc}/sha512_asm.S (100%)
rename {arch/x86/crypto => lib/crypto/x86}/sha512-avx-asm.S (98%)
rename {arch/x86/crypto => lib/crypto/x86}/sha512-avx2-asm.S (99%)
rename {arch/x86/crypto => lib/crypto/x86}/sha512-ssse3-asm.S (97%)
create mode 100644 lib/crypto/x86/sha512.hSignature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 1/16 crypto: sha512 - Rename conflicting symbols
Rename existing functions and structs in architecture-optimized SHA-512 code that had names conflicting with the upcoming library interface which will be added to <crypto/sha2.h>: sha384_init, sha512_init, sha512_update, sha384, and sha512.
Note: all affected code will be superseded by later commits that migrate the arch-optimized SHA-512 code into the library. This commit simply keeps the kernel building for the initial introduction of the library.
--- arch/arm64/crypto/sha512-glue.c | 8 ++++---- arch/s390/crypto/sha512_s390.c | 8 ++++---- arch/sparc/crypto/sha512_glue.c | 14 +++++++------- arch/x86/crypto/sha512_ssse3_glue.c | 10 +++++----- 4 files changed, 20 insertions(+), 20 deletions(-)
arch/arm64/crypto/sha512-glue.c +4 -4
@@ -25,12 +25,12 @@ static void sha512_arm64_transform(struct sha512_state *sst, u8 const *src, int blocks) {sha512_blocks_arch (sst->state, src, blocks); } -static int sha512_update (struct shash_desc *desc,const u8 *data, -unsigned int len) +static int sha512_update_arm64 (struct shash_desc *desc,const u8 *data, +unsigned int len) {return sha512_base_do_update_blocks (desc, data, len, sha512_arm64_transform); }
@@ -42,11 +42,11 @@ static int sha512_finup(struct shash_desc *desc, const u8 *data, }static struct shash_alg algs[] = { { .digestsize =SHA512_DIGEST_SIZE , .init = sha512_base_init, - .update = sha512_update, + .update = sha512_update_arm64, .finup = sha512_finup, .descsize =SHA512_STATE_SIZE , .base.cra_name ="sha512" , .base.cra_driver_name ="sha512-arm64" , .base.cra_priority =150 ,
@@ -55,11 +55,11 @@ static struct shash_alg algs[] = { { .base.cra_blocksize = SHA512_BLOCK_SIZE, .base.cra_module = THIS_MODULE, }, { .digestsize = SHA384_DIGEST_SIZE, .init = sha384_base_init, - .update = sha512_update, + .update = sha512_update_arm64, .finup = sha512_finup, .descsize = SHA512_STATE_SIZE, .base.cra_name = "sha384", .base.cra_driver_name = "sha384-arm64", .base.cra_priority = 150,
arch/s390/crypto/sha512_s390.c +4 -4
@@ -15,11 +15,11 @@#include <linux/kernel.h> #include <linux/module.h> #include "sha.h" -static int sha512_init (struct shash_desc *desc) +static int sha512_init_s390 (struct shash_desc *desc) {struct s390_sha_ctx *ctx =shash_desc_ctx (desc); ctx->sha512.state[0 ] =SHA512_H0 ; ctx->sha512.state[1 ] =SHA512_H1 ;
@@ -60,11 +60,11 @@ static int sha512_import(struct shash_desc *desc, const void *in)return 0 ; }static struct shash_alg sha512_alg = { .digestsize =SHA512_DIGEST_SIZE , - .init = sha512_init, + .init = sha512_init_s390, .update = s390_sha_update_blocks, .finup = s390_sha_finup, .export = sha512_export, .import = sha512_import, .descsize =sizeof (struct s390_sha_ctx ),
@@ -80,11 +80,11 @@ static struct shash_alg sha512_alg = { } };MODULE_ALIAS_CRYPTO ("sha512" ); -static int sha384_init (struct shash_desc *desc) +static int sha384_init_s390 (struct shash_desc *desc) {struct s390_sha_ctx *ctx =shash_desc_ctx (desc); ctx->sha512.state[0 ] =SHA384_H0 ; ctx->sha512.state[1 ] =SHA384_H1 ;
@@ -101,11 +101,11 @@ static int sha384_init(struct shash_desc *desc)return 0 ; }static struct shash_alg sha384_alg = { .digestsize =SHA384_DIGEST_SIZE , - .init = sha384_init, + .init = sha384_init_s390, .update = s390_sha_update_blocks, .finup = s390_sha_finup, .export = sha512_export, .import = sha512_import, .descsize =sizeof (struct s390_sha_ctx ),
arch/sparc/crypto/sha512_glue.c +7 -7
@@ -38,11 +38,11 @@ static int sha512_sparc64_finup(struct shash_desc *desc, const u8 *src, {sha512_base_do_finup (desc, src, len, sha512_block);return sha512_base_finish (desc, out); } -static struct shash_alg sha512 = { +static struct shash_alg sha512_alg = { .digestsize =SHA512_DIGEST_SIZE , .init = sha512_base_init, .update = sha512_sparc64_update, .finup = sha512_sparc64_finup, .descsize =SHA512_STATE_SIZE ,
@@ -53,11 +53,11 @@ static struct shash_alg sha512 = { .cra_blocksize = SHA512_BLOCK_SIZE, .cra_module = THIS_MODULE, } }; -static struct shash_alg sha384 = { +static struct shash_alg sha384_alg = { .digestsize =SHA384_DIGEST_SIZE , .init = sha384_base_init, .update = sha512_sparc64_update, .finup = sha512_sparc64_finup, .descsize =SHA512_STATE_SIZE ,
@@ -85,17 +85,17 @@ static bool __init sparc64_has_sha512_opcode(void) }static int __init sha512_sparc64_mod_init (void ) {if (sparc64_has_sha512_opcode ()) { -int ret =crypto_register_shash (&sha384); +int ret =crypto_register_shash (&sha384_alg);if (ret <0 )return ret; - ret =crypto_register_shash (&sha512); + ret =crypto_register_shash (&sha512_alg);if (ret <0 ) { -crypto_unregister_shash (&sha384); +crypto_unregister_shash (&sha384_alg);return ret; }pr_info ("Using sparc64 sha512 opcode optimized SHA-512/SHA-384 implementation\n" );return 0 ;
@@ -104,12 +104,12 @@ static int __init sha512_sparc64_mod_init(void)return -ENODEV ; } static void __exit sha512_sparc64_mod_fini(void) { -crypto_unregister_shash (&sha384); -crypto_unregister_shash (&sha512); +crypto_unregister_shash (&sha384_alg); +crypto_unregister_shash (&sha512_alg); }module_init (sha512_sparc64_mod_init);module_exit (sha512_sparc64_mod_fini);
arch/x86/crypto/sha512_ssse3_glue.c +5 -5
@@ -36,12 +36,12 @@#include <crypto/sha512_base.h> asmlinkage void sha512_transform_ssse3(struct sha512_state *state,const u8 *data,int blocks); -static int sha512_update (struct shash_desc *desc,const u8 *data, -unsigned int len,sha512_block_fn *sha512_xform) +static int sha512_update_x86 (struct shash_desc *desc,const u8 *data, +unsigned int len,sha512_block_fn *sha512_xform) {int remain;/* * Make sure struct sha512_state begins directly with the SHA512
@@ -67,11 +67,11 @@ static int sha512_finup(struct shash_desc *desc, const u8 *data, }static int sha512_ssse3_update (struct shash_desc *desc,const u8 *data,unsigned int len) { -return sha512_update (desc, data, len, sha512_transform_ssse3); +return sha512_update_x86 (desc, data, len, sha512_transform_ssse3); }static int sha512_ssse3_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) {
@@ -139,11 +139,11 @@ static bool avx_usable(void) }static int sha512_avx_update (struct shash_desc *desc,const u8 *data,unsigned int len) { -return sha512_update (desc, data, len, sha512_transform_avx); +return sha512_update_x86 (desc, data, len, sha512_transform_avx); }static int sha512_avx_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) {
@@ -201,11 +201,11 @@ asmlinkage void sha512_transform_rorx(struct sha512_state *state, const u8 *data, int blocks);static int sha512_avx2_update (struct shash_desc *desc,const u8 *data,unsigned int len) { -return sha512_update (desc, data, len, sha512_transform_rorx); +return sha512_update_x86 (desc, data, len, sha512_transform_rorx); }static int sha512_avx2_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) {
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 2/16 lib/crypto: sha512: Add support for SHA-384 and SHA-512
In reply to Eric BiggersAdd basic support for SHA-384 and SHA-512 to lib/crypto/.
Various in-kernel users will be able to use this instead of the old-school crypto API, which is harder to use and has more overhead.
The basic support added by this commit consists of the API and its documentation, backed by a C implementation of the algorithms. sha512_block_generic() is derived from crypto/sha512_generic.c.
--- include/crypto/sha2.h | 128 ++++++++++++++++++++ lib/crypto/Kconfig | 10 ++ lib/crypto/Makefile | 6 + lib/crypto/sha512.c | 265 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 409 insertions(+) create mode 100644 lib/crypto/sha512.c
include/crypto/sha2.h +128 -0
@@ -127,6 +127,134 @@ static inline void sha224_init(struct sha256_state *sctx) sha224_block_init(&sctx->ctx); }/* Simply use sha256_update as it is equivalent to sha224_update. */ void sha224_final (struct sha256_state *sctx,u8 out[SHA224_DIGEST_SIZE ]); +/* State for the SHA-512 (and SHA-384) compression function */ +struct sha512_block_state { +u64 h[8 ]; +}; + +/* +* Context structure, shared by SHA-384 and SHA-512. The sha384_ctx and +* sha512_ctx structs wrap this one so that the API has proper typing and +* doesn't allow mixing the SHA-384 and SHA-512 functions arbitrarily. +*/ +struct __sha512_ctx { +struct sha512_block_state state; +u64 bytecount_lo; +u64 bytecount_hi; +u8 buf[SHA512_BLOCK_SIZE ]__aligned(__alignof__(__be64)) ; +}; +void __sha512_update (struct __sha512_ctx *ctx,const u8 *data,size_t len); + +/** +* struct sha384_ctx - Context for hashing a message with SHA-384 +* @ctx: private +*/ +struct sha384_ctx { +struct __sha512_ctx ctx; +}; + +/** +* sha384_init() - Initialize a SHA-384 context for a new message +* @ctx: the context to initialize +* +* If you don't need incremental computation, consider sha384() instead. +* +* Context: Any context. +*/ +void sha384_init (struct sha384_ctx *ctx); + +/** +* sha384_update() - Update a SHA-384 context with message data +* @ctx: the context to update; must have been initialized +* @data: the message data +* @len: the data length in bytes +* +* This can be called any number of times. +* +* Context: Any context. +*/ +static inline void sha384_update (struct sha384_ctx *ctx, +const u8 *data,size_t len) +{ +__sha512_update (&ctx->ctx, data, len); +} + +/** +* sha384_final() - Finish computing a SHA-384 message digest +* @ctx: the context to finalize; must have been initialized +* @out: (output) the resulting SHA-384 message digest +* +* After finishing, this zeroizes @ctx. So the caller does not need to do it. +* +* Context: Any context. +*/ +void sha384_final (struct sha384_ctx *ctx,u8 out[SHA384_DIGEST_SIZE ]); + +/** +* sha384() - Compute SHA-384 message digest in one shot +* @data: the message data +* @len: the data length in bytes +* @out: (output) the resulting SHA-384 message digest +* +* Context: Any context. +*/ +void sha384 (const u8 *data,size_t len,u8 out[SHA384_DIGEST_SIZE ]); + +/** +* struct sha512_ctx - Context for hashing a message with SHA-512 +* @ctx: private +*/ +struct sha512_ctx { +struct __sha512_ctx ctx; +}; + +/** +* sha512_init() - Initialize a SHA-512 context for a new message +* @ctx: the context to initialize +* +* If you don't need incremental computation, consider sha512() instead. +* +* Context: Any context. +*/ +void sha512_init (struct sha512_ctx *ctx); + +/** +* sha512_update() - Update a SHA-512 context with message data +* @ctx: the context to update; must have been initialized +* @data: the message data +* @len: the data length in bytes +* +* This can be called any number of times. +* +* Context: Any context. +*/ +static inline void sha512_update (struct sha512_ctx *ctx, +const u8 *data,size_t len) +{ +__sha512_update (&ctx->ctx, data, len); +} + +/** +* sha512_final() - Finish computing a SHA-512 message digest +* @ctx: the context to finalize; must have been initialized +* @out: (output) the resulting SHA-512 message digest +* +* After finishing, this zeroizes @ctx. So the caller does not need to do it. +* +* Context: Any context. +*/ +void sha512_final (struct sha512_ctx *ctx,u8 out[SHA512_DIGEST_SIZE ]); + +/** +* sha512() - Compute SHA-512 message digest in one shot +* @data: the message data +* @len: the data length in bytes +* @out: (output) the resulting SHA-512 message digest +* +* Context: Any context. +*/ +void sha512 (const u8 *data,size_t len,u8 out[SHA512_DIGEST_SIZE ]); +#endif /* _CRYPTO_SHA2_H */
lib/crypto/Kconfig +10 -0
@@ -165,10 +165,20 @@ config CRYPTO_LIB_SHA256_GENERIC This symbol can be selected by arch implementations of the SHA-256 library interface that require the generic code as a fallback, e.g., for SIMD implementations. If no arch specific implementation is enabled, this implementation serves the users of CRYPTO_LIB_SHA256. +config CRYPTO_LIB_SHA512 + tristate + help + The SHA-384 and SHA-512 library functions. Select this if your module + uses any of these functions from <crypto/sha2.h>. + +config CRYPTO_LIB_SHA512_ARCH + bool + depends on CRYPTO_LIB_SHA512 && !UML + config CRYPTO_LIB_SM3 tristate if !KMSAN # avoid false positives from assembly if ARM
lib/crypto/Makefile +6 -0
@@ -62,10 +62,16 @@ obj-$(CONFIG_CRYPTO_LIB_SHA256) += libsha256.o libsha256-y := sha256.o obj-$(CONFIG_CRYPTO_LIB_SHA256_GENERIC) += libsha256-generic.o libsha256-generic-y := sha256-generic.o +obj-$(CONFIG_CRYPTO_LIB_SHA512) += libsha512.o +libsha512-y := sha512.o +ifeq ($(CONFIG_CRYPTO_LIB_SHA512_ARCH),y) +CFLAGS_sha512.o += -I$(src)/$(SRCARCH) +endif # CONFIG_CRYPTO_LIB_SHA512_ARCH + obj-$(CONFIG_MPILIB) += mpi/ obj-$(CONFIG_CRYPTO_SELFTESTS_FULL) += simd.o obj-$(CONFIG_CRYPTO_LIB_SM3) += libsm3.o
lib/crypto/sha512.c new file +265 -0
265 lines not on this page · raw message
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 3/16 lib/crypto: sha512: Add HMAC-SHA384 and HMAC-SHA512 support
In reply to Eric BiggersSince HMAC support is commonly needed and is fairly simple, include it as a first-class citizen of the SHA-512 library.
The API supports both incremental and one-shot computation, and either preparing the key ahead of time or just using a raw key. The implementation is much more streamlined than crypto/hmac.c.
--- include/crypto/sha2.h | 222 ++++++++++++++++++++++++++++++++++++++++++ lib/crypto/Kconfig | 5 +- lib/crypto/sha512.c | 141 ++++++++++++++++++++++++++- 3 files changed, 364 insertions(+), 4 deletions(-)
include/crypto/sha2.h +222 -0
@@ -145,10 +145,26 @@ struct __sha512_ctx {u64 bytecount_hi;u8 buf[SHA512_BLOCK_SIZE ]__aligned(__alignof__(__be64)) ; };void __sha512_update (struct __sha512_ctx *ctx,const u8 *data,size_t len); +/* +* HMAC key and message context structs, shared by HMAC-SHA384 and HMAC-SHA512. +* The hmac_sha384_* and hmac_sha512_* structs wrap this one so that the API has +* proper typing and doesn't allow mixing the functions arbitrarily. +*/ +struct __hmac_sha512_key { +struct sha512_block_state istate; +struct sha512_block_state ostate; +}; +struct __hmac_sha512_ctx { +struct __sha512_ctx sha_ctx; +struct sha512_block_state ostate; +}; +void __hmac_sha512_init (struct __hmac_sha512_ctx *ctx, +const struct __hmac_sha512_key *key); +/** * struct sha384_ctx - Context for hashing a message with SHA-384 * @ctx: private */ struct sha384_ctx {
@@ -200,10 +216,113 @@ void sha384_final(struct sha384_ctx *ctx, u8 out[SHA384_DIGEST_SIZE]);* * Context: Any context. */ void sha384 (const u8 *data,size_t len,u8 out[SHA384_DIGEST_SIZE ]); +/** +* struct hmac_sha384_key - Prepared key for HMAC-SHA384 +* @key: private +*/ +struct hmac_sha384_key { +struct __hmac_sha512_key key; +}; + +/** +* struct hmac_sha384_ctx - Context for computing HMAC-SHA384 of a message +* @ctx: private +*/ +struct hmac_sha384_ctx { +struct __hmac_sha512_ctx ctx; +}; + +/** +* hmac_sha384_preparekey() - Prepare a key for HMAC-SHA384 +* @key: (output) the key structure to initialize +* @raw_key: the raw HMAC-SHA384 key +* @raw_key_len: the key length in bytes. All key lengths are supported. +* +* Note: the caller is responsible for zeroizing both the struct hmac_sha384_key +* and the raw key once they are no longer needed. +* +* Context: Any context. +*/ +void hmac_sha384_preparekey (struct hmac_sha384_key *key, +const u8 *raw_key,size_t raw_key_len); + +/** +* hmac_sha384_init() - Initialize an HMAC-SHA384 context for a new message +* @ctx: (output) the HMAC context to initialize +* @key: the prepared HMAC key +* +* If you don't need incremental computation, consider hmac_sha384() instead. +* +* Context: Any context. +*/ +static inline void hmac_sha384_init (struct hmac_sha384_ctx *ctx, +const struct hmac_sha384_key *key) +{ +__hmac_sha512_init (&ctx->ctx, &key->key); +} + +/** +* hmac_sha384_update() - Update an HMAC-SHA384 context with message data +* @ctx: the HMAC context to update; must have been initialized +* @data: the message data +* @data_len: the data length in bytes +* +* This can be called any number of times. +* +* Context: Any context. +*/ +static inline void hmac_sha384_update (struct hmac_sha384_ctx *ctx, +const u8 *data,size_t data_len) +{ +__sha512_update (&ctx->ctx.sha_ctx, data, data_len); +} + +/** +* hmac_sha384_final() - Finish computing an HMAC-SHA384 value +* @ctx: the HMAC context to finalize; must have been initialized +* @out: (output) the resulting HMAC-SHA384 value +* +* After finishing, this zeroizes @ctx. So the caller does not need to do it. +* +* Context: Any context. +*/ +void hmac_sha384_final (struct hmac_sha384_ctx *ctx,u8 out[SHA384_DIGEST_SIZE ]); + +/** +* hmac_sha384() - Compute HMAC-SHA384 in one shot, using a prepared key +* @key: the prepared HMAC key +* @data: the message data +* @data_len: the data length in bytes +* @out: (output) the resulting HMAC-SHA384 value +* +* If you're using the key only once, consider using hmac_sha384_usingrawkey(). +* +* Context: Any context. +*/ +void hmac_sha384 (const struct hmac_sha384_key *key, +const u8 *data,size_t data_len,u8 out[SHA384_DIGEST_SIZE ]); + +/** +* hmac_sha384_usingrawkey() - Compute HMAC-SHA384 in one shot, using a raw key +* @raw_key: the raw HMAC-SHA384 key +* @raw_key_len: the key length in bytes. All key lengths are supported. +* @data: the message data +* @data_len: the data length in bytes +* @out: (output) the resulting HMAC-SHA384 value +* +* If you're using the key multiple times, prefer to use +* hmac_sha384_preparekey() followed by multiple calls to hmac_sha384() instead. +* +* Context: Any context. +*/ +void hmac_sha384_usingrawkey (const u8 *raw_key,size_t raw_key_len, +const u8 *data,size_t data_len, +u8 out[SHA384_DIGEST_SIZE ]); +/** * struct sha512_ctx - Context for hashing a message with SHA-512 * @ctx: private */ struct sha512_ctx {
@@ -255,6 +374,109 @@ void sha512_final(struct sha512_ctx *ctx, u8 out[SHA512_DIGEST_SIZE]);* * Context: Any context. */ void sha512 (const u8 *data,size_t len,u8 out[SHA512_DIGEST_SIZE ]); +/** +* struct hmac_sha512_key - Prepared key for HMAC-SHA512 +* @key: private +*/ +struct hmac_sha512_key { +struct __hmac_sha512_key key; +}; + +/** +* struct hmac_sha512_ctx - Context for computing HMAC-SHA512 of a message +* @ctx: private +*/ +struct hmac_sha512_ctx { +struct __hmac_sha512_ctx ctx; +}; + +/** +* hmac_sha512_preparekey() - Prepare a key for HMAC-SHA512 +* @key: (output) the key structure to initialize +* @raw_key: the raw HMAC-SHA512 key +* @raw_key_len: the key length in bytes. All key lengths are supported. +* +* Note: the caller is responsible for zeroizing both the struct hmac_sha512_key +* and the raw key once they are no longer needed. +* +* Context: Any context. +*/ +void hmac_sha512_preparekey (struct hmac_sha512_key *key, +const u8 *raw_key,size_t raw_key_len); + +/** +* hmac_sha512_init() - Initialize an HMAC-SHA512 context for a new message +* @ctx: (output) the HMAC context to initialize +* @key: the prepared HMAC key +* +* If you don't need incremental computation, consider hmac_sha512() instead. +* +* Context: Any context. +*/ +static inline void hmac_sha512_init (struct hmac_sha512_ctx *ctx, +const struct hmac_sha512_key *key) +{ +__hmac_sha512_init (&ctx->ctx, &key->key); +} + +/** +* hmac_sha512_update() - Update an HMAC-SHA512 context with message data +* @ctx: the HMAC context to update; must have been initialized +* @data: the message data +* @data_len: the data length in bytes +* +* This can be called any number of times. +* +* Context: Any context. +*/ +static inline void hmac_sha512_update (struct hmac_sha512_ctx *ctx, +const u8 *data,size_t data_len) +{ +__sha512_update (&ctx->ctx.sha_ctx, data, data_len); +} + +/** +* hmac_sha512_final() - Finish computing an HMAC-SHA512 value +* @ctx: the HMAC context to finalize; must have been initialized +* @out: (output) the resulting HMAC-SHA512 value +* +* After finishing, this zeroizes @ctx. So the caller does not need to do it. +* +* Context: Any context. +*/ +void hmac_sha512_final (struct hmac_sha512_ctx *ctx,u8 out[SHA512_DIGEST_SIZE ]); + +/** +* hmac_sha512() - Compute HMAC-SHA512 in one shot, using a prepared key +* @key: the prepared HMAC key +* @data: the message data +* @data_len: the data length in bytes +* @out: (output) the resulting HMAC-SHA512 value +* +* If you're using the key only once, consider using hmac_sha512_usingrawkey(). +* +* Context: Any context. +*/ +void hmac_sha512 (const struct hmac_sha512_key *key, +const u8 *data,size_t data_len,u8 out[SHA512_DIGEST_SIZE ]); + +/** +* hmac_sha512_usingrawkey() - Compute HMAC-SHA512 in one shot, using a raw key +* @raw_key: the raw HMAC-SHA512 key +* @raw_key_len: the key length in bytes. All key lengths are supported. +* @data: the message data +* @data_len: the data length in bytes +* @out: (output) the resulting HMAC-SHA512 value +* +* If you're using the key multiple times, prefer to use +* hmac_sha512_preparekey() followed by multiple calls to hmac_sha512() instead. +* +* Context: Any context. +*/ +void hmac_sha512_usingrawkey (const u8 *raw_key,size_t raw_key_len, +const u8 *data,size_t data_len, +u8 out[SHA512_DIGEST_SIZE ]); +#endif /* _CRYPTO_SHA2_H */
lib/crypto/Kconfig +3 -2
@@ -168,12 +168,13 @@ config CRYPTO_LIB_SHA256_GENERIC enabled, this implementation serves the users of CRYPTO_LIB_SHA256. config CRYPTO_LIB_SHA512 tristate help - The SHA-384 and SHA-512 library functions. Select this if your module - uses any of these functions from <crypto/sha2.h>. + The SHA-384, SHA-512, HMAC-SHA384, and HMAC-SHA512 library functions. + Select this if your module uses any of these functions from + <crypto/sha2.h>. config CRYPTO_LIB_SHA512_ARCH bool depends on CRYPTO_LIB_SHA512 && !UML
lib/crypto/sha512.c +139 -2
178 lines not on this page · raw message
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 4/16 crypto: riscv/sha512 - Stop depending on sha512_generic_block_fn
In reply to Eric Biggerssha512_generic_block_fn() will no longer be available when the SHA-512 support in the old-school crypto API is changed to just wrap the SHA-512 library. Replace the use of sha512_generic_block_fn() in sha512-riscv64-glue.c with temporary code that uses the library's __sha512_update(). This is just a temporary workaround to keep the kernel building and functional at each commit; this code gets superseded when the RISC-V optimized SHA-512 is migrated to lib/crypto/ anyway.
--- arch/riscv/crypto/Kconfig | 1 + arch/riscv/crypto/sha512-riscv64-glue.c | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-)
arch/riscv/crypto/Kconfig +1 -0
@@ -29,10 +29,11 @@ config CRYPTO_GHASH_RISCV64 - Zvkg vector crypto extension config CRYPTO_SHA512_RISCV64 tristate "Hash functions: SHA-384 and SHA-512" depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO + select CRYPTO_LIB_SHA512 select CRYPTO_SHA512 help SHA-384 and SHA-512 secure hash algorithm (FIPS 180) Architecture: riscv64 using:
arch/riscv/crypto/sha512-riscv64-glue.c +7 -1
@@ -36,11 +36,17 @@ static void sha512_block(struct sha512_state *state, const u8 *data,if (crypto_simd_usable ()) {kernel_vector_begin ();sha512_transform_zvknhb_zvkb (state, data, num_blocks);kernel_vector_end (); }else { -sha512_generic_block_fn (state, data, num_blocks); +struct __sha512_ctx ctx = {}; + +static_assert (sizeof (ctx.state) ==sizeof (state->state)); +memcpy (&ctx.state, state->state,sizeof (ctx.state)); +__sha512_update (&ctx, data, + (size_t )num_blocks *SHA512_BLOCK_SIZE ); +memcpy (state->state, &ctx.state,sizeof (state->state)); } }static int riscv64_sha512_update (struct shash_desc *desc,const u8 *data,unsigned int len)
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 5/16 crypto: sha512 - Replace sha512_generic with wrapper around SHA-512 library
In reply to Eric BiggersDelete crypto/sha512_generic.c, which provided "generic" SHA-384 and SHA-512 crypto_shash algorithms. Replace it with crypto/sha512.c which provides SHA-384, SHA-512, HMAC-SHA384, and HMAC-SHA512 crypto_shash algorithms using the corresponding library functions.
This is a prerequisite for migrating all the arch-optimized SHA-512 code (which is almost 3000 lines) to lib/crypto/ rather than duplicating it.
Since the replacement crypto_shash algorithms are implemented using the (potentially arch-optimized) library functions, give them cra_driver_names ending with "-lib" rather than "-generic". Update crypto/testmgr.c and one odd driver to take this change in driver name into account. Besides these cases which are accounted for, there are no known cases where the cra_driver_name was being depended on.
This change does mean that the abstract partial block handling code in crypto/shash.c, which got added in 6.16, no longer gets used. But that's fine; the library has to implement the partial block handling anyway, and it's better to do it in the library since the block size and other properties of the algorithm are all fixed at compile time there, resulting in more streamlined code.
--- crypto/Kconfig | 4 +- crypto/Makefile | 2 +- crypto/sha512.c | 258 ++++++++++++++++++++++++++ crypto/sha512_generic.c | 217 ---------------------- crypto/testmgr.c | 10 + drivers/crypto/starfive/jh7110-hash.c | 8 +- include/crypto/sha512_base.h | 3 - 7 files changed, 276 insertions(+), 226 deletions(-) create mode 100644 crypto/sha512.c delete mode 100644 crypto/sha512_generic.c
crypto/Kconfig +3 -1
@@ -1000,12 +1000,14 @@ config CRYPTO_SHA256 Used by the btrfs filesystem, Ceph, NFS, and SMB. config CRYPTO_SHA512 tristate "SHA-384 and SHA-512" select CRYPTO_HASH + select CRYPTO_LIB_SHA512 help - SHA-384 and SHA-512 secure hash algorithms (FIPS 180, ISO/IEC 10118-3) + SHA-384 and SHA-512 secure hash algorithms (FIPS 180, ISO/IEC + 10118-3), including HMAC support. config CRYPTO_SHA3 tristate "SHA-3" select CRYPTO_HASH help
crypto/Makefile +1 -1
@@ -76,11 +76,11 @@ obj-$(CONFIG_CRYPTO_MD4) += md4.o obj-$(CONFIG_CRYPTO_MD5) += md5.o obj-$(CONFIG_CRYPTO_RMD160) += rmd160.o obj-$(CONFIG_CRYPTO_SHA1) += sha1_generic.o obj-$(CONFIG_CRYPTO_SHA256) += sha256.o CFLAGS_sha256.o += -DARCH=$(ARCH) -obj-$(CONFIG_CRYPTO_SHA512) += sha512_generic.o +obj-$(CONFIG_CRYPTO_SHA512) += sha512.o obj-$(CONFIG_CRYPTO_SHA3) += sha3_generic.o obj-$(CONFIG_CRYPTO_SM3_GENERIC) += sm3_generic.o obj-$(CONFIG_CRYPTO_STREEBOG) += streebog_generic.o obj-$(CONFIG_CRYPTO_WP512) += wp512.o CFLAGS_wp512.o := $(call cc-option,-fno-schedule-insns) # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=79149
crypto/sha512.c new file +258 -0
258 lines not on this page · raw message
crypto/sha512_generic.c deleted +0 -217
217 lines not on this page · raw message
crypto/testmgr.c +10 -0
@@ -4313,49 +4313,55 @@ static const struct alg_test_desc alg_test_descs[] = { .alg = "authenc(hmac(sha256),rfc3686(ctr(aes)))", .test = alg_test_null, .fips_allowed = 1, }, { .alg = "authenc(hmac(sha384),cbc(des))", + .generic_driver = "authenc(hmac-sha384-lib,cbc(des-generic))", .test = alg_test_aead, .suite = { .aead = __VECS(hmac_sha384_des_cbc_tv_temp) } }, { .alg = "authenc(hmac(sha384),cbc(des3_ede))", + .generic_driver = "authenc(hmac-sha384-lib,cbc(des3_ede-generic))", .test = alg_test_aead, .suite = { .aead = __VECS(hmac_sha384_des3_ede_cbc_tv_temp) } }, { .alg = "authenc(hmac(sha384),ctr(aes))", .test = alg_test_null, .fips_allowed = 1, }, { .alg = "authenc(hmac(sha384),cts(cbc(aes)))", + .generic_driver = "authenc(hmac-sha384-lib,cts(cbc(aes-generic)))", .test = alg_test_aead, .suite = { .aead = __VECS(krb5_test_aes256_cts_hmac_sha384_192) } }, { .alg = "authenc(hmac(sha384),rfc3686(ctr(aes)))", .test = alg_test_null, .fips_allowed = 1, }, { .alg = "authenc(hmac(sha512),cbc(aes))", + .generic_driver = "authenc(hmac-sha512-lib,cbc(aes-generic))", .fips_allowed = 1, .test = alg_test_aead, .suite = { .aead = __VECS(hmac_sha512_aes_cbc_tv_temp) } }, { .alg = "authenc(hmac(sha512),cbc(des))", + .generic_driver = "authenc(hmac-sha512-lib,cbc(des-generic))", .test = alg_test_aead, .suite = { .aead = __VECS(hmac_sha512_des_cbc_tv_temp) } }, { .alg = "authenc(hmac(sha512),cbc(des3_ede))", + .generic_driver = "authenc(hmac-sha512-lib,cbc(des3_ede-generic))", .test = alg_test_aead, .suite = { .aead = __VECS(hmac_sha512_des3_ede_cbc_tv_temp) } }, {
@@ -5155,17 +5161,19 @@ static const struct alg_test_desc alg_test_descs[] = { .suite = { .hash = __VECS(hmac_sha3_512_tv_template) } }, { .alg = "hmac(sha384)", + .generic_driver = "hmac-sha384-lib", .test = alg_test_hash, .fips_allowed = 1, .suite = { .hash = __VECS(hmac_sha384_tv_template) } }, { .alg = "hmac(sha512)", + .generic_driver = "hmac-sha512-lib", .test = alg_test_hash, .fips_allowed = 1, .suite = { .hash =__VECS (hmac_sha512_tv_template) }
@@ -5491,17 +5499,19 @@ static const struct alg_test_desc alg_test_descs[] = { .suite = { .hash = __VECS(sha3_512_tv_template) } }, { .alg = "sha384", + .generic_driver = "sha384-lib", .test = alg_test_hash, .fips_allowed = 1, .suite = { .hash = __VECS(sha384_tv_template) } }, { .alg = "sha512", + .generic_driver = "sha512-lib", .test = alg_test_hash, .fips_allowed = 1, .suite = { .hash =__VECS (sha512_tv_template) }
drivers/crypto/starfive/jh7110-hash.c +4 -4
@@ -503,17 +503,17 @@ static int starfive_sha256_init_tfm(struct crypto_ahash *hash) STARFIVE_HASH_SHA256, 0); }static int starfive_sha384_init_tfm (struct crypto_ahash *hash) { -return starfive_hash_init_tfm (hash,"sha384-generic" , +return starfive_hash_init_tfm (hash,"sha384-lib" ,STARFIVE_HASH_SHA384 ,0 ); }static int starfive_sha512_init_tfm (struct crypto_ahash *hash) { -return starfive_hash_init_tfm (hash,"sha512-generic" , +return starfive_hash_init_tfm (hash,"sha512-lib" ,STARFIVE_HASH_SHA512 ,0 ); }static int starfive_sm3_init_tfm (struct crypto_ahash *hash) {
@@ -533,17 +533,17 @@ static int starfive_hmac_sha256_init_tfm(struct crypto_ahash *hash) STARFIVE_HASH_SHA256, 1); }static int starfive_hmac_sha384_init_tfm (struct crypto_ahash *hash) { -return starfive_hash_init_tfm (hash,"hmac(sha384-generic)" , +return starfive_hash_init_tfm (hash,"hmac-sha384-lib" ,STARFIVE_HASH_SHA384 ,1 ); }static int starfive_hmac_sha512_init_tfm (struct crypto_ahash *hash) { -return starfive_hash_init_tfm (hash,"hmac(sha512-generic)" , +return starfive_hash_init_tfm (hash,"hmac-sha512-lib" ,STARFIVE_HASH_SHA512 ,1 ); }static int starfive_hmac_sm3_init_tfm (struct crypto_ahash *hash) {
include/crypto/sha512_base.h +0 -3
@@ -112,9 +112,6 @@ static inline int sha512_base_finish(struct shash_desc *desc, u8 *out)for (i =0 ; digest_size >0 ; i++, digest_size -=sizeof (__be64))put_unaligned_be64 (sctx->state[i], digest++); return 0; } -void sha512_generic_block_fn (struct sha512_state *sst,u8 const *src, -int blocks); -#endif /* _CRYPTO_SHA512_BASE_H */
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 6/16 crypto: sha512 - Use same state format as legacy drivers
In reply to Eric BiggersMake the export and import functions for the sha384, sha512, hmac(sha384), and hmac(sha512) shash algorithms use the same format as the padlock-sha and nx-sha512 drivers, as required by Herbert.
--- crypto/sha512.c | 96 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 96 insertions(+)
crypto/sha512.c +96 -0
@@ -10,10 +10,48 @@#include <crypto/internal/hash.h> #include <crypto/sha2.h> #include <linux/kernel.h> #include <linux/module.h> +/* +* Export and import functions. crypto_shash wants a particular format that +* matches that used by some legacy drivers. It currently is the same as the +* library SHA context, except the value in bytecount_lo must be block-aligned +* and the remainder must be stored in an extra u8 appended to the struct. +*/ + +#define SHA512_SHASH_STATE_SIZE 209 +static_assert (offsetof(struct __sha512_ctx , state) ==0 ); +static_assert (offsetof(struct __sha512_ctx , bytecount_lo) ==64 ); +static_assert (offsetof(struct __sha512_ctx , bytecount_hi) ==72 ); +static_assert (offsetof(struct __sha512_ctx , buf) ==80 ); +static_assert (sizeof (struct __sha512_ctx ) +1 ==SHA512_SHASH_STATE_SIZE ); + +static int __crypto_sha512_export (const struct __sha512_ctx *ctx0,void *out) +{ +struct __sha512_ctx ctx = *ctx0; +unsigned int partial; +u8 *p = out; + + partial = ctx.bytecount_lo %SHA512_BLOCK_SIZE ; + ctx.bytecount_lo -= partial; +memcpy (p, &ctx,sizeof (ctx)); + p +=sizeof (ctx); + *p = partial; +return 0 ; +} + +static int __crypto_sha512_import (struct __sha512_ctx *ctx,const void *in) +{ +const u8 *p = in; + +memcpy (ctx, p,sizeof (*ctx)); + p +=sizeof (*ctx); + ctx->bytecount_lo += *p; +return 0 ; +} +/* SHA-384 */ const u8 sha384_zero_message_hash[SHA384_DIGEST_SIZE ] = {0x38 ,0xb0 ,0x60 ,0xa7 ,0x51 ,0xac ,0x96 ,0x38 ,0x4c ,0xd9 ,0x32 ,0x7e ,0xb1 ,0xb1 ,0xe3 ,0x6a ,
@@ -50,10 +88,20 @@ static int crypto_sha384_digest(struct shash_desc *desc, {sha384 (data, len, out);return 0 ; } +static int crypto_sha384_export (struct shash_desc *desc,void *out) +{ +return __crypto_sha512_export (&SHA384_CTX (desc)->ctx, out); +} + +static int crypto_sha384_import (struct shash_desc *desc,const void *in) +{ +return __crypto_sha512_import (&SHA384_CTX (desc)->ctx, in); +} +/* SHA-512 */ const u8 sha512_zero_message_hash[SHA512_DIGEST_SIZE ] = {0xcf ,0x83 ,0xe1 ,0x35 ,0x7e ,0xef ,0xb8 ,0xbd ,0xf1 ,0x54 ,0x28 ,0x50 ,0xd6 ,0x6d ,0x80 ,0x07 ,
@@ -92,10 +140,20 @@ static int crypto_sha512_digest(struct shash_desc *desc, {sha512 (data, len, out);return 0 ; } +static int crypto_sha512_export (struct shash_desc *desc,void *out) +{ +return __crypto_sha512_export (&SHA512_CTX (desc)->ctx, out); +} + +static int crypto_sha512_import (struct shash_desc *desc,const void *in) +{ +return __crypto_sha512_import (&SHA512_CTX (desc)->ctx, in); +} +/* HMAC-SHA384 */ #define HMAC_SHA384_KEY (tfm) ((struct hmac_sha384_key *)crypto_shash_ctx(tfm))#define HMAC_SHA384_CTX (desc) ((struct hmac_sha384_ctx *)shash_desc_ctx(desc))
@@ -131,10 +189,23 @@ static int crypto_hmac_sha384_digest(struct shash_desc *desc, {hmac_sha384 (HMAC_SHA384_KEY (desc->tfm), data, len, out);return 0 ; } +static int crypto_hmac_sha384_export (struct shash_desc *desc,void *out) +{ +return __crypto_sha512_export (&HMAC_SHA384_CTX (desc)->ctx.sha_ctx, out); +} + +static int crypto_hmac_sha384_import (struct shash_desc *desc,const void *in) +{ +struct hmac_sha384_ctx *ctx =HMAC_SHA384_CTX (desc); + + ctx->ctx.ostate =HMAC_SHA384_KEY (desc->tfm)->key.ostate; +return __crypto_sha512_import (&ctx->ctx.sha_ctx, in); +} +/* HMAC-SHA512 */ #define HMAC_SHA512_KEY (tfm) ((struct hmac_sha512_key *)crypto_shash_ctx(tfm))#define HMAC_SHA512_CTX (desc) ((struct hmac_sha512_ctx *)shash_desc_ctx(desc))
@@ -170,10 +241,23 @@ static int crypto_hmac_sha512_digest(struct shash_desc *desc, {hmac_sha512 (HMAC_SHA512_KEY (desc->tfm), data, len, out);return 0 ; } +static int crypto_hmac_sha512_export (struct shash_desc *desc,void *out) +{ +return __crypto_sha512_export (&HMAC_SHA512_CTX (desc)->ctx.sha_ctx, out); +} + +static int crypto_hmac_sha512_import (struct shash_desc *desc,const void *in) +{ +struct hmac_sha512_ctx *ctx =HMAC_SHA512_CTX (desc); + + ctx->ctx.ostate =HMAC_SHA512_KEY (desc->tfm)->key.ostate; +return __crypto_sha512_import (&ctx->ctx.sha_ctx, in); +} +/* Algorithm definitions */ static struct shash_alg algs[] = { { .base.cra_name ="sha384" ,
@@ -184,11 +268,14 @@ static struct shash_alg algs[] = { .digestsize = SHA384_DIGEST_SIZE, .init = crypto_sha384_init, .update = crypto_sha384_update, .final = crypto_sha384_final, .digest = crypto_sha384_digest, + .export = crypto_sha384_export, + .import = crypto_sha384_import, .descsize = sizeof(struct sha384_ctx), + .statesize = SHA512_SHASH_STATE_SIZE, }, { .base.cra_name = "sha512", .base.cra_driver_name = "sha512-lib", .base.cra_priority = 300,
@@ -197,11 +284,14 @@ static struct shash_alg algs[] = { .digestsize = SHA512_DIGEST_SIZE, .init = crypto_sha512_init, .update = crypto_sha512_update, .final = crypto_sha512_final, .digest = crypto_sha512_digest, + .export = crypto_sha512_export, + .import = crypto_sha512_import, .descsize = sizeof(struct sha512_ctx), + .statesize = SHA512_SHASH_STATE_SIZE, }, { .base.cra_name = "hmac(sha384)", .base.cra_driver_name = "hmac-sha384-lib", .base.cra_priority = 300,
@@ -212,11 +302,14 @@ static struct shash_alg algs[] = { .setkey = crypto_hmac_sha384_setkey, .init = crypto_hmac_sha384_init, .update = crypto_hmac_sha384_update, .final = crypto_hmac_sha384_final, .digest = crypto_hmac_sha384_digest, + .export = crypto_hmac_sha384_export, + .import = crypto_hmac_sha384_import, .descsize = sizeof(struct hmac_sha384_ctx), + .statesize = SHA512_SHASH_STATE_SIZE, }, { .base.cra_name = "hmac(sha512)", .base.cra_driver_name = "hmac-sha512-lib", .base.cra_priority = 300,
@@ -227,11 +320,14 @@ static struct shash_alg algs[] = { .setkey = crypto_hmac_sha512_setkey, .init = crypto_hmac_sha512_init, .update = crypto_hmac_sha512_update, .final = crypto_hmac_sha512_final, .digest = crypto_hmac_sha512_digest, + .export = crypto_hmac_sha512_export, + .import = crypto_hmac_sha512_import, .descsize = sizeof(struct hmac_sha512_ctx), + .statesize = SHA512_SHASH_STATE_SIZE, }, };static int __init crypto_sha512_mod_init (void ) {
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 7/16 lib/crypto: arm/sha512: Migrate optimized SHA-512 code to library
In reply to Eric BiggersInstead of exposing the arm-optimized SHA-512 code via arm-specific crypto_shash algorithms, instead just implement the sha512_blocks() library function. This is much simpler, it makes the SHA-512 (and SHA-384) library functions be arm-optimized, and it fixes the longstanding issue where the arm-optimized SHA-512 code was disabled by default. SHA-512 still remains available through crypto_shash, but individual architectures no longer need to handle it.
To match sha512_blocks(), change the type of the nblocks parameter of the assembly functions from int to size_t. The assembly functions actually already treated it as size_t.
---
arch/arm/configs/exynos_defconfig | 1 -
arch/arm/configs/milbeaut_m10v_defconfig | 1 -
arch/arm/configs/multi_v7_defconfig | 1 -
arch/arm/configs/omap2plus_defconfig | 1 -
arch/arm/configs/pxa_defconfig | 1 -
arch/arm/crypto/Kconfig | 10 --
arch/arm/crypto/Makefile | 15 ---
arch/arm/crypto/sha512-glue.c | 110 ------------------
arch/arm/crypto/sha512-neon-glue.c | 75 ------------
arch/arm/crypto/sha512.h | 3 -
lib/crypto/Kconfig | 1 +
lib/crypto/Makefile | 14 +++
lib/crypto/arm/.gitignore | 2 +
.../crypto => lib/crypto/arm}/sha512-armv4.pl | 0
lib/crypto/arm/sha512.h | 38 ++++++
15 files changed, 55 insertions(+), 218 deletions(-)
delete mode 100644 arch/arm/crypto/sha512-glue.c
delete mode 100644 arch/arm/crypto/sha512-neon-glue.c
delete mode 100644 arch/arm/crypto/sha512.h
create mode 100644 lib/crypto/arm/.gitignore
rename {arch/arm/crypto => lib/crypto/arm}/sha512-armv4.pl (100%)
create mode 100644 lib/crypto/arm/sha512.harch/arm/configs/exynos_defconfig +0 -1
@@ -362,11 +362,10 @@ CONFIG_CRYPTO_LZ4=m CONFIG_CRYPTO_USER_API_HASH=m CONFIG_CRYPTO_USER_API_SKCIPHER=m CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_USER_API_AEAD=m CONFIG_CRYPTO_SHA1_ARM_NEON=m -CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRYPTO_DEV_EXYNOS_RNG=y CONFIG_CRYPTO_DEV_S5P=y CONFIG_DMA_CMA=y
arch/arm/configs/milbeaut_m10v_defconfig +0 -1
@@ -98,11 +98,10 @@ CONFIG_CRYPTO_SELFTESTS=y CONFIG_CRYPTO_AES=y CONFIG_CRYPTO_SEQIV=m CONFIG_CRYPTO_GHASH_ARM_CE=m CONFIG_CRYPTO_SHA1_ARM_NEON=m CONFIG_CRYPTO_SHA1_ARM_CE=m -CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_AES_ARM_CE=m CONFIG_CRYPTO_CHACHA20_NEON=m # CONFIG_CRYPTO_HW is not set
arch/arm/configs/multi_v7_defconfig +0 -1
@@ -1280,11 +1280,10 @@ CONFIG_CRYPTO_USER_API_SKCIPHER=m CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_USER_API_AEAD=m CONFIG_CRYPTO_GHASH_ARM_CE=m CONFIG_CRYPTO_SHA1_ARM_NEON=m CONFIG_CRYPTO_SHA1_ARM_CE=m -CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_AES_ARM_CE=m CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRYPTO_DEV_SUN4I_SS=m
arch/arm/configs/omap2plus_defconfig +0 -1
@@ -703,11 +703,10 @@ CONFIG_NLS_CODEPAGE_437=y CONFIG_NLS_ISO8859_1=y CONFIG_SECURITY=y CONFIG_CRYPTO_MICHAEL_MIC=y CONFIG_CRYPTO_GHASH_ARM_CE=m CONFIG_CRYPTO_SHA1_ARM_NEON=m -CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRYPTO_DEV_OMAP=m CONFIG_CRYPTO_DEV_OMAP_SHAM=m
arch/arm/configs/pxa_defconfig +0 -1
@@ -657,11 +657,10 @@ CONFIG_CRYPTO_WP512=m CONFIG_CRYPTO_ANUBIS=m CONFIG_CRYPTO_XCBC=m CONFIG_CRYPTO_DEFLATE=y CONFIG_CRYPTO_LZO=y CONFIG_CRYPTO_SHA1_ARM=m -CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM=m CONFIG_FONTS=y CONFIG_FONT_8x8=y CONFIG_FONT_8x16=y CONFIG_FONT_6x11=y
arch/arm/crypto/Kconfig +0 -10
@@ -91,20 +91,10 @@ config CRYPTO_SHA1_ARM_CE help SHA-1 secure hash algorithm (FIPS 180) Architecture: arm using ARMv8 Crypto Extensions -config CRYPTO_SHA512_ARM - tristate "Hash functions: SHA-384 and SHA-512 (NEON)" - select CRYPTO_HASH - depends on !CPU_V7M - help - SHA-384 and SHA-512 secure hash algorithms (FIPS 180) - - Architecture: arm using - - NEON (Advanced SIMD) extensions - config CRYPTO_AES_ARM tristate "Ciphers: AES" select CRYPTO_ALGAPI select CRYPTO_AES help
arch/arm/crypto/Makefile +0 -15
@@ -5,11 +5,10 @@ obj-$(CONFIG_CRYPTO_AES_ARM) += aes-arm.o obj-$(CONFIG_CRYPTO_AES_ARM_BS) += aes-arm-bs.o obj-$(CONFIG_CRYPTO_SHA1_ARM) += sha1-arm.o obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o -obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o obj-$(CONFIG_CRYPTO_BLAKE2B_NEON) += blake2b-neon.o obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) += nhpoly1305-neon.o obj-$(CONFIG_CRYPTO_CURVE25519_NEON) += curve25519-neon.o obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o
@@ -18,25 +17,11 @@ obj-$(CONFIG_CRYPTO_GHASH_ARM_CE) += ghash-arm-ce.o aes-arm-y := aes-cipher-core.o aes-cipher-glue.o aes-arm-bs-y := aes-neonbs-core.o aes-neonbs-glue.o sha1-arm-y := sha1-armv4-large.o sha1_glue.o sha1-arm-neon-y := sha1-armv7-neon.o sha1_neon_glue.o -sha512-arm-neon-$(CONFIG_KERNEL_MODE_NEON) := sha512-neon-glue.o -sha512-arm-y := sha512-core.o sha512-glue.o $(sha512-arm-neon-y) blake2b-neon-y := blake2b-neon-core.o blake2b-neon-glue.o sha1-arm-ce-y := sha1-ce-core.o sha1-ce-glue.o aes-arm-ce-y := aes-ce-core.o aes-ce-glue.o ghash-arm-ce-y := ghash-ce-core.o ghash-ce-glue.o nhpoly1305-neon-y := nh-neon-core.o nhpoly1305-neon-glue.o curve25519-neon-y := curve25519-core.o curve25519-glue.o - -quiet_cmd_perl = PERL $@ - cmd_perl = $(PERL) $(<) > $(@) - -$(obj)/%-core.S: $(src)/%-armv4.pl - $(call cmd,perl) - -clean-files += sha512-core.S - -aflags-thumb2-$(CONFIG_THUMB2_KERNEL) := -U__thumb2__ -D__thumb2__=1 - -AFLAGS_sha512-core.o += $(aflags-thumb2-y)
arch/arm/crypto/sha512-glue.c deleted +0 -110
110 lines not on this page · raw message
arch/arm/crypto/sha512-neon-glue.c deleted +0 -75
75 lines not on this page · raw message
arch/arm/crypto/sha512.h deleted +0 -3
@@ -1,3 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ - -extern struct shash_alg sha512_neon_algs[2 ];
lib/crypto/Kconfig +1 -0
@@ -175,10 +175,11 @@ config CRYPTO_LIB_SHA512 <crypto/sha2.h>. config CRYPTO_LIB_SHA512_ARCH bool depends on CRYPTO_LIB_SHA512 && !UML + default y if ARM && !CPU_V7M config CRYPTO_LIB_SM3 tristate if !KMSAN # avoid false positives from assembly
lib/crypto/Makefile +14 -0
@@ -1,7 +1,12 @@ # SPDX-License-Identifier: GPL-2.0 +aflags-thumb2-$(CONFIG_THUMB2_KERNEL) := -U__thumb2__ -D__thumb2__=1 + +quiet_cmd_perlasm = PERLASM $@ + cmd_perlasm = $(PERL) $(<) > $(@) + obj-$(CONFIG_CRYPTO_LIB_UTILS) += libcryptoutils.o libcryptoutils-y := memneq.o utils.o # chacha is used by the /dev/random driver which is always builtin obj-y += chacha.o
@@ -66,10 +71,19 @@ libsha256-generic-y := sha256-generic.o obj-$(CONFIG_CRYPTO_LIB_SHA512) += libsha512.o libsha512-y := sha512.o ifeq ($(CONFIG_CRYPTO_LIB_SHA512_ARCH),y) CFLAGS_sha512.o += -I$(src)/$(SRCARCH) + +ifeq ($(CONFIG_ARM),y) +libsha512-y += arm/sha512-core.o +$(obj)/arm/sha512-core.S: $(src)/arm/sha512-armv4.pl + $(call cmd,perlasm) +clean-files += arm/sha512-core.S +AFLAGS_arm/sha512-core.o += $(aflags-thumb2-y) +endif + endif # CONFIG_CRYPTO_LIB_SHA512_ARCH obj-$(CONFIG_MPILIB) += mpi/ obj-$(CONFIG_CRYPTO_SELFTESTS_FULL) += simd.o
lib/crypto/arm/.gitignore new file +2 -0
@@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +sha512-core.S
lib/crypto/arm/sha512-armv4.pl renamed from arch/arm/crypto/sha512-armv4.pl +0 -0
lib/crypto/arm/sha512.h new file +38 -0
@@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* arm32-optimized SHA-512 block function +* +* Copyright 2025 Google LLC +*/ + +#include <asm/neon.h> +#include <crypto/internal/simd.h> + +static __ro_after_init DEFINE_STATIC_KEY_FALSE (have_neon); + +asmlinkage void sha512_block_data_order(struct sha512_block_state *state, +const u8 *data,size_t nblocks); +asmlinkage void sha512_block_data_order_neon(struct sha512_block_state *state, +const u8 *data,size_t nblocks); + +static void sha512_blocks (struct sha512_block_state *state, +const u8 *data,size_t nblocks) +{ +if (IS_ENABLED (CONFIG_KERNEL_MODE_NEON ) && +static_branch_likely (&have_neon) &&likely (crypto_simd_usable ())) { +kernel_neon_begin (); +sha512_block_data_order_neon (state, data, nblocks); +kernel_neon_end (); + }else { +sha512_block_data_order (state, data, nblocks); + } +} + +#ifdef CONFIG_KERNEL_MODE_NEON +#define sha512_mod_init_arch sha512_mod_init_arch +static inline void sha512_mod_init_arch (void ) +{ +if (cpu_has_neon ()) +static_branch_enable (&have_neon); +} +#endif /* CONFIG_KERNEL_MODE_NEON */
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 8/16 lib/crypto: arm64/sha512: Migrate optimized SHA-512 code to library
In reply to Eric BiggersInstead of exposing the arm64-optimized SHA-512 code via arm64-specific crypto_shash algorithms, instead just implement the sha512_blocks() library function. This is much simpler, it makes the SHA-512 (and SHA-384) library functions be arm64-optimized, and it fixes the longstanding issue where the arm64-optimized SHA-512 code was disabled by default. SHA-512 still remains available through crypto_shash, but individual architectures no longer need to handle it.
To match sha512_blocks(), change the type of the nblocks parameter of the assembly functions from int or 'unsigned int' to size_t. Update the ARMv8 CE assembly function accordingly. The scalar assembly function actually already treated it as size_t.
---
arch/arm64/configs/defconfig | 1 -
arch/arm64/crypto/Kconfig | 19 ----
arch/arm64/crypto/Makefile | 14 ---
arch/arm64/crypto/sha512-ce-glue.c | 96 -------------------
arch/arm64/crypto/sha512-glue.c | 83 ----------------
lib/crypto/Kconfig | 1 +
lib/crypto/Makefile | 10 ++
lib/crypto/arm64/.gitignore | 2 +
.../crypto/arm64}/sha512-ce-core.S | 10 +-
lib/crypto/arm64/sha512.h | 46 +++++++++
10 files changed, 64 insertions(+), 218 deletions(-)
delete mode 100644 arch/arm64/crypto/sha512-ce-glue.c
delete mode 100644 arch/arm64/crypto/sha512-glue.c
create mode 100644 lib/crypto/arm64/.gitignore
rename {arch/arm64/crypto => lib/crypto/arm64}/sha512-ce-core.S (97%)
create mode 100644 lib/crypto/arm64/sha512.harch/arm64/configs/defconfig +0 -1
@@ -1742,11 +1742,10 @@ CONFIG_CRYPTO_ECHAINIV=y CONFIG_CRYPTO_MICHAEL_MIC=m CONFIG_CRYPTO_ANSI_CPRNG=y CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_GHASH_ARM64_CE=y CONFIG_CRYPTO_SHA1_ARM64_CE=y -CONFIG_CRYPTO_SHA512_ARM64_CE=m CONFIG_CRYPTO_SHA3_ARM64=m CONFIG_CRYPTO_SM3_ARM64_CE=m CONFIG_CRYPTO_AES_ARM64_CE_BLK=y CONFIG_CRYPTO_AES_ARM64_BS=m CONFIG_CRYPTO_AES_ARM64_CE_CCM=y
arch/arm64/crypto/Kconfig +0 -19
@@ -34,29 +34,10 @@ config CRYPTO_SHA1_ARM64_CE SHA-1 secure hash algorithm (FIPS 180) Architecture: arm64 using: - ARMv8 Crypto Extensions -config CRYPTO_SHA512_ARM64 - tristate "Hash functions: SHA-384 and SHA-512" - select CRYPTO_HASH - help - SHA-384 and SHA-512 secure hash algorithms (FIPS 180) - - Architecture: arm64 - -config CRYPTO_SHA512_ARM64_CE - tristate "Hash functions: SHA-384 and SHA-512 (ARMv8 Crypto Extensions)" - depends on KERNEL_MODE_NEON - select CRYPTO_HASH - select CRYPTO_SHA512_ARM64 - help - SHA-384 and SHA-512 secure hash algorithms (FIPS 180) - - Architecture: arm64 using: - - ARMv8 Crypto Extensions - config CRYPTO_SHA3_ARM64 tristate "Hash functions: SHA-3 (ARMv8.2 Crypto Extensions)" depends on KERNEL_MODE_NEON select CRYPTO_HASH select CRYPTO_SHA3
arch/arm64/crypto/Makefile +0 -14
@@ -6,13 +6,10 @@ # obj-$(CONFIG_CRYPTO_SHA1_ARM64_CE) += sha1-ce.o sha1-ce-y := sha1-ce-glue.o sha1-ce-core.o -obj-$(CONFIG_CRYPTO_SHA512_ARM64_CE) += sha512-ce.o -sha512-ce-y := sha512-ce-glue.o sha512-ce-core.o - obj-$(CONFIG_CRYPTO_SHA3_ARM64) += sha3-ce.o sha3-ce-y := sha3-ce-glue.o sha3-ce-core.o obj-$(CONFIG_CRYPTO_SM3_NEON) += sm3-neon.o sm3-neon-y := sm3-neon-glue.o sm3-neon-core.o
@@ -51,24 +48,13 @@ obj-$(CONFIG_CRYPTO_AES_ARM64_CE_BLK) += aes-ce-blk.o aes-ce-blk-y := aes-glue-ce.o aes-ce.o obj-$(CONFIG_CRYPTO_AES_ARM64_NEON_BLK) += aes-neon-blk.o aes-neon-blk-y := aes-glue-neon.o aes-neon.o -obj-$(CONFIG_CRYPTO_SHA512_ARM64) += sha512-arm64.o -sha512-arm64-y := sha512-glue.o sha512-core.o - obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) += nhpoly1305-neon.o nhpoly1305-neon-y := nh-neon-core.o nhpoly1305-neon-glue.o obj-$(CONFIG_CRYPTO_AES_ARM64) += aes-arm64.o aes-arm64-y := aes-cipher-core.o aes-cipher-glue.o obj-$(CONFIG_CRYPTO_AES_ARM64_BS) += aes-neon-bs.o aes-neon-bs-y := aes-neonbs-core.o aes-neonbs-glue.o - -quiet_cmd_perlasm = PERLASM $@ - cmd_perlasm = $(PERL) $(<) void $(@) - -$(obj)/sha512-core.S: $(src)/../lib/crypto/sha2-armv8.pl - $(call cmd,perlasm) - -clean-files += sha512-core.S
arch/arm64/crypto/sha512-ce-glue.c deleted +0 -96
96 lines not on this page · raw message
arch/arm64/crypto/sha512-glue.c deleted +0 -83
83 lines not on this page · raw message
lib/crypto/Kconfig +1 -0
@@ -176,10 +176,11 @@ config CRYPTO_LIB_SHA512 config CRYPTO_LIB_SHA512_ARCH bool depends on CRYPTO_LIB_SHA512 && !UML default y if ARM && !CPU_V7M + default y if ARM64 config CRYPTO_LIB_SM3 tristate if !KMSAN # avoid false positives from assembly
lib/crypto/Makefile +10 -0
@@ -3,10 +3,13 @@ aflags-thumb2-$(CONFIG_THUMB2_KERNEL) := -U__thumb2__ -D__thumb2__=1 quiet_cmd_perlasm = PERLASM $@ cmd_perlasm = $(PERL) $(<) > $(@) +quiet_cmd_perlasm_with_args = PERLASM $@ + cmd_perlasm_with_args = $(PERL) $(<) void $(@) + obj-$(CONFIG_CRYPTO_LIB_UTILS) += libcryptoutils.o libcryptoutils-y := memneq.o utils.o # chacha is used by the /dev/random driver which is always builtin obj-y += chacha.o
@@ -80,10 +83,17 @@ $(obj)/arm/sha512-core.S: $(src)/arm/sha512-armv4.pl $(call cmd,perlasm) clean-files += arm/sha512-core.S AFLAGS_arm/sha512-core.o += $(aflags-thumb2-y) endif +ifeq ($(CONFIG_ARM64),y) +libsha512-y += arm64/sha512-core.o +$(obj)/arm64/sha512-core.S: $(src)/../../arch/arm64/lib/crypto/sha2-armv8.pl + $(call cmd,perlasm_with_args) +clean-files += arm64/sha512-core.S +libsha512-$(CONFIG_KERNEL_MODE_NEON) += arm64/sha512-ce-core.o +endif endif # CONFIG_CRYPTO_LIB_SHA512_ARCH obj-$(CONFIG_MPILIB) += mpi/ obj-$(CONFIG_CRYPTO_SELFTESTS_FULL) += simd.o
lib/crypto/arm64/.gitignore new file +2 -0
@@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +sha512-core.S
lib/crypto/arm64/sha512-ce-core.S renamed from arch/arm64/crypto/sha512-ce-core.S +5 -5
@@ -100,12 +100,12 @@ add v\i4\().2d, v\i1\().2d, v\i3\().2d sha512h2 q\i3, q\i1, v\i0\().2d .endm /* - * int __sha512_ce_transform(struct sha512_state *sst, u8 const *src, - * int blocks) + * size_t __sha512_ce_transform(struct sha512_block_state *state, + * const u8 *data, size_t nblocks); */ .text SYM_FUNC_START(__sha512_ce_transform) /* load state */ ld1 {v8.2d-v11.2d}, [x0]
@@ -115,11 +115,11 @@ SYM_FUNC_START(__sha512_ce_transform) ld1 {v20.2d-v23.2d}, [x3], #64 /* load input */ 0: ld1 {v12.2d-v15.2d}, [x1], #64 ld1 {v16.2d-v19.2d}, [x1], #64 - sub w2, w2, #1 + sub x2, x2, #1 CPU_LE( rev64 v12.16b, v12.16b ) CPU_LE( rev64 v13.16b, v13.16b ) CPU_LE( rev64 v14.16b, v14.16b ) CPU_LE( rev64 v15.16b, v15.16b )
@@ -195,12 +195,12 @@ CPU_LE( rev64 v19.16b, v19.16b ) add v10.2d, v10.2d, v2.2d add v11.2d, v11.2d, v3.2d cond_yield 3f, x4, x5 /* handled all input blocks? */ - cbnz w2, 0b + cbnz x2, 0b /* store new state */ 3: st1 {v8.2d-v11.2d}, [x0] - mov w0, w2 + mov x0, x2 ret SYM_FUNC_END(__sha512_ce_transform)
lib/crypto/arm64/sha512.h new file +46 -0
46 lines not on this page · raw message
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 9/16 mips: cavium-octeon: Move octeon-crypto.h into asm directory
In reply to Eric BiggersSince arch/mips/cavium-octeon/crypto/octeon-crypto.h is now needed outside of its directory, move it to arch/mips/include/asm/octeon/crypto.h so that it can be included as <asm/octeon/crypto.h>.
---
arch/mips/cavium-octeon/crypto/octeon-crypto.c | 3 +--
arch/mips/cavium-octeon/crypto/octeon-md5.c | 3 +--
arch/mips/cavium-octeon/crypto/octeon-sha1.c | 3 +--
arch/mips/cavium-octeon/crypto/octeon-sha256.c | 3 +--
arch/mips/cavium-octeon/crypto/octeon-sha512.c | 3 +--
.../crypto/octeon-crypto.h => include/asm/octeon/crypto.h} | 0
6 files changed, 5 insertions(+), 10 deletions(-)
rename arch/mips/{cavium-octeon/crypto/octeon-crypto.h => include/asm/octeon/crypto.h} (100%)arch/mips/cavium-octeon/crypto/octeon-crypto.c +1 -2
@@ -5,16 +5,15 @@* * Copyright (C) 2004-2012 Cavium Networks */ #include <asm/cop2.h> +#include <asm/octeon/crypto.h> #include <linux/export.h> #include <linux/interrupt.h> #include <linux/sched/task_stack.h> -#include "octeon-crypto.h" -/** * Enable access to Octeon's COP2 crypto hardware for kernel use. Wrap any * crypto operations in calls to octeon_crypto_enable/disable in order to make * sure the state of COP2 isn't corrupted if userspace is also performing * hardware crypto operations. Allocate the state parameter on the stack.
arch/mips/cavium-octeon/crypto/octeon-md5.c +1 -2
@@ -17,20 +17,19 @@* under the terms of the GNU General Public License as published by the Free * Software Foundation; either version 2 of the License, or (at your option) * any later version. */ +#include <asm/octeon/crypto.h> #include <asm/octeon/octeon.h> #include <crypto/internal/hash.h> #include <crypto/md5.h> #include <linux/kernel.h> #include <linux/module.h> #include <linux/string.h> #include <linux/unaligned.h> -#include "octeon-crypto.h" -struct octeon_md5_state {__le32 hash[MD5_HASH_WORDS ];u64 byte_count; };
arch/mips/cavium-octeon/crypto/octeon-sha1.c +1 -2
@@ -11,20 +11,19 @@* Copyright (c) Alan Smithee. * Copyright (c) Andrew McDonald <[email protected]> * Copyright (c) Jean-Francois Dive <[email protected]> */ +#include <asm/octeon/crypto.h> #include <asm/octeon/octeon.h> #include <crypto/internal/hash.h> #include <crypto/sha1.h> #include <crypto/sha1_base.h> #include <linux/errno.h> #include <linux/kernel.h> #include <linux/module.h> -#include "octeon-crypto.h" -/* * We pass everything as 64-bit. OCTEON can handle misaligned data. */ static void octeon_sha1_store_hash (struct sha1_state *sctx)
arch/mips/cavium-octeon/crypto/octeon-sha256.c +1 -2
@@ -10,17 +10,16 @@* Copyright (c) Andrew McDonald <[email protected]> * Copyright (c) 2002 James Morris <[email protected]> * SHA224 Support Copyright 2007 Intel Corporation <[email protected]> */ +#include <asm/octeon/crypto.h> #include <asm/octeon/octeon.h> #include <crypto/internal/sha2.h> #include <linux/kernel.h> #include <linux/module.h> -#include "octeon-crypto.h" -/* * We pass everything as 64-bit. OCTEON can handle misaligned data. */ void sha256_blocks_arch(u32 state[SHA256_STATE_WORDS],
arch/mips/cavium-octeon/crypto/octeon-sha512.c +1 -2
@@ -11,19 +11,18 @@* Copyright (c) Jean-Luc Cooke <[email protected]> * Copyright (c) Andrew McDonald <[email protected]> * Copyright (c) 2003 Kyle McMartin <[email protected]> */ +#include <asm/octeon/crypto.h> #include <asm/octeon/octeon.h> #include <crypto/internal/hash.h> #include <crypto/sha2.h> #include <crypto/sha512_base.h> #include <linux/kernel.h> #include <linux/module.h> -#include "octeon-crypto.h" -/* * We pass everything as 64-bit. OCTEON can handle misaligned data. */ static void octeon_sha512_store_hash (struct sha512_state *sctx)
arch/mips/include/asm/octeon/crypto.h renamed from arch/mips/cavium-octeon/crypto/octeon-crypto.h +0 -0
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 10/16 lib/crypto: mips/sha512: Migrate optimized SHA-512 code to library
In reply to Eric BiggersInstead of exposing the mips-optimized SHA-512 code via mips-specific crypto_shash algorithms, instead just implement the sha512_blocks() library function. This is much simpler, it makes the SHA-512 (and SHA-384) library functions be mips-optimized, and it fixes the longstanding issue where the mips-optimized SHA-512 code was disabled by default. SHA-512 still remains available through crypto_shash, but individual architectures no longer need to handle it.
Note: to see the diff from arch/mips/cavium-octeon/crypto/octeon-sha512.c to lib/crypto/mips/sha512.h, view this commit with 'git show -M10'.
--- arch/mips/cavium-octeon/crypto/Makefile | 1 - .../mips/cavium-octeon/crypto/octeon-sha512.c | 166 ------------------ arch/mips/configs/cavium_octeon_defconfig | 1 - arch/mips/crypto/Kconfig | 10 -- lib/crypto/Kconfig | 1 + lib/crypto/mips/sha512.h | 74 ++++++++ 6 files changed, 75 insertions(+), 178 deletions(-) delete mode 100644 arch/mips/cavium-octeon/crypto/octeon-sha512.c create mode 100644 lib/crypto/mips/sha512.h
arch/mips/cavium-octeon/crypto/Makefile +0 -1
@@ -6,6 +6,5 @@ obj-y += octeon-crypto.o obj-$(CONFIG_CRYPTO_MD5_OCTEON) += octeon-md5.o obj-$(CONFIG_CRYPTO_SHA1_OCTEON) += octeon-sha1.o obj-$(CONFIG_CRYPTO_SHA256_OCTEON) += octeon-sha256.o -obj-$(CONFIG_CRYPTO_SHA512_OCTEON) += octeon-sha512.o
arch/mips/cavium-octeon/crypto/octeon-sha512.c deleted +0 -166
166 lines not on this page · raw message
arch/mips/configs/cavium_octeon_defconfig +0 -1
@@ -155,11 +155,10 @@ CONFIG_SECURITY=y CONFIG_SECURITY_NETWORK=y CONFIG_CRYPTO_CBC=y CONFIG_CRYPTO_HMAC=y CONFIG_CRYPTO_MD5_OCTEON=y CONFIG_CRYPTO_SHA1_OCTEON=m -CONFIG_CRYPTO_SHA512_OCTEON=m CONFIG_CRYPTO_DES=y CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y CONFIG_DEBUG_FS=y CONFIG_MAGIC_SYSRQ=y # CONFIG_SCHED_DEBUG is not set
arch/mips/crypto/Kconfig +0 -10
@@ -20,16 +20,6 @@ config CRYPTO_SHA1_OCTEON help SHA-1 secure hash algorithm (FIPS 180) Architecture: mips OCTEON -config CRYPTO_SHA512_OCTEON - tristate "Hash functions: SHA-384 and SHA-512 (OCTEON)" - depends on CPU_CAVIUM_OCTEON - select CRYPTO_SHA512 - select CRYPTO_HASH - help - SHA-384 and SHA-512 secure hash algorithms (FIPS 180) - - Architecture: mips OCTEON using crypto instructions, when available - endmenu
lib/crypto/Kconfig +1 -0
@@ -177,10 +177,11 @@ config CRYPTO_LIB_SHA512 config CRYPTO_LIB_SHA512_ARCH bool depends on CRYPTO_LIB_SHA512 && !UML default y if ARM && !CPU_V7M default y if ARM64 + default y if MIPS && CPU_CAVIUM_OCTEON config CRYPTO_LIB_SM3 tristate if !KMSAN # avoid false positives from assembly
lib/crypto/mips/sha512.h new file +74 -0
74 lines not on this page · raw message
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 11/16 lib/crypto: riscv/sha512: Migrate optimized SHA-512 code to library
In reply to Eric BiggersInstead of exposing the riscv-optimized SHA-512 code via riscv-specific crypto_shash algorithms, instead just implement the sha512_blocks() library function. This is much simpler, it makes the SHA-512 (and SHA-384) library functions be riscv-optimized, and it fixes the longstanding issue where the riscv-optimized SHA-512 code was disabled by default. SHA-512 still remains available through crypto_shash, but individual architectures no longer need to handle it.
To match sha512_blocks(), change the type of the nblocks parameter of the assembly function from int to size_t. The assembly function actually already treated it as size_t.
Note: to see the diff from arch/riscv/crypto/sha512-riscv64-glue.c to lib/crypto/riscv/sha512.h, view this commit with 'git show -M10'.
---
arch/riscv/crypto/Kconfig | 12 --
arch/riscv/crypto/Makefile | 3 -
arch/riscv/crypto/sha512-riscv64-glue.c | 130 ------------------
lib/crypto/Kconfig | 1 +
lib/crypto/Makefile | 2 +
.../riscv}/sha512-riscv64-zvknhb-zvkb.S | 4 +-
lib/crypto/riscv/sha512.h | 41 ++++++
7 files changed, 46 insertions(+), 147 deletions(-)
delete mode 100644 arch/riscv/crypto/sha512-riscv64-glue.c
rename {arch/riscv/crypto => lib/crypto/riscv}/sha512-riscv64-zvknhb-zvkb.S (98%)
create mode 100644 lib/crypto/riscv/sha512.harch/riscv/crypto/Kconfig +0 -12
@@ -26,22 +26,10 @@ config CRYPTO_GHASH_RISCV64 GCM GHASH function (NIST SP 800-38D) Architecture: riscv64 using: - Zvkg vector crypto extension -config CRYPTO_SHA512_RISCV64 - tristate "Hash functions: SHA-384 and SHA-512" - depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO - select CRYPTO_LIB_SHA512 - select CRYPTO_SHA512 - help - SHA-384 and SHA-512 secure hash algorithm (FIPS 180) - - Architecture: riscv64 using: - - Zvknhb vector crypto extension - - Zvkb vector crypto extension - config CRYPTO_SM3_RISCV64 tristate "Hash functions: SM3 (ShangMi 3)" depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO select CRYPTO_HASH select CRYPTO_LIB_SM3
arch/riscv/crypto/Makefile +0 -3
@@ -5,13 +5,10 @@ aes-riscv64-y := aes-riscv64-glue.o aes-riscv64-zvkned.o \ aes-riscv64-zvkned-zvbb-zvkg.o aes-riscv64-zvkned-zvkb.o obj-$(CONFIG_CRYPTO_GHASH_RISCV64) += ghash-riscv64.o ghash-riscv64-y := ghash-riscv64-glue.o ghash-riscv64-zvkg.o -obj-$(CONFIG_CRYPTO_SHA512_RISCV64) += sha512-riscv64.o -sha512-riscv64-y := sha512-riscv64-glue.o sha512-riscv64-zvknhb-zvkb.o - obj-$(CONFIG_CRYPTO_SM3_RISCV64) += sm3-riscv64.o sm3-riscv64-y := sm3-riscv64-glue.o sm3-riscv64-zvksh-zvkb.o obj-$(CONFIG_CRYPTO_SM4_RISCV64) += sm4-riscv64.o sm4-riscv64-y := sm4-riscv64-glue.o sm4-riscv64-zvksed-zvkb.o
arch/riscv/crypto/sha512-riscv64-glue.c deleted +0 -130
130 lines not on this page · raw message
lib/crypto/Kconfig +1 -0
@@ -178,10 +178,11 @@ config CRYPTO_LIB_SHA512_ARCH bool depends on CRYPTO_LIB_SHA512 && !UML default y if ARM && !CPU_V7M default y if ARM64 default y if MIPS && CPU_CAVIUM_OCTEON + default y if RISCV && 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO config CRYPTO_LIB_SM3 tristate if !KMSAN # avoid false positives from assembly
lib/crypto/Makefile +2 -0
@@ -90,10 +90,12 @@ libsha512-y += arm64/sha512-core.o $(obj)/arm64/sha512-core.S: $(src)/../../arch/arm64/lib/crypto/sha2-armv8.pl $(call cmd,perlasm_with_args) clean-files += arm64/sha512-core.S libsha512-$(CONFIG_KERNEL_MODE_NEON) += arm64/sha512-ce-core.o endif + +libsha512-$(CONFIG_RISCV) += riscv/sha512-riscv64-zvknhb-zvkb.o endif # CONFIG_CRYPTO_LIB_SHA512_ARCH obj-$(CONFIG_MPILIB) += mpi/ obj-$(CONFIG_CRYPTO_SELFTESTS_FULL) += simd.o
lib/crypto/riscv/sha512-riscv64-zvknhb-zvkb.S renamed from arch/riscv/crypto/sha512-riscv64-zvknhb-zvkb.S +2 -2
@@ -91,12 +91,12 @@ sha512_4rounds \last, W1, W2, W3, W0 sha512_4rounds \last, W2, W3, W0, W1 sha512_4rounds \last, W3, W0, W1, W2 .endm -// void sha512_transform_zvknhb_zvkb(u64 state[8], const u8 *data, -// int num_blocks); +// void sha512_transform_zvknhb_zvkb(struct sha512_block_state *state, +// const u8 *data, size_t nblocks); SYM_FUNC_START(sha512_transform_zvknhb_zvkb) // Setup mask for the vmerge to replace the first word (idx==0) in // message scheduling. There are 4 words, so an 8-bit mask suffices. vsetivli zero, 1, e8, m1, ta, ma
lib/crypto/riscv/sha512.h new file +41 -0
41 lines not on this page · raw message
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 12/16 lib/crypto: s390/sha512: Migrate optimized SHA-512 code to library
In reply to Eric BiggersInstead of exposing the s390-optimized SHA-512 code via s390-specific crypto_shash algorithms, instead just implement the sha512_blocks() library function. This is much simpler, it makes the SHA-512 (and SHA-384) library functions be s390-optimized, and it fixes the longstanding issue where the s390-optimized SHA-512 code was disabled by default. SHA-512 still remains available through crypto_shash, but individual architectures no longer need to handle it.
--- arch/s390/configs/debug_defconfig | 1 - arch/s390/configs/defconfig | 1 - arch/s390/crypto/Kconfig | 10 -- arch/s390/crypto/Makefile | 1 - arch/s390/crypto/sha512_s390.c | 151 ------------------------------ lib/crypto/Kconfig | 1 + lib/crypto/s390/sha512.h | 28 ++++++ 7 files changed, 29 insertions(+), 164 deletions(-) delete mode 100644 arch/s390/crypto/sha512_s390.c create mode 100644 lib/crypto/s390/sha512.h
arch/s390/configs/debug_defconfig +0 -1
@@ -802,11 +802,10 @@ CONFIG_CRYPTO_ZSTD=m CONFIG_CRYPTO_ANSI_CPRNG=m CONFIG_CRYPTO_USER_API_HASH=m CONFIG_CRYPTO_USER_API_SKCIPHER=m CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_USER_API_AEAD=m -CONFIG_CRYPTO_SHA512_S390=m CONFIG_CRYPTO_SHA1_S390=m CONFIG_CRYPTO_SHA3_256_S390=m CONFIG_CRYPTO_SHA3_512_S390=m CONFIG_CRYPTO_GHASH_S390=m CONFIG_CRYPTO_AES_S390=m
arch/s390/configs/defconfig +0 -1
@@ -789,11 +789,10 @@ CONFIG_CRYPTO_ANSI_CPRNG=m CONFIG_CRYPTO_JITTERENTROPY_OSR=1 CONFIG_CRYPTO_USER_API_HASH=m CONFIG_CRYPTO_USER_API_SKCIPHER=m CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_USER_API_AEAD=m -CONFIG_CRYPTO_SHA512_S390=m CONFIG_CRYPTO_SHA1_S390=m CONFIG_CRYPTO_SHA3_256_S390=m CONFIG_CRYPTO_SHA3_512_S390=m CONFIG_CRYPTO_GHASH_S390=m CONFIG_CRYPTO_AES_S390=m
arch/s390/crypto/Kconfig +0 -10
@@ -1,19 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 menu "Accelerated Cryptographic Algorithms for CPU (s390)" -config CRYPTO_SHA512_S390 - tristate "Hash functions: SHA-384 and SHA-512" - select CRYPTO_HASH - help - SHA-384 and SHA-512 secure hash algorithms (FIPS 180) - - Architecture: s390 - - It is available as of z10. - config CRYPTO_SHA1_S390 tristate "Hash functions: SHA-1" select CRYPTO_HASH help SHA-1 secure hash algorithm (FIPS 180)
arch/s390/crypto/Makefile +0 -1
@@ -2,11 +2,10 @@ # # Cryptographic API # obj-$(CONFIG_CRYPTO_SHA1_S390) += sha1_s390.o sha_common.o -obj-$(CONFIG_CRYPTO_SHA512_S390) += sha512_s390.o sha_common.o obj-$(CONFIG_CRYPTO_SHA3_256_S390) += sha3_256_s390.o sha_common.o obj-$(CONFIG_CRYPTO_SHA3_512_S390) += sha3_512_s390.o sha_common.o obj-$(CONFIG_CRYPTO_DES_S390) += des_s390.o obj-$(CONFIG_CRYPTO_AES_S390) += aes_s390.o obj-$(CONFIG_CRYPTO_PAES_S390) += paes_s390.o
arch/s390/crypto/sha512_s390.c deleted +0 -151
151 lines not on this page · raw message
lib/crypto/Kconfig +1 -0
@@ -179,10 +179,11 @@ config CRYPTO_LIB_SHA512_ARCH depends on CRYPTO_LIB_SHA512 && !UML default y if ARM && !CPU_V7M default y if ARM64 default y if MIPS && CPU_CAVIUM_OCTEON default y if RISCV && 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO + default y if S390 config CRYPTO_LIB_SM3 tristate if !KMSAN # avoid false positives from assembly
lib/crypto/s390/sha512.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* SHA-512 optimized using the CP Assist for Cryptographic Functions (CPACF) +* +* Copyright 2025 Google LLC +*/ +#include <asm/cpacf.h> +#include <linux/cpufeature.h> + +static __ro_after_init DEFINE_STATIC_KEY_FALSE (have_cpacf_sha512); + +static void sha512_blocks (struct sha512_block_state *state, +const u8 *data,size_t nblocks) +{ +if (static_branch_likely (&have_cpacf_sha512)) +cpacf_kimd (CPACF_KIMD_SHA_512 , state, data, + nblocks *SHA512_BLOCK_SIZE ); +else +sha512_blocks_generic (state, data, nblocks); +} + +#define sha512_mod_init_arch sha512_mod_init_arch +static inline void sha512_mod_init_arch (void ) +{ +if (cpu_have_feature (S390_CPU_FEATURE_MSA ) && +cpacf_query_func (CPACF_KIMD ,CPACF_KIMD_SHA_512 )) +static_branch_enable (&have_cpacf_sha512); +}
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 13/16 lib/crypto: sparc/sha512: Migrate optimized SHA-512 code to library
In reply to Eric BiggersInstead of exposing the sparc-optimized SHA-512 code via sparc-specific crypto_shash algorithms, instead just implement the sha512_blocks() library function. This is much simpler, it makes the SHA-512 (and SHA-384) library functions be sparc-optimized, and it fixes the longstanding issue where the sparc-optimized SHA-512 code was disabled by default. SHA-512 still remains available through crypto_shash, but individual architectures no longer need to handle it.
To match sha512_blocks(), change the type of the nblocks parameter of the assembly function from int to size_t. The assembly function actually already treated it as size_t.
Note: to see the diff from arch/sparc/crypto/sha512_glue.c to lib/crypto/sparc/sha512.h, view this commit with 'git show -M10'.
---
arch/sparc/crypto/Kconfig | 10 --
arch/sparc/crypto/Makefile | 2 -
arch/sparc/crypto/sha512_glue.c | 122 ------------------
lib/crypto/Kconfig | 1 +
lib/crypto/Makefile | 1 +
lib/crypto/sparc/sha512.h | 42 ++++++
.../crypto => lib/crypto/sparc}/sha512_asm.S | 0
7 files changed, 44 insertions(+), 134 deletions(-)
delete mode 100644 arch/sparc/crypto/sha512_glue.c
create mode 100644 lib/crypto/sparc/sha512.h
rename {arch/sparc/crypto => lib/crypto/sparc}/sha512_asm.S (100%)arch/sparc/crypto/Kconfig +0 -10
@@ -34,20 +34,10 @@ config CRYPTO_SHA1_SPARC64 help SHA-1 secure hash algorithm (FIPS 180) Architecture: sparc64 -config CRYPTO_SHA512_SPARC64 - tristate "Hash functions: SHA-384 and SHA-512" - depends on SPARC64 - select CRYPTO_SHA512 - select CRYPTO_HASH - help - SHA-384 and SHA-512 secure hash algorithms (FIPS 180) - - Architecture: sparc64 using crypto instructions, when available - config CRYPTO_AES_SPARC64 tristate "Ciphers: AES, modes: ECB, CBC, CTR" depends on SPARC64 select CRYPTO_SKCIPHER help
arch/sparc/crypto/Makefile +0 -2
@@ -2,19 +2,17 @@ # # Arch-specific CryptoAPI modules. # obj-$(CONFIG_CRYPTO_SHA1_SPARC64) += sha1-sparc64.o -obj-$(CONFIG_CRYPTO_SHA512_SPARC64) += sha512-sparc64.o obj-$(CONFIG_CRYPTO_MD5_SPARC64) += md5-sparc64.o obj-$(CONFIG_CRYPTO_AES_SPARC64) += aes-sparc64.o obj-$(CONFIG_CRYPTO_DES_SPARC64) += des-sparc64.o obj-$(CONFIG_CRYPTO_CAMELLIA_SPARC64) += camellia-sparc64.o sha1-sparc64-y := sha1_asm.o sha1_glue.o -sha512-sparc64-y := sha512_asm.o sha512_glue.o md5-sparc64-y := md5_asm.o md5_glue.o aes-sparc64-y := aes_asm.o aes_glue.o des-sparc64-y := des_asm.o des_glue.o camellia-sparc64-y := camellia_asm.o camellia_glue.o
arch/sparc/crypto/sha512_glue.c deleted +0 -122
122 lines not on this page · raw message
lib/crypto/Kconfig +1 -0
@@ -180,10 +180,11 @@ config CRYPTO_LIB_SHA512_ARCH default y if ARM && !CPU_V7M default y if ARM64 default y if MIPS && CPU_CAVIUM_OCTEON default y if RISCV && 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO default y if S390 + default y if SPARC64 config CRYPTO_LIB_SM3 tristate if !KMSAN # avoid false positives from assembly
lib/crypto/Makefile +1 -0
@@ -92,10 +92,11 @@ $(obj)/arm64/sha512-core.S: $(src)/../../arch/arm64/lib/crypto/sha2-armv8.pl clean-files += arm64/sha512-core.S libsha512-$(CONFIG_KERNEL_MODE_NEON) += arm64/sha512-ce-core.o endif libsha512-$(CONFIG_RISCV) += riscv/sha512-riscv64-zvknhb-zvkb.o +libsha512-$(CONFIG_SPARC) += sparc/sha512_asm.o endif # CONFIG_CRYPTO_LIB_SHA512_ARCH obj-$(CONFIG_MPILIB) += mpi/ obj-$(CONFIG_CRYPTO_SELFTESTS_FULL) += simd.o
lib/crypto/sparc/sha512.h new file +42 -0
42 lines not on this page · raw message
lib/crypto/sparc/sha512_asm.S renamed from arch/sparc/crypto/sha512_asm.S +0 -0
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 14/16 lib/crypto: x86/sha512: Migrate optimized SHA-512 code to library
In reply to Eric BiggersInstead of exposing the x86-optimized SHA-512 code via x86-specific crypto_shash algorithms, instead just implement the sha512_blocks() library function. This is much simpler, it makes the SHA-512 (and SHA-384) library functions be x86-optimized, and it fixes the longstanding issue where the x86-optimized SHA-512 code was disabled by default. SHA-512 still remains available through crypto_shash, but individual architectures no longer need to handle it.
To match sha512_blocks(), change the type of the nblocks parameter of the assembly functions from int to size_t. The assembly functions actually already treated it as size_t.
---
arch/x86/crypto/Kconfig | 13 -
arch/x86/crypto/Makefile | 3 -
arch/x86/crypto/sha512_ssse3_glue.c | 322 ------------------
lib/crypto/Kconfig | 1 +
lib/crypto/Makefile | 3 +
.../crypto/x86}/sha512-avx-asm.S | 9 +-
.../crypto/x86}/sha512-avx2-asm.S | 9 +-
.../crypto/x86}/sha512-ssse3-asm.S | 10 +-
lib/crypto/x86/sha512.h | 54 +++
9 files changed, 72 insertions(+), 352 deletions(-)
delete mode 100644 arch/x86/crypto/sha512_ssse3_glue.c
rename {arch/x86/crypto => lib/crypto/x86}/sha512-avx-asm.S (98%)
rename {arch/x86/crypto => lib/crypto/x86}/sha512-avx2-asm.S (99%)
rename {arch/x86/crypto => lib/crypto/x86}/sha512-ssse3-asm.S (97%)
create mode 100644 lib/crypto/x86/sha512.harch/x86/crypto/Kconfig +0 -13
@@ -388,23 +388,10 @@ config CRYPTO_SHA1_SSSE3 - SSSE3 (Supplemental SSE3) - AVX (Advanced Vector Extensions) - AVX2 (Advanced Vector Extensions 2) - SHA-NI (SHA Extensions New Instructions) -config CRYPTO_SHA512_SSSE3 - tristate "Hash functions: SHA-384 and SHA-512 (SSSE3/AVX/AVX2)" - depends on 64BIT - select CRYPTO_SHA512 - select CRYPTO_HASH - help - SHA-384 and SHA-512 secure hash algorithms (FIPS 180) - - Architecture: x86_64 using: - - SSSE3 (Supplemental SSE3) - - AVX (Advanced Vector Extensions) - - AVX2 (Advanced Vector Extensions 2) - config CRYPTO_SM3_AVX_X86_64 tristate "Hash functions: SM3 (AVX)" depends on 64BIT select CRYPTO_HASH select CRYPTO_LIB_SM3
arch/x86/crypto/Makefile +0 -3
@@ -52,13 +52,10 @@ aesni-intel-$(CONFIG_64BIT) += aes-gcm-avx10-x86_64.o endif obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ni_asm.o sha1_ssse3_glue.o -obj-$(CONFIG_CRYPTO_SHA512_SSSE3) += sha512-ssse3.o -sha512-ssse3-y := sha512-ssse3-asm.o sha512-avx-asm.o sha512-avx2-asm.o sha512_ssse3_glue.o - obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o obj-$(CONFIG_CRYPTO_POLYVAL_CLMUL_NI) += polyval-clmulni.o polyval-clmulni-y := polyval-clmulni_asm.o polyval-clmulni_glue.o
arch/x86/crypto/sha512_ssse3_glue.c deleted +0 -322
322 lines not on this page · raw message
lib/crypto/Kconfig +1 -0
@@ -181,10 +181,11 @@ config CRYPTO_LIB_SHA512_ARCH default y if ARM64 default y if MIPS && CPU_CAVIUM_OCTEON default y if RISCV && 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO default y if S390 default y if SPARC64 + default y if X86_64 config CRYPTO_LIB_SM3 tristate if !KMSAN # avoid false positives from assembly
lib/crypto/Makefile +3 -0
@@ -93,10 +93,13 @@ clean-files += arm64/sha512-core.S libsha512-$(CONFIG_KERNEL_MODE_NEON) += arm64/sha512-ce-core.o endif libsha512-$(CONFIG_RISCV) += riscv/sha512-riscv64-zvknhb-zvkb.o libsha512-$(CONFIG_SPARC) += sparc/sha512_asm.o +libsha512-$(CONFIG_X86) += x86/sha512-ssse3-asm.o \ + x86/sha512-avx-asm.o \ + x86/sha512-avx2-asm.o endif # CONFIG_CRYPTO_LIB_SHA512_ARCH obj-$(CONFIG_MPILIB) += mpi/ obj-$(CONFIG_CRYPTO_SELFTESTS_FULL) += simd.o
lib/crypto/x86/sha512-avx-asm.S renamed from arch/x86/crypto/sha512-avx-asm.S +5 -4
@@ -46,11 +46,10 @@ # and search for that title. # ######################################################################## #include <linux/linkage.h> -#include <linux/cfi_types.h> .text # Virtual Registers # ARG1
@@ -265,18 +264,20 @@ frame_size = frame_WK + WK_SIZE add tmp0, h_64 RotateState .endm ######################################################################## -# void sha512_transform_avx(sha512_state *state, const u8 *data, int blocks) +# void sha512_transform_avx(struct sha512_block_state *state, +# const u8 *data, size_t nblocks); # Purpose: Updates the SHA512 digest stored at "state" with the message # stored in "data". # The size of the message pointed to by "data" must be an integer multiple # of SHA512 message blocks. -# "blocks" is the message length in SHA512 blocks +# "nblocks" is the message length in SHA512 blocks ######################################################################## -SYM_TYPED_FUNC_START(sha512_transform_avx) +SYM_FUNC_START(sha512_transform_avx) + test msglen, msglen je .Lnowork # Save GPRs push %rbx
lib/crypto/x86/sha512-avx2-asm.S renamed from arch/x86/crypto/sha512-avx2-asm.S +5 -4
@@ -48,11 +48,10 @@ ######################################################################## # This code schedules 1 blocks at a time, with 4 lanes per block ######################################################################## #include <linux/linkage.h> -#include <linux/cfi_types.h> .text # Virtual Registers Y_0 = %ymm4
@@ -557,18 +556,20 @@ frame_size = frame_CTX + CTX_SIZE RotateState .endm ######################################################################## -# void sha512_transform_rorx(sha512_state *state, const u8 *data, int blocks) +# void sha512_transform_rorx(struct sha512_block_state *state, +# const u8 *data, size_t nblocks); # Purpose: Updates the SHA512 digest stored at "state" with the message # stored in "data". # The size of the message pointed to by "data" must be an integer multiple # of SHA512 message blocks. -# "blocks" is the message length in SHA512 blocks +# "nblocks" is the message length in SHA512 blocks ######################################################################## -SYM_TYPED_FUNC_START(sha512_transform_rorx) +SYM_FUNC_START(sha512_transform_rorx) + # Save GPRs push %rbx push %r12 push %r13 push %r14
lib/crypto/x86/sha512-ssse3-asm.S renamed from arch/x86/crypto/sha512-ssse3-asm.S +4 -6
@@ -46,11 +46,10 @@ # and search for that title. # ######################################################################## #include <linux/linkage.h> -#include <linux/cfi_types.h> .text # Virtual Registers # ARG1
@@ -264,20 +263,19 @@ frame_size = frame_WK + WK_SIZE lea (T1, T2), h_64 RotateState .endm ######################################################################## -## void sha512_transform_ssse3(struct sha512_state *state, const u8 *data, -## int blocks); -# (struct sha512_state is assumed to begin with u64 state[8]) +# void sha512_transform_ssse3(struct sha512_block_state *state, +# const u8 *data, size_t nblocks); # Purpose: Updates the SHA512 digest stored at "state" with the message # stored in "data". # The size of the message pointed to by "data" must be an integer multiple # of SHA512 message blocks. -# "blocks" is the message length in SHA512 blocks. +# "nblocks" is the message length in SHA512 blocks ######################################################################## -SYM_TYPED_FUNC_START(sha512_transform_ssse3) +SYM_FUNC_START(sha512_transform_ssse3) test msglen, msglen je .Lnowork # Save GPRs
lib/crypto/x86/sha512.h new file +54 -0
54 lines not on this page · raw message
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 15/16 lib/crypto: x86/sha512: Remove unnecessary checks for nblocks==0
In reply to Eric BiggersSince sha512_blocks() is called only with nblocks >= 1, remove unnecessary checks for nblocks == 0 from the x86 SHA-512 assembly code.
--- lib/crypto/x86/sha512-avx-asm.S | 6 +----- lib/crypto/x86/sha512-avx2-asm.S | 5 +---- lib/crypto/x86/sha512-ssse3-asm.S | 6 +----- 3 files changed, 3 insertions(+), 14 deletions(-)
lib/crypto/x86/sha512-avx-asm.S +1 -5
@@ -270,17 +270,14 @@ frame_size = frame_WK + WK_SIZE # const u8 *data, size_t nblocks); # Purpose: Updates the SHA512 digest stored at "state" with the message # stored in "data". # The size of the message pointed to by "data" must be an integer multiple # of SHA512 message blocks. -# "nblocks" is the message length in SHA512 blocks +# "nblocks" is the message length in SHA512 blocks. Must be >= 1. ######################################################################## SYM_FUNC_START(sha512_transform_avx) - test msglen, msglen - je .Lnowork - # Save GPRs push %rbx push %r12 push %r13 push %r14
@@ -360,11 +357,10 @@ SYM_FUNC_START(sha512_transform_avx) pop %r14 pop %r13 pop %r12 pop %rbx -.Lnowork: RET SYM_FUNC_END(sha512_transform_avx) ######################################################################## ### Binary Data
lib/crypto/x86/sha512-avx2-asm.S +1 -4
@@ -562,11 +562,11 @@ frame_size = frame_CTX + CTX_SIZE # const u8 *data, size_t nblocks); # Purpose: Updates the SHA512 digest stored at "state" with the message # stored in "data". # The size of the message pointed to by "data" must be an integer multiple # of SHA512 message blocks. -# "nblocks" is the message length in SHA512 blocks +# "nblocks" is the message length in SHA512 blocks. Must be >= 1. ######################################################################## SYM_FUNC_START(sha512_transform_rorx) # Save GPRs push %rbx
@@ -580,11 +580,10 @@ SYM_FUNC_START(sha512_transform_rorx) mov %rsp, %rbp sub $frame_size, %rsp and $~(0x20 - 1), %rsp shl $7, NUM_BLKS # convert to bytes - jz .Ldone_hash add INP, NUM_BLKS # pointer to end of data mov NUM_BLKS, frame_INPEND(%rsp) ## load initial digest mov 8*0(CTX1), a
@@ -666,12 +665,10 @@ SYM_FUNC_START(sha512_transform_rorx) mov frame_INP(%rsp), INP add $128, INP cmp frame_INPEND(%rsp), INP jne .Lloop0 -.Ldone_hash: - # Restore Stack Pointer mov %rbp, %rsp pop %rbp # Restore GPRs
lib/crypto/x86/sha512-ssse3-asm.S +1 -5
@@ -269,17 +269,14 @@ frame_size = frame_WK + WK_SIZE # const u8 *data, size_t nblocks); # Purpose: Updates the SHA512 digest stored at "state" with the message # stored in "data". # The size of the message pointed to by "data" must be an integer multiple # of SHA512 message blocks. -# "nblocks" is the message length in SHA512 blocks +# "nblocks" is the message length in SHA512 blocks. Must be >= 1. ######################################################################## SYM_FUNC_START(sha512_transform_ssse3) - test msglen, msglen - je .Lnowork - # Save GPRs push %rbx push %r12 push %r13 push %r14
@@ -359,11 +356,10 @@ SYM_FUNC_START(sha512_transform_ssse3) pop %r14 pop %r13 pop %r12 pop %rbx -.Lnowork: RET SYM_FUNC_END(sha512_transform_ssse3) ######################################################################## ### Binary Data
Signature · 1 line
2.50.0
Eric Biggers
maintainerPATCH v3 16/16 crypto: sha512 - Remove sha512_base.h
In reply to Eric Biggerssha512_base.h is no longer used, so remove it.
--- include/crypto/sha512_base.h | 117 ----------------------------------- 1 file changed, 117 deletions(-) delete mode 100644 include/crypto/sha512_base.h
include/crypto/sha512_base.h deleted +0 -117
@@ -1,117 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* -* sha512_base.h - core logic for SHA-512 implementations -* -* Copyright (C) 2015 Linaro Ltd <[email protected]> -*/ - -#ifndef _CRYPTO_SHA512_BASE_H -#define _CRYPTO_SHA512_BASE_H - -#include <crypto/internal/hash.h> -#include <crypto/sha2.h> -#include <linux/compiler.h> -#include <linux/math.h> -#include <linux/string.h> -#include <linux/types.h> -#include <linux/unaligned.h> - -typedef void (sha512_block_fn )(struct sha512_state *sst,u8 const *src, -int blocks); - -static inline int sha384_base_init (struct shash_desc *desc) -{ -struct sha512_state *sctx =shash_desc_ctx (desc); - - sctx->state[0 ] =SHA384_H0 ; - sctx->state[1 ] =SHA384_H1 ; - sctx->state[2 ] =SHA384_H2 ; - sctx->state[3 ] =SHA384_H3 ; - sctx->state[4 ] =SHA384_H4 ; - sctx->state[5 ] =SHA384_H5 ; - sctx->state[6 ] =SHA384_H6 ; - sctx->state[7 ] =SHA384_H7 ; - sctx->count[0 ] = sctx->count[1 ] =0 ; - -return 0 ; -} - -static inline int sha512_base_init (struct shash_desc *desc) -{ -struct sha512_state *sctx =shash_desc_ctx (desc); - - sctx->state[0 ] =SHA512_H0 ; - sctx->state[1 ] =SHA512_H1 ; - sctx->state[2 ] =SHA512_H2 ; - sctx->state[3 ] =SHA512_H3 ; - sctx->state[4 ] =SHA512_H4 ; - sctx->state[5 ] =SHA512_H5 ; - sctx->state[6 ] =SHA512_H6 ; - sctx->state[7 ] =SHA512_H7 ; - sctx->count[0 ] = sctx->count[1 ] =0 ; - -return 0 ; -} - -static inline int sha512_base_do_update_blocks (struct shash_desc *desc, -const u8 *data, -unsigned int len, -sha512_block_fn *block_fn) -{ -unsigned int remain = len -round_down (len,SHA512_BLOCK_SIZE ); -struct sha512_state *sctx =shash_desc_ctx (desc); - - len -= remain; - sctx->count[0 ] += len; -if (sctx->count[0 ] < len) - sctx->count[1 ]++; -block_fn (sctx, data, len /SHA512_BLOCK_SIZE ); -return remain; -} - -static inline int sha512_base_do_finup (struct shash_desc *desc,const u8 *src, -unsigned int len, -sha512_block_fn *block_fn) -{ -unsigned int bit_offset =SHA512_BLOCK_SIZE /8 -2 ; -struct sha512_state *sctx =shash_desc_ctx (desc); -union { -__be64 b64[SHA512_BLOCK_SIZE /4 ]; -u8 u8[SHA512_BLOCK_SIZE *2 ]; - } block = {}; - -if (len >=SHA512_BLOCK_SIZE ) { -int remain; - - remain =sha512_base_do_update_blocks (desc, src, len, block_fn); - src += len - remain; - len = remain; - } - -if (len >= bit_offset *8 ) - bit_offset +=SHA512_BLOCK_SIZE /8 ; -memcpy (&block, src, len); - block.u8[len] =0x80 ; - sctx->count[0 ] += len; - block.b64[bit_offset] =cpu_to_be64 (sctx->count[1 ] <<3 | - sctx->count[0 ] >>61 ); - block.b64[bit_offset +1 ] =cpu_to_be64 (sctx->count[0 ] <<3 ); -block_fn (sctx, block.u8, (bit_offset +2 ) *8 /SHA512_BLOCK_SIZE ); -memzero_explicit (&block,sizeof (block)); - -return 0 ; -} - -static inline int sha512_base_finish (struct shash_desc *desc,u8 *out) -{ -unsigned int digest_size =crypto_shash_digestsize (desc->tfm); -struct sha512_state *sctx =shash_desc_ctx (desc); -__be64 *digest = (__be64 *)out; -int i; - -for (i =0 ; digest_size >0 ; i++, digest_size -=sizeof (__be64)) -put_unaligned_be64 (sctx->state[i], digest++); -return 0 ; -} - -#endif /* _CRYPTO_SHA512_BASE_H */
Signature · 1 line
2.50.0