ChaCha and BLAKE2s cleanups
Subsystems (8)
Eric Biggers
maintainercover ChaCha and BLAKE2s cleanups
This series is targeting libcrypto-next. It can also be retrieved from:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git chacha-blake2s-v1
This series consolidates how the ChaCha and BLAKE2s code is organized. This is essentially the same change that I made to the other algorithms, so this should be fairly boring by now.
These algorithms were the last two users of lib/crypto/$(SRCARCH)/{Makefile,Kconfig}. So this series removes all those files, finishing the transition to the centralized build process (at least for the algorithms supported by lib/crypto/ so far).
This series also makes the arch-optimized BLAKE2s code start being enabled by default, again following the pattern of the other algorithms.
Finally, it adds a KUnit test suite for BLAKE2s and deletes the older blake2s-selftest.
Eric Biggers (12):
arm: configs: Remove obsolete assignments to CRYPTO_CHACHA20_NEON
crypto: chacha - register only "-lib" drivers
lib/crypto: chacha: Remove unused function chacha_is_arch_optimized() lib/crypto: chacha: Rename chacha.c to chacha-block-generic.c lib/crypto: chacha: Rename libchacha.c to chacha.c lib/crypto: chacha: Consolidate into single module lib/crypto: x86/blake2s: Reduce size of BLAKE2S_SIGMA2 lib/crypto: blake2s: Remove obsolete self-test
lib/crypto: blake2s: Always enable arch-optimized BLAKE2s code lib/crypto: blake2s: Move generic code into blake2s.c lib/crypto: blake2s: Consolidate into single C translation unit lib/crypto: tests: Add KUnit tests for BLAKE2s
arch/arm/configs/exynos_defconfig | 1 -
arch/arm/configs/milbeaut_m10v_defconfig | 1 -
arch/arm/configs/multi_v7_defconfig | 1 -
arch/arm/configs/omap2plus_defconfig | 1 -
crypto/Kconfig | 1 -
crypto/chacha.c | 129 +---
crypto/testmgr.c | 9 +-
include/crypto/chacha.h | 37 +-
include/crypto/internal/blake2s.h | 21 -
lib/crypto/Kconfig | 74 +-
lib/crypto/Makefile | 57 +-
lib/crypto/arm/Kconfig | 19 -
lib/crypto/arm/Makefile | 8 -
lib/crypto/arm/blake2s-core.S | 5 +-
lib/crypto/arm/blake2s-glue.c | 7 -
lib/crypto/arm/blake2s.h | 5 +
lib/crypto/arm/{chacha-glue.c => chacha.h} | 35 +-
lib/crypto/arm64/Kconfig | 8 -
lib/crypto/arm64/Makefile | 4 -
.../arm64/{chacha-neon-glue.c => chacha.h} | 32 +-
lib/crypto/blake2s-generic.c | 111 ---
lib/crypto/blake2s-selftest.c | 651 ------------------
lib/crypto/blake2s.c | 105 ++-
lib/crypto/chacha-block-generic.c | 114 +++
lib/crypto/chacha.c | 142 ++--
lib/crypto/libchacha.c | 35 -
lib/crypto/mips/Kconfig | 7 -
lib/crypto/mips/Makefile | 5 -
lib/crypto/mips/chacha-glue.c | 29 -
lib/crypto/mips/chacha.h | 14 +
lib/crypto/powerpc/Kconfig | 8 -
lib/crypto/powerpc/Makefile | 4 -
.../powerpc/{chacha-p10-glue.c => chacha.h} | 36 +-
lib/crypto/riscv/Kconfig | 8 -
lib/crypto/riscv/Makefile | 4 -
.../riscv/{chacha-riscv64-glue.c => chacha.h} | 36 +-
lib/crypto/s390/Kconfig | 7 -
lib/crypto/s390/Makefile | 4 -
lib/crypto/s390/{chacha-glue.c => chacha.h} | 29 +-
lib/crypto/tests/Kconfig | 10 +
lib/crypto/tests/Makefile | 1 +
lib/crypto/tests/blake2s-testvecs.h | 238 +++++++
lib/crypto/tests/blake2s_kunit.c | 134 ++++
lib/crypto/x86/Kconfig | 20 -
lib/crypto/x86/Makefile | 7 -
lib/crypto/x86/blake2s-core.S | 28 +-
lib/crypto/x86/{blake2s-glue.c => blake2s.h} | 16 +-
lib/crypto/x86/{chacha_glue.c => chacha.h} | 36 +-
scripts/crypto/gen-hash-testvecs.py | 27 +-
49 files changed, 840 insertions(+), 1481 deletions(-)
delete mode 100644 include/crypto/internal/blake2s.h
delete mode 100644 lib/crypto/arm/Kconfig
delete mode 100644 lib/crypto/arm/Makefile
delete mode 100644 lib/crypto/arm/blake2s-glue.c
create mode 100644 lib/crypto/arm/blake2s.h
rename lib/crypto/arm/{chacha-glue.c => chacha.h} (76%)
delete mode 100644 lib/crypto/arm64/Kconfig
delete mode 100644 lib/crypto/arm64/Makefile
rename lib/crypto/arm64/{chacha-neon-glue.c => chacha.h} (75%)
delete mode 100644 lib/crypto/blake2s-generic.c
delete mode 100644 lib/crypto/blake2s-selftest.c
create mode 100644 lib/crypto/chacha-block-generic.c
delete mode 100644 lib/crypto/libchacha.c
delete mode 100644 lib/crypto/mips/Kconfig
delete mode 100644 lib/crypto/mips/Makefile
delete mode 100644 lib/crypto/mips/chacha-glue.c
create mode 100644 lib/crypto/mips/chacha.h
delete mode 100644 lib/crypto/powerpc/Kconfig
delete mode 100644 lib/crypto/powerpc/Makefile
rename lib/crypto/powerpc/{chacha-p10-glue.c => chacha.h} (62%)
delete mode 100644 lib/crypto/riscv/Kconfig
delete mode 100644 lib/crypto/riscv/Makefile
rename lib/crypto/riscv/{chacha-riscv64-glue.c => chacha.h} (57%)
delete mode 100644 lib/crypto/s390/Kconfig
delete mode 100644 lib/crypto/s390/Makefile
rename lib/crypto/s390/{chacha-glue.c => chacha.h} (51%)
create mode 100644 lib/crypto/tests/blake2s-testvecs.h
create mode 100644 lib/crypto/tests/blake2s_kunit.c
delete mode 100644 lib/crypto/x86/Kconfig
delete mode 100644 lib/crypto/x86/Makefile
rename lib/crypto/x86/{blake2s-glue.c => blake2s.h} (83%)
rename lib/crypto/x86/{chacha_glue.c => chacha.h} (85%)base-commit: 44781c45f26623c3b92b28e933bf349144c10fe6
Signature · 1 line
2.50.1
Eric Biggers
maintainerPATCH 1/12 arm: configs: Remove obsolete assignments to CRYPTO_CHACHA20_NEON
Since v6.15, CRYPTO_CHACHA20_NEON is a hidden option and is selected automatically. Therefore, assigning a value to it in a defconfig no longer has any effect. Remove it from all files that did this.
--- arch/arm/configs/exynos_defconfig | 1 - arch/arm/configs/milbeaut_m10v_defconfig | 1 - arch/arm/configs/multi_v7_defconfig | 1 - arch/arm/configs/omap2plus_defconfig | 1 - 4 files changed, 4 deletions(-)
arch/arm/configs/exynos_defconfig +0 -1
@@ -362,11 +362,10 @@ CONFIG_CRYPTO_LZ4=m CONFIG_CRYPTO_USER_API_HASH=m CONFIG_CRYPTO_USER_API_SKCIPHER=m CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_USER_API_AEAD=m CONFIG_CRYPTO_AES_ARM_BS=m -CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRYPTO_DEV_EXYNOS_RNG=y CONFIG_CRYPTO_DEV_S5P=y CONFIG_DMA_CMA=y CONFIG_CMA_SIZE_MBYTES=96 CONFIG_FONTS=y
arch/arm/configs/milbeaut_m10v_defconfig +0 -1
@@ -99,11 +99,10 @@ CONFIG_CRYPTO_AES=y CONFIG_CRYPTO_SEQIV=m CONFIG_CRYPTO_GHASH_ARM_CE=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_AES_ARM_CE=m -CONFIG_CRYPTO_CHACHA20_NEON=m # CONFIG_CRYPTO_HW is not set CONFIG_DMA_CMA=y CONFIG_CMA_SIZE_MBYTES=64 CONFIG_PRINTK_TIME=y CONFIG_MAGIC_SYSRQ=y
arch/arm/configs/multi_v7_defconfig +0 -1
@@ -1289,11 +1289,10 @@ CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_USER_API_AEAD=m CONFIG_CRYPTO_GHASH_ARM_CE=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_AES_ARM_CE=m -CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRYPTO_DEV_SUN4I_SS=m CONFIG_CRYPTO_DEV_FSL_CAAM=m CONFIG_CRYPTO_DEV_EXYNOS_RNG=m CONFIG_CRYPTO_DEV_S5P=m CONFIG_CRYPTO_DEV_ATMEL_AES=m
arch/arm/configs/omap2plus_defconfig +0 -1
@@ -706,11 +706,10 @@ CONFIG_NLS_ISO8859_1=y CONFIG_SECURITY=y CONFIG_CRYPTO_MICHAEL_MIC=y CONFIG_CRYPTO_GHASH_ARM_CE=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m -CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRYPTO_DEV_OMAP=m CONFIG_CRYPTO_DEV_OMAP_SHAM=m CONFIG_CRYPTO_DEV_OMAP_AES=m CONFIG_CRYPTO_DEV_OMAP_DES=m CONFIG_DMA_CMA=y
Signature · 1 line
2.50.1
Eric Biggers
maintainerPATCH 2/12 crypto: chacha - register only "-lib" drivers
In reply to Eric BiggersFor the "chacha20", "xchacha20", and "xchacha12" skcipher algorithms, instead of registering "*-generic" drivers as well as conditionally registering "*-$(ARCH)" drivers, instead just register "*-lib" drivers. These just use the regular library functions, so they just do the right thing and are fully accelerated when supported by the CPU.
This eliminates the need for the ChaCha library to support chacha_crypt_generic() and hchacha_block_generic() as part of its external interface. A later commit will make chacha_crypt_generic() a static function.
Since this commit removes several "*-generic" driver names which crypto/testmgr.c expects to exist, update testmgr.c accordingly.
--- crypto/Kconfig | 1 - crypto/chacha.c | 129 ++++++++--------------------------------------- crypto/testmgr.c | 9 +++- 3 files changed, 29 insertions(+), 110 deletions(-)
crypto/Kconfig +0 -1
@@ -646,11 +646,10 @@ config CRYPTO_ARC4 weakness of the algorithm. config CRYPTO_CHACHA20 tristate "ChaCha" select CRYPTO_LIB_CHACHA - select CRYPTO_LIB_CHACHA_GENERIC select CRYPTO_SKCIPHER help The ChaCha20, XChaCha20, and XChaCha12 stream cipher algorithms ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J.
crypto/chacha.c +22 -107
233 lines not on this page · raw message
crypto/testmgr.c +7 -2
@@ -4150,18 +4150,18 @@ static int alg_test_null(const struct alg_test_desc *desc,/* Please keep this list sorted by algorithm name. */ static const struct alg_test_desc alg_test_descs[] = { { .alg ="adiantum(xchacha12,aes)" , - .generic_driver ="adiantum(xchacha12-generic,aes-generic,nhpoly1305-generic)" , + .generic_driver ="adiantum(xchacha12-lib,aes-generic,nhpoly1305-generic)" , .test = alg_test_skcipher, .suite = { .cipher =__VECS (adiantum_xchacha12_aes_tv_template) }, }, { .alg ="adiantum(xchacha20,aes)" , - .generic_driver ="adiantum(xchacha20-generic,aes-generic,nhpoly1305-generic)" , + .generic_driver ="adiantum(xchacha20-lib,aes-generic,nhpoly1305-generic)" , .test = alg_test_skcipher, .suite = { .cipher =__VECS (adiantum_xchacha20_aes_tv_template) }, }, {
@@ -4483,10 +4483,11 @@ static const struct alg_test_desc alg_test_descs[] = { .einval_allowed = 1, } } }, { .alg = "chacha20", + .generic_driver = "chacha20-lib", .test = alg_test_skcipher, .suite = { .cipher =__VECS (chacha20_tv_template) }, }, {
@@ -5418,16 +5419,18 @@ static const struct alg_test_desc alg_test_descs[] = { .aad_iv = 1, } } }, { .alg = "rfc7539(chacha20,poly1305)", + .generic_driver = "rfc7539(chacha20-lib,poly1305-generic)", .test = alg_test_aead, .suite = { .aead = __VECS(rfc7539_tv_template) } }, { .alg = "rfc7539esp(chacha20,poly1305)", + .generic_driver = "rfc7539esp(chacha20-lib,poly1305-generic)", .test = alg_test_aead, .suite = { .aead = { ____VECS(rfc7539esp_tv_template), .einval_allowed = 1,
@@ -5589,16 +5592,18 @@ static const struct alg_test_desc alg_test_descs[] = { .suite = { .hash = __VECS(sm4_xcbc128_tv_template) } }, { .alg = "xchacha12", + .generic_driver = "xchacha12-lib", .test = alg_test_skcipher, .suite = { .cipher =__VECS (xchacha12_tv_template) }, }, { .alg = "xchacha20", + .generic_driver = "xchacha20-lib", .test = alg_test_skcipher, .suite = { .cipher =__VECS (xchacha20_tv_template) }, }, {
Signature · 1 line
2.50.1
Eric Biggers
maintainerPATCH 3/12 lib/crypto: chacha: Remove unused function chacha_is_arch_optimized()
In reply to Eric Biggerschacha_is_arch_optimized() is no longer used, so remove it.
--- include/crypto/chacha.h | 9 --------- lib/crypto/arm/chacha-glue.c | 7 ------- lib/crypto/arm64/chacha-neon-glue.c | 6 ------ lib/crypto/mips/chacha-glue.c | 6 ------ lib/crypto/powerpc/chacha-p10-glue.c | 6 ------ lib/crypto/riscv/chacha-riscv64-glue.c | 6 ------ lib/crypto/s390/chacha-glue.c | 6 ------ lib/crypto/x86/chacha_glue.c | 6 ------ 8 files changed, 52 deletions(-)
include/crypto/chacha.h +0 -9
@@ -117,15 +117,6 @@ static inline void chacha20_crypt(struct chacha_state *state,static inline void chacha_zeroize_state (struct chacha_state *state) {memzero_explicit (state,sizeof (*state)); } -#if IS_ENABLED (CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA ) -bool chacha_is_arch_optimized (void ); -#else -static inline bool chacha_is_arch_optimized (void ) -{ -return false; -} -#endif -#endif /* _CRYPTO_CHACHA_H */
lib/crypto/arm/chacha-glue.c +0 -7
@@ -99,17 +99,10 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src, dst += todo; }while (bytes); }EXPORT_SYMBOL (chacha_crypt_arch); -bool chacha_is_arch_optimized (void ) -{ -/* We always can use at least the ARM scalar implementation. */ -return true; -} -EXPORT_SYMBOL (chacha_is_arch_optimized); -static int __init chacha_arm_mod_init (void ) {if (IS_ENABLED (CONFIG_KERNEL_MODE_NEON ) && (elf_hwcap &HWCAP_NEON )) {switch (read_cpuid_part ()) {case ARM_CPU_PART_CORTEX_A7 :
lib/crypto/arm64/chacha-neon-glue.c +0 -6
@@ -93,16 +93,10 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src, dst += todo; }while (bytes); }EXPORT_SYMBOL (chacha_crypt_arch); -bool chacha_is_arch_optimized (void ) -{ -return static_key_enabled (&have_neon); -} -EXPORT_SYMBOL (chacha_is_arch_optimized); -static int __init chacha_simd_mod_init (void ) {if (cpu_have_named_feature (ASIMD ))static_branch_enable (&have_neon);return 0 ;
lib/crypto/mips/chacha-glue.c +0 -6
@@ -16,14 +16,8 @@ EXPORT_SYMBOL(chacha_crypt_arch); asmlinkage void hchacha_block_arch(const struct chacha_state *state,u32 out[HCHACHA_OUT_WORDS ],int nrounds);EXPORT_SYMBOL (hchacha_block_arch); -bool chacha_is_arch_optimized (void ) -{ -return true; -} -EXPORT_SYMBOL (chacha_is_arch_optimized); -MODULE_DESCRIPTION ("ChaCha and HChaCha functions (MIPS optimized)" );MODULE_AUTHOR ("Ard Biesheuvel <[email protected]>" );MODULE_LICENSE ("GPL v2" );
lib/crypto/powerpc/chacha-p10-glue.c +0 -6
@@ -74,16 +74,10 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src, dst += todo; }while (bytes); }EXPORT_SYMBOL (chacha_crypt_arch); -bool chacha_is_arch_optimized (void ) -{ -return static_key_enabled (&have_p10); -} -EXPORT_SYMBOL (chacha_is_arch_optimized); -static int __init chacha_p10_init (void ) {if (cpu_has_feature (CPU_FTR_ARCH_31 ))static_branch_enable (&have_p10);return 0 ;
lib/crypto/riscv/chacha-riscv64-glue.c +0 -6
@@ -48,16 +48,10 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src, }kernel_vector_end (); }EXPORT_SYMBOL (chacha_crypt_arch); -bool chacha_is_arch_optimized (void ) -{ -return static_key_enabled (&use_zvkb); -} -EXPORT_SYMBOL (chacha_is_arch_optimized); -static int __init riscv64_chacha_mod_init (void ) {if (riscv_isa_extension_available (NULL ,ZVKB ) &&riscv_vector_vlen () >=128 )static_branch_enable (&use_zvkb);
lib/crypto/s390/chacha-glue.c +0 -6
@@ -45,13 +45,7 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,CHACHA_BLOCK_SIZE ; } }EXPORT_SYMBOL (chacha_crypt_arch); -bool chacha_is_arch_optimized (void ) -{ -return cpu_has_vx (); -} -EXPORT_SYMBOL (chacha_is_arch_optimized); -MODULE_DESCRIPTION ("ChaCha stream cipher (s390 optimized)" );MODULE_LICENSE ("GPL v2" );
lib/crypto/x86/chacha_glue.c +0 -6
@@ -158,16 +158,10 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src, dst += todo; }while (bytes); }EXPORT_SYMBOL (chacha_crypt_arch); -bool chacha_is_arch_optimized (void ) -{ -return static_key_enabled (&chacha_use_simd); -} -EXPORT_SYMBOL (chacha_is_arch_optimized); -static int __init chacha_simd_mod_init (void ) {if (!boot_cpu_has (X86_FEATURE_SSSE3 ))return 0 ;
Signature · 1 line
2.50.1
Eric Biggers
maintainerPATCH 4/12 lib/crypto: chacha: Rename chacha.c to chacha-block-generic.c
In reply to Eric BiggersRename chacha.c to chacha-block-generic.c to free up the name chacha.c for the high-level API entry points (chacha_crypt() and hchacha_block()), similar to the other algorithms.
---
lib/crypto/Makefile | 4 ++--
lib/crypto/{chacha.c => chacha-block-generic.c} | 0
2 files changed, 2 insertions(+), 2 deletions(-)
rename lib/crypto/{chacha.c => chacha-block-generic.c} (100%)lib/crypto/Makefile +2 -2
@@ -13,12 +13,12 @@ obj-$(CONFIG_KUNIT) += tests/ obj-$(CONFIG_CRYPTO_HASH_INFO) += hash_info.o obj-$(CONFIG_CRYPTO_LIB_UTILS) += libcryptoutils.o libcryptoutils-y := memneq.o utils.o -# chacha is used by the /dev/random driver which is always builtin -obj-y += chacha.o +# chacha20_block() is used by the /dev/random driver which is always builtin +obj-y += chacha-block-generic.o obj-$(CONFIG_CRYPTO_LIB_CHACHA_GENERIC) += libchacha.o obj-$(CONFIG_CRYPTO_LIB_AES) += libaes.o libaes-y := aes.o
lib/crypto/chacha-block-generic.c renamed from lib/crypto/chacha.c +0 -0
Signature · 1 line
2.50.1
Eric Biggers
maintainerPATCH 5/12 lib/crypto: chacha: Rename libchacha.c to chacha.c
In reply to Eric BiggersRename libchacha.c to chacha.c to make the naming consistent with other algorithms and allow additional source files to be added to the libchacha module. This file currently contains chacha_crypt_generic(), but it will soon be updated to contain chacha_crypt().
---
lib/crypto/Makefile | 1 +
lib/crypto/{libchacha.c => chacha.c} | 0
2 files changed, 1 insertion(+)
rename lib/crypto/{libchacha.c => chacha.c} (100%)lib/crypto/Makefile +1 -0
@@ -16,10 +16,11 @@ obj-$(CONFIG_CRYPTO_LIB_UTILS) += libcryptoutils.o libcryptoutils-y := memneq.o utils.o # chacha20_block() is used by the /dev/random driver which is always builtin obj-y += chacha-block-generic.o obj-$(CONFIG_CRYPTO_LIB_CHACHA_GENERIC) += libchacha.o +libchacha-y := chacha.o obj-$(CONFIG_CRYPTO_LIB_AES) += libaes.o libaes-y := aes.o obj-$(CONFIG_CRYPTO_LIB_AESCFB) += libaescfb.o
lib/crypto/chacha.c renamed from lib/crypto/libchacha.c +0 -0
Signature · 1 line
2.50.1
Eric Biggers
maintainerPATCH 6/12 lib/crypto: chacha: Consolidate into single module
In reply to Eric BiggersConsolidate the ChaCha code into a single module (excluding chacha-block-generic.c which remains always built-in for random.c), similar to various other algorithms:
- Each arch now provides a header file lib/crypto/$(SRCARCH)/chacha.h, replacing lib/crypto/$(SRCARCH)/chacha*.c. The header defines chacha_crypt_arch() and hchacha_block_arch(). It is included by lib/crypto/chacha.c, and thus the code gets built into the single libchacha module, with improved inlining in some cases.
- Whether arch-optimized ChaCha is buildable is now controlled centrally by lib/crypto/Kconfig instead of by lib/crypto/$(SRCARCH)/Kconfig. The conditions for enabling it remain the same as before, and it remains enabled by default.
- Any additional arch-specific translation units for the optimized ChaCha code, such as assembly files, are now compiled by lib/crypto/Makefile instead of lib/crypto/$(SRCARCH)/Makefile.
This removes the last use for the Makefile and Kconfig files in the arm64, mips, powerpc, riscv, and s390 subdirectories of lib/crypto/. So also remove those files and the references to them.
---
include/crypto/chacha.h | 28 ++---------
lib/crypto/Kconfig | 47 +++++--------------
lib/crypto/Makefile | 43 +++++++++++++----
lib/crypto/arm/Kconfig | 5 --
lib/crypto/arm/Makefile | 4 --
lib/crypto/arm/{chacha-glue.c => chacha.h} | 28 +++--------
lib/crypto/arm64/Kconfig | 8 ----
lib/crypto/arm64/Makefile | 4 --
.../arm64/{chacha-neon-glue.c => chacha.h} | 26 +++-------
lib/crypto/chacha.c | 41 ++++++++++++++--
lib/crypto/mips/Kconfig | 7 ---
lib/crypto/mips/Makefile | 5 --
lib/crypto/mips/{chacha-glue.c => chacha.h} | 11 +----
lib/crypto/powerpc/Kconfig | 8 ----
lib/crypto/powerpc/Makefile | 4 --
.../powerpc/{chacha-p10-glue.c => chacha.h} | 30 +++---------
lib/crypto/riscv/Kconfig | 8 ----
lib/crypto/riscv/Makefile | 4 --
.../riscv/{chacha-riscv64-glue.c => chacha.h} | 30 +++---------
lib/crypto/s390/Kconfig | 7 ---
lib/crypto/s390/Makefile | 4 --
lib/crypto/s390/{chacha-glue.c => chacha.h} | 23 ++-------
lib/crypto/x86/Kconfig | 7 ---
lib/crypto/x86/Makefile | 3 --
lib/crypto/x86/{chacha_glue.c => chacha.h} | 30 ++++--------
25 files changed, 126 insertions(+), 289 deletions(-)
rename lib/crypto/arm/{chacha-glue.c => chacha.h} (79%)
delete mode 100644 lib/crypto/arm64/Kconfig
delete mode 100644 lib/crypto/arm64/Makefile
rename lib/crypto/arm64/{chacha-neon-glue.c => chacha.h} (78%)
delete mode 100644 lib/crypto/mips/Kconfig
delete mode 100644 lib/crypto/mips/Makefile
rename lib/crypto/mips/{chacha-glue.c => chacha.h} (55%)
delete mode 100644 lib/crypto/powerpc/Kconfig
delete mode 100644 lib/crypto/powerpc/Makefile
rename lib/crypto/powerpc/{chacha-p10-glue.c => chacha.h} (66%)
delete mode 100644 lib/crypto/riscv/Kconfig
delete mode 100644 lib/crypto/riscv/Makefile
rename lib/crypto/riscv/{chacha-riscv64-glue.c => chacha.h} (61%)
delete mode 100644 lib/crypto/s390/Kconfig
delete mode 100644 lib/crypto/s390/Makefile
rename lib/crypto/s390/{chacha-glue.c => chacha.h} (55%)
rename lib/crypto/x86/{chacha_glue.c => chacha.h} (87%)include/crypto/chacha.h +4 -24
@@ -43,23 +43,15 @@ static inline void chacha20_block(struct chacha_state *state, u8 out[CHACHA_BLOCK_SIZE]) {chacha_block_generic (state, out,20 ); } -void hchacha_block_arch (const struct chacha_state *state, -u32 out[HCHACHA_OUT_WORDS ],int nrounds);void hchacha_block_generic (const struct chacha_state *state,u32 out[HCHACHA_OUT_WORDS ],int nrounds); -static inline void hchacha_block (const struct chacha_state *state, -u32 out[HCHACHA_OUT_WORDS ],int nrounds) -{ -if (IS_ENABLED (CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA )) -hchacha_block_arch (state, out, nrounds); -else -hchacha_block_generic (state, out, nrounds); -} +void hchacha_block (const struct chacha_state *state, +u32 out[HCHACHA_OUT_WORDS ],int nrounds);enum chacha_constants {/* expand 32-byte k */ CHACHA_CONSTANT_EXPA =0x61707865U ,CHACHA_CONSTANT_ND_3 =0x3320646eU ,CHACHA_CONSTANT_2_BY =0x79622d32U ,
@@ -91,24 +83,12 @@ static inline void chacha_init(struct chacha_state *state, state->x[13 ] =get_unaligned_le32 (iv +4 ); state->x[14 ] =get_unaligned_le32 (iv +8 ); state->x[15 ] =get_unaligned_le32 (iv +12 ); } -void chacha_crypt_arch (struct chacha_state *state,u8 *dst,const u8 *src, -unsigned int bytes,int nrounds); -void chacha_crypt_generic (struct chacha_state *state,u8 *dst,const u8 *src, -unsigned int bytes,int nrounds); - -static inline void chacha_crypt (struct chacha_state *state, -u8 *dst,const u8 *src, -unsigned int bytes,int nrounds) -{ -if (IS_ENABLED (CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA )) -chacha_crypt_arch (state, dst, src, bytes, nrounds); -else -chacha_crypt_generic (state, dst, src, bytes, nrounds); -} +void chacha_crypt (struct chacha_state *state,u8 *dst,const u8 *src, +unsigned int bytes,int nrounds);static inline void chacha20_crypt (struct chacha_state *state,u8 *dst,const u8 *src,unsigned int bytes) {chacha_crypt (state, dst, src, bytes,20 );
lib/crypto/Kconfig +13 -34
71 lines not on this page · raw message
lib/crypto/Makefile +33 -10
73 lines not on this page · raw message
lib/crypto/arm/Kconfig +0 -5
@@ -10,10 +10,5 @@ config CRYPTO_BLAKE2S_ARM This is faster than the generic implementations of BLAKE2s and BLAKE2b, but slower than the NEON implementation of BLAKE2b. There is no NEON implementation of BLAKE2s, since NEON doesn't really help with it. - -config CRYPTO_CHACHA20_NEON - tristate - default CRYPTO_LIB_CHACHA - select CRYPTO_ARCH_HAVE_LIB_CHACHA
lib/crypto/arm/Makefile +0 -4
@@ -1,8 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_CRYPTO_BLAKE2S_ARM) += libblake2s-arm.o libblake2s-arm-y := blake2s-core.o blake2s-glue.o - -obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha-neon.o -chacha-neon-y := chacha-scalar-core.o chacha-glue.o -chacha-neon-$(CONFIG_KERNEL_MODE_NEON) += chacha-neon-core.o
lib/crypto/arm/chacha.h renamed from lib/crypto/arm/chacha-glue.c +7 -21
@@ -1,18 +1,16 @@ -// SPDX-License-Identifier: GPL-2.0 +/* SPDX-License-Identifier: GPL-2.0 */ /* * ChaCha and HChaCha functions (ARM optimized) * * Copyright (C) 2016-2019 Linaro, Ltd. <[email protected]> * Copyright (C) 2015 Martin Willi */ -#include <crypto/chacha.h> #include <crypto/internal/simd.h> #include <linux/jump_label.h> #include <linux/kernel.h> -#include <linux/module.h> #include <asm/cputype.h> #include <asm/hwcap.h> #include <asm/neon.h> #include <asm/simd.h>
@@ -62,25 +60,24 @@ static void chacha_doneon(struct chacha_state *state, u8 *dst, const u8 *src,memcpy (dst, buf, bytes); state->x[12 ]++; } } -void hchacha_block_arch (const struct chacha_state *state, -u32 out[HCHACHA_OUT_WORDS ],int nrounds) +static void hchacha_block_arch (const struct chacha_state *state, +u32 out[HCHACHA_OUT_WORDS ],int nrounds) {if (!IS_ENABLED (CONFIG_KERNEL_MODE_NEON ) || !neon_usable ()) {hchacha_block_arm (state, out, nrounds); }else {kernel_neon_begin ();hchacha_block_neon (state, out, nrounds);kernel_neon_end (); } } -EXPORT_SYMBOL (hchacha_block_arch); -void chacha_crypt_arch (struct chacha_state *state,u8 *dst,const u8 *src, -unsigned int bytes,int nrounds) +static void chacha_crypt_arch (struct chacha_state *state,u8 *dst, +const u8 *src,unsigned int bytes,int nrounds) {if (!IS_ENABLED (CONFIG_KERNEL_MODE_NEON ) || !neon_usable () || bytes <=CHACHA_BLOCK_SIZE ) {chacha_doarm (dst, src, bytes, state, nrounds); state->x[12 ] +=DIV_ROUND_UP (bytes,CHACHA_BLOCK_SIZE );
@@ -97,13 +94,13 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src, bytes -= todo; src += todo; dst += todo; }while (bytes); } -EXPORT_SYMBOL (chacha_crypt_arch); -static int __init chacha_arm_mod_init (void ) +#define chacha_mod_init_arch chacha_mod_init_arch +static void chacha_mod_init_arch (void ) {if (IS_ENABLED (CONFIG_KERNEL_MODE_NEON ) && (elf_hwcap &HWCAP_NEON )) {switch (read_cpuid_part ()) {case ARM_CPU_PART_CORTEX_A7 :case ARM_CPU_PART_CORTEX_A5 :
@@ -115,17 +112,6 @@ static int __init chacha_arm_mod_init(void)break ;default :static_branch_enable (&use_neon); } } -return 0 ; } -subsys_initcall (chacha_arm_mod_init); - -static void __exit chacha_arm_mod_exit(void) -{ -} -module_exit (chacha_arm_mod_exit); - -MODULE_DESCRIPTION ("ChaCha and HChaCha functions (ARM optimized)" ); -MODULE_AUTHOR ("Ard Biesheuvel <[email protected]>" ); -MODULE_LICENSE ("GPL v2" );
lib/crypto/arm64/Kconfig deleted +0 -8
@@ -1,8 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -config CRYPTO_CHACHA20_NEON - tristate - depends on KERNEL_MODE_NEON - default CRYPTO_LIB_CHACHA - select CRYPTO_LIB_CHACHA_GENERIC - select CRYPTO_ARCH_HAVE_LIB_CHACHA
lib/crypto/arm64/Makefile deleted +0 -4
@@ -1,4 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha-neon.o -chacha-neon-y := chacha-neon-core.o chacha-neon-glue.o
lib/crypto/arm64/chacha.h renamed from lib/crypto/arm64/chacha-neon-glue.c +6 -20
@@ -16,15 +16,13 @@* it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. */ -#include <crypto/chacha.h> #include <crypto/internal/simd.h> #include <linux/jump_label.h> #include <linux/kernel.h> -#include <linux/module.h> #include <asm/hwcap.h> #include <asm/neon.h> #include <asm/simd.h>
@@ -59,25 +57,24 @@ static void chacha_doneon(struct chacha_state *state, u8 *dst, const u8 *src, dst += l; state->x[12 ] +=DIV_ROUND_UP (l,CHACHA_BLOCK_SIZE ); } } -void hchacha_block_arch (const struct chacha_state *state, -u32 out[HCHACHA_OUT_WORDS ],int nrounds) +static void hchacha_block_arch (const struct chacha_state *state, +u32 out[HCHACHA_OUT_WORDS ],int nrounds) {if (!static_branch_likely (&have_neon) || !crypto_simd_usable ()) {hchacha_block_generic (state, out, nrounds); }else {kernel_neon_begin ();hchacha_block_neon (state, out, nrounds);kernel_neon_end (); } } -EXPORT_SYMBOL (hchacha_block_arch); -void chacha_crypt_arch (struct chacha_state *state,u8 *dst,const u8 *src, -unsigned int bytes,int nrounds) +static void chacha_crypt_arch (struct chacha_state *state,u8 *dst, +const u8 *src,unsigned int bytes,int nrounds) {if (!static_branch_likely (&have_neon) || bytes <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return chacha_crypt_generic (state, dst, src, bytes, nrounds);
@@ -91,23 +88,12 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src, bytes -= todo; src += todo; dst += todo; }while (bytes); } -EXPORT_SYMBOL (chacha_crypt_arch); -static int __init chacha_simd_mod_init (void ) +#define chacha_mod_init_arch chacha_mod_init_arch +static void chacha_mod_init_arch (void ) {if (cpu_have_named_feature (ASIMD ))static_branch_enable (&have_neon); -return 0 ; } -subsys_initcall (chacha_simd_mod_init); - -static void __exit chacha_simd_mod_exit(void) -{ -} -module_exit (chacha_simd_mod_exit); - -MODULE_DESCRIPTION ("ChaCha and HChaCha functions (ARM64 optimized)" ); -MODULE_AUTHOR ("Ard Biesheuvel <[email protected]>" ); -MODULE_LICENSE ("GPL v2" );
lib/crypto/chacha.c +38 -3
59 lines not on this page · raw message
lib/crypto/mips/Kconfig deleted +0 -7
@@ -1,7 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -config CRYPTO_CHACHA_MIPS - tristate - depends on CPU_MIPS32_R2 - default CRYPTO_LIB_CHACHA - select CRYPTO_ARCH_HAVE_LIB_CHACHA
lib/crypto/mips/Makefile deleted +0 -5
@@ -1,5 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -obj-$(CONFIG_CRYPTO_CHACHA_MIPS) += chacha-mips.o -chacha-mips-y := chacha-core.o chacha-glue.o -AFLAGS_chacha-core.o += -O2 # needed to fill branch delay slots
lib/crypto/mips/chacha.h renamed from lib/crypto/mips/chacha-glue.c +1 -10
@@ -1,23 +1,14 @@ -// SPDX-License-Identifier: GPL-2.0 +/* SPDX-License-Identifier: GPL-2.0 */ /* * ChaCha and HChaCha functions (MIPS optimized) * * Copyright (C) 2019 Linaro, Ltd. <[email protected]> */ -#include <crypto/chacha.h> #include <linux/kernel.h> -#include <linux/module.h> asmlinkage void chacha_crypt_arch(struct chacha_state *state,u8 *dst,const u8 *src,unsigned int bytes,int nrounds); -EXPORT_SYMBOL (chacha_crypt_arch); - asmlinkage void hchacha_block_arch(const struct chacha_state *state,u32 out[HCHACHA_OUT_WORDS ],int nrounds); -EXPORT_SYMBOL (hchacha_block_arch); - -MODULE_DESCRIPTION ("ChaCha and HChaCha functions (MIPS optimized)" ); -MODULE_AUTHOR ("Ard Biesheuvel <[email protected]>" ); -MODULE_LICENSE ("GPL v2" );
lib/crypto/powerpc/Kconfig deleted +0 -8
@@ -1,8 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -config CRYPTO_CHACHA20_P10 - tristate - depends on PPC64 && CPU_LITTLE_ENDIAN && VSX - default CRYPTO_LIB_CHACHA - select CRYPTO_LIB_CHACHA_GENERIC - select CRYPTO_ARCH_HAVE_LIB_CHACHA
lib/crypto/powerpc/Makefile deleted +0 -4
@@ -1,4 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -obj-$(CONFIG_CRYPTO_CHACHA20_P10) += chacha-p10-crypto.o -chacha-p10-crypto-y := chacha-p10-glue.o chacha-p10le-8x.o
lib/crypto/powerpc/chacha.h renamed from lib/crypto/powerpc/chacha-p10-glue.c +6 -24
@@ -1,16 +1,14 @@ -// SPDX-License-Identifier: GPL-2.0-or-later +/* SPDX-License-Identifier: GPL-2.0-or-later */ /* * ChaCha stream cipher (P10 accelerated) * * Copyright 2023- IBM Corp. All rights reserved. */ -#include <crypto/chacha.h> #include <crypto/internal/simd.h> #include <linux/kernel.h> -#include <linux/module.h> #include <linux/cpufeature.h> #include <linux/sizes.h> #include <asm/simd.h> #include <asm/switch_to.h>
@@ -46,19 +44,14 @@ static void chacha_p10_do_8x(struct chacha_state *state, u8 *dst, const u8 *src,if (bytes >0 )chacha_crypt_generic (state, dst, src, bytes, nrounds); } -void hchacha_block_arch (const struct chacha_state *state, -u32 out[HCHACHA_OUT_WORDS ],int nrounds) -{ -hchacha_block_generic (state, out, nrounds); -} -EXPORT_SYMBOL (hchacha_block_arch); +#define hchacha_block_arch hchacha_block_generic/* not implemented yet */ -void chacha_crypt_arch (struct chacha_state *state,u8 *dst,const u8 *src, -unsigned int bytes,int nrounds) +static void chacha_crypt_arch (struct chacha_state *state,u8 *dst, +const u8 *src,unsigned int bytes,int nrounds) {if (!static_branch_likely (&have_p10) || bytes <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return chacha_crypt_generic (state, dst, src, bytes, nrounds);
@@ -72,23 +65,12 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src, bytes -= todo; src += todo; dst += todo; }while (bytes); } -EXPORT_SYMBOL (chacha_crypt_arch); -static int __init chacha_p10_init (void ) +#define chacha_mod_init_arch chacha_mod_init_arch +static void chacha_mod_init_arch (void ) {if (cpu_has_feature (CPU_FTR_ARCH_31 ))static_branch_enable (&have_p10); -return 0 ; -} -subsys_initcall (chacha_p10_init); - -static void __exit chacha_p10_exit(void) -{ } -module_exit (chacha_p10_exit); - -MODULE_DESCRIPTION ("ChaCha stream cipher (P10 accelerated)" ); -MODULE_AUTHOR ("Danny Tsen <[email protected]>" ); -MODULE_LICENSE ("GPL v2" );
lib/crypto/riscv/Kconfig deleted +0 -8
@@ -1,8 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -config CRYPTO_CHACHA_RISCV64 - tristate - depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO - default CRYPTO_LIB_CHACHA - select CRYPTO_ARCH_HAVE_LIB_CHACHA - select CRYPTO_LIB_CHACHA_GENERIC
lib/crypto/riscv/Makefile deleted +0 -4
@@ -1,4 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -obj-$(CONFIG_CRYPTO_CHACHA_RISCV64) += chacha-riscv64.o -chacha-riscv64-y := chacha-riscv64-glue.o chacha-riscv64-zvkb.o
lib/crypto/riscv/chacha.h renamed from lib/crypto/riscv/chacha-riscv64-glue.c +6 -24
@@ -1,34 +1,27 @@ -// SPDX-License-Identifier: GPL-2.0-only +/* SPDX-License-Identifier: GPL-2.0-only */ /* * ChaCha stream cipher (RISC-V optimized) * * Copyright (C) 2023 SiFive, Inc. * Author: Jerry Shih <[email protected]> */ #include <asm/simd.h> #include <asm/vector.h> -#include <crypto/chacha.h> #include <crypto/internal/simd.h> #include <linux/linkage.h> -#include <linux/module.h> static __ro_after_init DEFINE_STATIC_KEY_FALSE (use_zvkb); asmlinkage void chacha_zvkb(struct chacha_state *state, const u8 *in, u8 *out,size_t nblocks,int nrounds); -void hchacha_block_arch (const struct chacha_state *state, -u32 out[HCHACHA_OUT_WORDS ],int nrounds) -{ -hchacha_block_generic (state, out, nrounds); -} -EXPORT_SYMBOL (hchacha_block_arch); +#define hchacha_block_arch hchacha_block_generic/* not implemented yet */ -void chacha_crypt_arch (struct chacha_state *state,u8 *dst,const u8 *src, -unsigned int bytes,int nrounds) +static void chacha_crypt_arch (struct chacha_state *state,u8 *dst, +const u8 *src,unsigned int bytes,int nrounds) {u8 block_buffer[CHACHA_BLOCK_SIZE ];unsigned int full_blocks = bytes /CHACHA_BLOCK_SIZE ;unsigned int tail_bytes = bytes %CHACHA_BLOCK_SIZE ;
@@ -46,24 +39,13 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,chacha_zvkb (state, block_buffer, block_buffer,1 , nrounds);memcpy (dst, block_buffer, tail_bytes); }kernel_vector_end (); } -EXPORT_SYMBOL (chacha_crypt_arch); -static int __init riscv64_chacha_mod_init (void ) +#define chacha_mod_init_arch chacha_mod_init_arch +static void chacha_mod_init_arch (void ) {if (riscv_isa_extension_available (NULL ,ZVKB ) &&riscv_vector_vlen () >=128 )static_branch_enable (&use_zvkb); -return 0 ; -} -subsys_initcall (riscv64_chacha_mod_init); - -static void __exit riscv64_chacha_mod_exit(void) -{ } -module_exit (riscv64_chacha_mod_exit); - -MODULE_DESCRIPTION ("ChaCha stream cipher (RISC-V optimized)" ); -MODULE_AUTHOR ("Jerry Shih <[email protected]>" ); -MODULE_LICENSE ("GPL" );
lib/crypto/s390/Kconfig deleted +0 -7
@@ -1,7 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -config CRYPTO_CHACHA_S390 - tristate - default CRYPTO_LIB_CHACHA - select CRYPTO_LIB_CHACHA_GENERIC - select CRYPTO_ARCH_HAVE_LIB_CHACHA
lib/crypto/s390/Makefile deleted +0 -4
@@ -1,4 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -obj-$(CONFIG_CRYPTO_CHACHA_S390) += chacha_s390.o -chacha_s390-y := chacha-glue.o chacha-s390.o
lib/crypto/s390/chacha.h renamed from lib/crypto/s390/chacha-glue.c +4 -19
@@ -1,34 +1,23 @@ -// SPDX-License-Identifier: GPL-2.0 +/* SPDX-License-Identifier: GPL-2.0 */ /* * ChaCha stream cipher (s390 optimized) * * Copyright IBM Corp. 2021 */ -#define KMSG_COMPONENT "chacha_s390" -#define pr_fmt (fmt) KMSG_COMPONENT ": " fmt - -#include <crypto/chacha.h> #include <linux/cpufeature.h> #include <linux/export.h> #include <linux/kernel.h> -#include <linux/module.h> #include <linux/sizes.h> #include <asm/fpu.h> #include "chacha-s390.h" -void hchacha_block_arch (const struct chacha_state *state, -u32 out[HCHACHA_OUT_WORDS ],int nrounds) -{ -/* TODO: implement hchacha_block_arch() in assembly */ -hchacha_block_generic (state, out, nrounds); -} -EXPORT_SYMBOL (hchacha_block_arch); +#define hchacha_block_arch hchacha_block_generic/* not implemented yet */ -void chacha_crypt_arch (struct chacha_state *state,u8 *dst,const u8 *src, -unsigned int bytes,int nrounds) +static void chacha_crypt_arch (struct chacha_state *state,u8 *dst, +const u8 *src,unsigned int bytes,int nrounds) {/* s390 chacha20 implementation has 20 rounds hard-coded, * it cannot handle a block of data or less, but otherwise * it can handle data of arbitrary size */
@@ -43,9 +32,5 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src, state->x[12 ] +=round_up (bytes,CHACHA_BLOCK_SIZE ) /CHACHA_BLOCK_SIZE ; } } -EXPORT_SYMBOL (chacha_crypt_arch); - -MODULE_DESCRIPTION ("ChaCha stream cipher (s390 optimized)" ); -MODULE_LICENSE ("GPL v2" );
lib/crypto/x86/Kconfig +0 -7
@@ -9,12 +9,5 @@ config CRYPTO_BLAKE2S_X86 BLAKE2s cryptographic hash function (RFC 7693) Architecture: x86_64 using: - SSSE3 (Supplemental SSE3) - AVX-512 (Advanced Vector Extensions-512) - -config CRYPTO_CHACHA20_X86_64 - tristate - depends on 64BIT - default CRYPTO_LIB_CHACHA - select CRYPTO_LIB_CHACHA_GENERIC - select CRYPTO_ARCH_HAVE_LIB_CHACHA
lib/crypto/x86/Makefile +0 -3
@@ -1,7 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_CRYPTO_BLAKE2S_X86) += libblake2s-x86_64.o libblake2s-x86_64-y := blake2s-core.o blake2s-glue.o - -obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha-x86_64.o -chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha-avx512vl-x86_64.o chacha_glue.o
lib/crypto/x86/chacha.h renamed from lib/crypto/x86/chacha_glue.c +8 -22
@@ -1,17 +1,15 @@ -// SPDX-License-Identifier: GPL-2.0-or-later +/* SPDX-License-Identifier: GPL-2.0-or-later */ /* * ChaCha and HChaCha functions (x86_64 optimized) * * Copyright (C) 2015 Martin Willi */ #include <asm/simd.h> -#include <crypto/chacha.h> #include <linux/jump_label.h> #include <linux/kernel.h> -#include <linux/module.h> #include <linux/sizes.h> asmlinkage void chacha_block_xor_ssse3(const struct chacha_state *state,u8 *dst,const u8 *src,unsigned int len,int nrounds);
@@ -124,25 +122,24 @@ static void chacha_dosimd(struct chacha_state *state, u8 *dst, const u8 *src,chacha_block_xor_ssse3 (state, dst, src, bytes, nrounds); state->x[12 ]++; } } -void hchacha_block_arch (const struct chacha_state *state, -u32 out[HCHACHA_OUT_WORDS ],int nrounds) +static void hchacha_block_arch (const struct chacha_state *state, +u32 out[HCHACHA_OUT_WORDS ],int nrounds) {if (!static_branch_likely (&chacha_use_simd)) {hchacha_block_generic (state, out, nrounds); }else {kernel_fpu_begin ();hchacha_block_ssse3 (state, out, nrounds);kernel_fpu_end (); } } -EXPORT_SYMBOL (hchacha_block_arch); -void chacha_crypt_arch (struct chacha_state *state,u8 *dst,const u8 *src, -unsigned int bytes,int nrounds) +static void chacha_crypt_arch (struct chacha_state *state,u8 *dst, +const u8 *src,unsigned int bytes,int nrounds) {if (!static_branch_likely (&chacha_use_simd) || bytes <=CHACHA_BLOCK_SIZE )return chacha_crypt_generic (state, dst, src, bytes, nrounds);
@@ -156,16 +153,16 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src, bytes -= todo; src += todo; dst += todo; }while (bytes); } -EXPORT_SYMBOL (chacha_crypt_arch); -static int __init chacha_simd_mod_init (void ) +#define chacha_mod_init_arch chacha_mod_init_arch +static void chacha_mod_init_arch (void ) {if (!boot_cpu_has (X86_FEATURE_SSSE3 )) -return 0 ; +return ;static_branch_enable (&chacha_use_simd); if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_AVX2) &&
@@ -174,17 +171,6 @@ static int __init chacha_simd_mod_init(void)if (boot_cpu_has (X86_FEATURE_AVX512VL ) &&boot_cpu_has (X86_FEATURE_AVX512BW ))/* kmovq */ static_branch_enable (&chacha_use_avx512vl); } -return 0 ; } -subsys_initcall (chacha_simd_mod_init); - -static void __exit chacha_simd_mod_exit(void) -{ -} -module_exit (chacha_simd_mod_exit); - -MODULE_LICENSE ("GPL" ); -MODULE_AUTHOR ("Martin Willi <[email protected]>" ); -MODULE_DESCRIPTION ("ChaCha and HChaCha functions (x86_64 optimized)" );
Signature · 1 line
2.50.1
Eric Biggers
maintainerPATCH 7/12 lib/crypto: x86/blake2s: Reduce size of BLAKE2S_SIGMA2
In reply to Eric BiggersSave 480 bytes of .rodata by replacing the .long constants with .bytes, and using the vpmovzxbd instruction to expand them.
Also update the code to do the loads before incrementing %rax rather than after. This avoids the need for the first load to use an offset.
--- lib/crypto/x86/blake2s-core.S | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
lib/crypto/x86/blake2s-core.S +14 -14
@@ -27,23 +27,23 @@ SIGMA: .byte 2, 6, 0, 8, 12, 10, 11, 3, 1, 4, 7, 15, 9, 13, 5, 14 .byte 12, 1, 14, 4, 5, 15, 13, 10, 8, 0, 6, 9, 11, 7, 3, 2 .byte 13, 7, 12, 3, 11, 14, 1, 9, 2, 5, 15, 8, 10, 0, 4, 6 .byte 6, 14, 11, 0, 15, 9, 3, 8, 10, 12, 13, 1, 5, 2, 7, 4 .byte 10, 8, 7, 1, 2, 4, 6, 5, 13, 15, 9, 3, 0, 11, 14, 12 -.section .rodata.cst64.BLAKE2S_SIGMA2, "aM", @progbits, 640 +.section .rodata.cst64.BLAKE2S_SIGMA2, "aM", @progbits, 160 .align 64 SIGMA2: -.long 0, 2, 4, 6, 1, 3, 5, 7, 14, 8, 10, 12, 15, 9, 11, 13 -.long 8, 2, 13, 15, 10, 9, 12, 3, 6, 4, 0, 14, 5, 11, 1, 7 -.long 11, 13, 8, 6, 5, 10, 14, 3, 2, 4, 12, 15, 1, 0, 7, 9 -.long 11, 10, 7, 0, 8, 15, 1, 13, 3, 6, 2, 12, 4, 14, 9, 5 -.long 4, 10, 9, 14, 15, 0, 11, 8, 1, 7, 3, 13, 2, 5, 6, 12 -.long 2, 11, 4, 15, 14, 3, 10, 8, 13, 6, 5, 7, 0, 12, 1, 9 -.long 4, 8, 15, 9, 14, 11, 13, 5, 3, 2, 1, 12, 6, 10, 7, 0 -.long 6, 13, 0, 14, 12, 2, 1, 11, 15, 4, 5, 8, 7, 9, 3, 10 -.long 15, 5, 4, 13, 10, 7, 3, 11, 12, 2, 0, 6, 9, 8, 1, 14 -.long 8, 7, 14, 11, 13, 15, 0, 12, 10, 4, 5, 6, 3, 2, 1, 9 +.byte 0, 2, 4, 6, 1, 3, 5, 7, 14, 8, 10, 12, 15, 9, 11, 13 +.byte 8, 2, 13, 15, 10, 9, 12, 3, 6, 4, 0, 14, 5, 11, 1, 7 +.byte 11, 13, 8, 6, 5, 10, 14, 3, 2, 4, 12, 15, 1, 0, 7, 9 +.byte 11, 10, 7, 0, 8, 15, 1, 13, 3, 6, 2, 12, 4, 14, 9, 5 +.byte 4, 10, 9, 14, 15, 0, 11, 8, 1, 7, 3, 13, 2, 5, 6, 12 +.byte 2, 11, 4, 15, 14, 3, 10, 8, 13, 6, 5, 7, 0, 12, 1, 9 +.byte 4, 8, 15, 9, 14, 11, 13, 5, 3, 2, 1, 12, 6, 10, 7, 0 +.byte 6, 13, 0, 14, 12, 2, 1, 11, 15, 4, 5, 8, 7, 9, 3, 10 +.byte 15, 5, 4, 13, 10, 7, 3, 11, 12, 2, 0, 6, 9, 8, 1, 14 +.byte 8, 7, 14, 11, 13, 15, 0, 12, 10, 4, 5, 6, 3, 2, 1, 9 .text SYM_FUNC_START(blake2s_compress_ssse3) testq %rdx,%rdx je .Lendofloop
@@ -191,13 +191,13 @@ SYM_FUNC_START(blake2s_compress_avx512) vmovdqu 0x20(%rsi),%ymm7 addq $0x40,%rsi leaq SIGMA2(%rip),%rax movb $0xa,%cl .Lblake2s_compress_avx512_roundloop: - addq $0x40,%rax - vmovdqa -0x40(%rax),%ymm8 - vmovdqa -0x20(%rax),%ymm9 + vpmovzxbd (%rax),%ymm8 + vpmovzxbd 0x8(%rax),%ymm9 + addq $0x10,%rax vpermi2d %ymm7,%ymm6,%ymm8 vpermi2d %ymm7,%ymm6,%ymm9 vmovdqa %ymm8,%ymm6 vmovdqa %ymm9,%ymm7 vpaddd %xmm8,%xmm0,%xmm0
Signature · 1 line
2.50.1
Eric Biggers
maintainerPATCH 8/12 lib/crypto: blake2s: Remove obsolete self-test
In reply to Eric BiggersRemove the original BLAKE2s self-test, since it will be superseded by blake2s_kunit.
--- include/crypto/internal/blake2s.h | 2 - lib/crypto/Makefile | 1 - lib/crypto/blake2s-selftest.c | 651 ------------------------------ lib/crypto/blake2s.c | 10 - 4 files changed, 664 deletions(-) delete mode 100644 lib/crypto/blake2s-selftest.c
include/crypto/internal/blake2s.h +0 -2
@@ -14,8 +14,6 @@ void blake2s_compress_generic(struct blake2s_state *state, const u8 *block, size_t nblocks, const u32 inc);void blake2s_compress (struct blake2s_state *state,const u8 *block,size_t nblocks,const u32 inc); -bool blake2s_selftest (void ); -#endif /* _CRYPTO_INTERNAL_BLAKE2S_H */
lib/crypto/Makefile +0 -1
@@ -31,11 +31,10 @@ obj-$(CONFIG_CRYPTO_LIB_GF128MUL) += gf128mul.o # blake2s is used by the /dev/random driver which is always builtin obj-y += libblake2s.o libblake2s-y := blake2s.o libblake2s-$(CONFIG_CRYPTO_LIB_BLAKE2S_GENERIC) += blake2s-generic.o -libblake2s-$(CONFIG_CRYPTO_SELFTESTS) += blake2s-selftest.o ################################################################################ # chacha20_block() is used by the /dev/random driver which is always builtin obj-y += chacha-block-generic.o
lib/crypto/blake2s-selftest.c deleted +0 -651
651 lines not on this page · raw message
lib/crypto/blake2s.c +0 -10
@@ -9,11 +9,10 @@*/ #include <crypto/internal/blake2s.h> #include <linux/bug.h> #include <linux/export.h> -#include <linux/init.h> #include <linux/kernel.h> #include <linux/module.h> #include <linux/string.h> #include <linux/types.h>
@@ -57,16 +56,7 @@ void blake2s_final(struct blake2s_state *state, u8 *out)memcpy (out, state->h, state->outlen);memzero_explicit (state,sizeof (*state)); }EXPORT_SYMBOL (blake2s_final); -static int __init blake2s_mod_init (void ) -{ -if (IS_ENABLED (CONFIG_CRYPTO_SELFTESTS ) && -WARN_ON (!blake2s_selftest ())) -return -ENODEV ; -return 0 ; -} - -module_init (blake2s_mod_init);MODULE_DESCRIPTION ("BLAKE2s hash function" );MODULE_AUTHOR ("Jason A. Donenfeld <[email protected]>" );
Signature · 1 line
2.50.1
Eric Biggers
maintainerPATCH 9/12 lib/crypto: blake2s: Always enable arch-optimized BLAKE2s code
In reply to Eric BiggersWhen support for a crypto algorithm is enabled, the arch-optimized implementation of that algorithm should be enabled too. We've learned this the hard way many times over the years: people regularly forget to enable the arch-optimized implementations of the crypto algorithms, resulting in significant performance being left on the table.
Currently, BLAKE2s support is always enabled ('obj-y'), since random.c uses it. Therefore, the arch-optimized BLAKE2s code, which exists for ARM and x86_64, should be always enabled too. Let's do that.
Note that the effect on kernel image size is very small and should not be a concern. On ARM, enabling CRYPTO_BLAKE2S_ARM actually *shrinks* the kernel size by about 1200 bytes, since the ARM-optimized blake2s_compress() completely replaces the generic blake2s_compress(). On x86_64, enabling CRYPTO_BLAKE2S_X86 increases the kernel size by about 1400 bytes, as the generic blake2s_compress() is still included as a fallback; however, for context, that is only about a quarter the size of the generic blake2s_compress(). The x86_64 optimized BLAKE2s code uses much less icache at runtime than the generic code.
--- lib/crypto/arm/Kconfig | 2 +- lib/crypto/x86/Kconfig | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
lib/crypto/arm/Kconfig +1 -1
@@ -1,9 +1,9 @@ # SPDX-License-Identifier: GPL-2.0-only config CRYPTO_BLAKE2S_ARM - bool "Hash functions: BLAKE2s" + def_bool y select CRYPTO_ARCH_HAVE_LIB_BLAKE2S help BLAKE2s cryptographic hash function (RFC 7693) Architecture: arm
lib/crypto/x86/Kconfig +1 -1
@@ -1,9 +1,9 @@ # SPDX-License-Identifier: GPL-2.0-only config CRYPTO_BLAKE2S_X86 - bool "Hash functions: BLAKE2s (SSSE3/AVX-512)" + def_bool y depends on 64BIT select CRYPTO_LIB_BLAKE2S_GENERIC select CRYPTO_ARCH_HAVE_LIB_BLAKE2S help BLAKE2s cryptographic hash function (RFC 7693)
Signature · 1 line
2.50.1
Re: [PATCH 09/12] lib/crypto: blake2s: Always enable arch-optimized BLAKE2s code
Quoted 3 lines · Eric Biggers
Currently, BLAKE2s support is always enabled ('obj-y'), since
random.cuses it. Therefore, the arch-optimized BLAKE2s code, which exists for ARM andx86_64, should be always enabled too.
Maybe a stupid question: what about ARM64? The current NEON implementation in kernel arch/arm/crypto/blake2s-core.S seems to be just for ARM.
While the upstream BLAKE2s with NEON is both for ARM and Aarch64 (ARM64):
Eric Biggers
maintainerRe: [PATCH 09/12] lib/crypto: blake2s: Always enable arch-optimized BLAKE2s code
Quoted 6 lines · Honza Fikar
Quoted 3 lines · Eric Biggers
Currently, BLAKE2s support is always enabled ('obj-y'), since
random.cuses it. Therefore, the arch-optimized BLAKE2s code, which exists for ARM andx86_64, should be always enabled too.
Quoted 7 lines
Maybe a stupid question: what about ARM64? The current NEON implementation in kernel
arch/arm/crypto/blake2s-core.Sseems to be just for ARM.While the upstream BLAKE2s with NEON is both for ARM and Aarch64 (ARM64):
There's no ARM64 optimized BLAKE2s code in the Linux kernel yet. If it's useful, someone would need to contribute it.
- Eric
Ard Biesheuvel
maintainerRe: [PATCH 09/12] lib/crypto: blake2s: Always enable arch-optimized BLAKE2s code
Quoted 4 lines · Eric Biggers
Quoted 2 lines · Honza Fikar
On Fri, Aug 29, 2025 at 2:54 PM Eric Biggers <[email protected]> wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Currently, BLAKE2s support is always enabled ('obj-y'), since
random.cuses it. Therefore, the arch-optimized BLAKE2s code, which exists for ARM andx86_64, should be always enabled too.Maybe a stupid question: what about ARM64? The current NEON implementation in kernel
arch/arm/crypto/blake2s-core.Sseems to be just for ARM.
That code is scalar not NEON, and is carefully tuned to make use of the ARM barrel shifter, which does not exist on arm64.
Quoted 7 lines
Quoted 3 lines
While the upstream BLAKE2s with NEON is both for ARM and Aarch64 (ARM64):
There's no ARM64 optimized BLAKE2s code in the Linux kernel yet. If it's useful, someone would need to contribute it.
NEON is cumbersome in the kernel so this only makes sense if it is substantially more performant, and I'm skeptical that this is the case, as you pointed out yourself in
commit 5172d322d34c30fb926b29aeb5a064e1fd8a5e13
Author: Eric Biggers <[email protected]> Date: Wed Dec 23 00:09:59 2020 -0800
crypto: arm/blake2s - add ARM scalar optimized BLAKE2s
Add an ARM scalar optimized implementation of BLAKE2s.
NEON isn't very useful for BLAKE2s because the BLAKE2s block size is too small for NEON to help. Each NEON instruction would depend on the previous one, resulting in poor performance.
Even if NEON code might be slightly faster on some cores, the fact that it is sensitive to micro-architectural details makes it less attractive.
Eric Biggers
maintainerRe: [PATCH 09/12] lib/crypto: blake2s: Always enable arch-optimized BLAKE2s code
Quoted 37 lines · Ard Biesheuvel
Quoted 4 lines · Eric Biggers
Quoted 2 lines · Honza Fikar
On Fri, Aug 29, 2025 at 2:54 PM Eric Biggers <[email protected]> wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Currently, BLAKE2s support is always enabled ('obj-y'), since
random.cuses it. Therefore, the arch-optimized BLAKE2s code, which exists for ARM andx86_64, should be always enabled too.Maybe a stupid question: what about ARM64? The current NEON implementation in kernel
arch/arm/crypto/blake2s-core.Sseems to be just for ARM.That code is scalar not NEON, and is carefully tuned to make use of the ARM barrel shifter, which does not exist on arm64.
Quoted 7 lines
Quoted 3 lines
While the upstream BLAKE2s with NEON is both for ARM and Aarch64 (ARM64):
There's no ARM64 optimized BLAKE2s code in the Linux kernel yet. If it's useful, someone would need to contribute it.
NEON is cumbersome in the kernel so this only makes sense if it is substantially more performant, and I'm skeptical that this is the case, as you pointed out yourself in
commit
5172d322d34c30fb926b29aeb5a064e1fd8a5e13Author: Eric Biggers <[email protected]> Date: Wed Dec 23 00:09:59 2020 -0800crypto: arm/blake2s - add ARM scalar optimized BLAKE2s
Add an ARM scalar optimized implementation of BLAKE2s.
Quoted 7 lines
NEON isn't very useful for BLAKE2s because the BLAKE2s block size is too small for NEON to help. Each NEON instruction would depend on the previous one, resulting in poor performance.
Even if NEON code might be slightly faster on some cores, the fact that it is sensitive to micro-architectural details makes it less attractive.
Yes, agreed: there isn't much opportunity for an ARM64 optimized BLAKE2s implementation to be faster than the generic C code.
- Eric
Eric Biggers
maintainerPATCH 10/12 lib/crypto: blake2s: Move generic code into blake2s.c
In reply to Eric BiggersMove blake2s_compress_generic() from blake2s-generic.c to blake2s.c.
For now it's still guarded by CONFIG_CRYPTO_LIB_BLAKE2S_GENERIC, but this prepares for changing it to a 'static __maybe_unused' function and just using the compiler to automatically decide its inclusion.
--- lib/crypto/Makefile | 1 - lib/crypto/blake2s-generic.c | 111 ----------------------------------- lib/crypto/blake2s.c | 94 +++++++++++++++++++++++++++++ 3 files changed, 94 insertions(+), 112 deletions(-) delete mode 100644 lib/crypto/blake2s-generic.c
lib/crypto/Makefile +0 -1
@@ -30,11 +30,10 @@ libarc4-y := arc4.o obj-$(CONFIG_CRYPTO_LIB_GF128MUL) += gf128mul.o # blake2s is used by the /dev/random driver which is always builtin obj-y += libblake2s.o libblake2s-y := blake2s.o -libblake2s-$(CONFIG_CRYPTO_LIB_BLAKE2S_GENERIC) += blake2s-generic.o ################################################################################ # chacha20_block() is used by the /dev/random driver which is always builtin obj-y += chacha-block-generic.o
lib/crypto/blake2s-generic.c deleted +0 -111
111 lines not on this page · raw message
lib/crypto/blake2s.c +94 -0
104 lines not on this page · raw message
Signature · 1 line
2.50.1
Eric Biggers
maintainerPATCH 11/12 lib/crypto: blake2s: Consolidate into single C translation unit
In reply to Eric BiggersAs was done with the other algorithms, reorganize the BLAKE2s code so that the generic implementation and the arch-specific "glue" code is consolidated into a single translation unit, so that the compiler will inline the functions and automatically decide whether to include the generic code in the resulting binary or not.
Similarly, also consolidate the build rules into lib/crypto/{Makefile,Kconfig}. This removes the last uses of lib/crypto/{arm,x86}/{Makefile,Kconfig}, so remove those too.
Don't keep the !KMSAN dependency. It was needed only for other algorithms such as ChaCha that initialize memory from assembly code.
---
include/crypto/internal/blake2s.h | 19 -------------
lib/crypto/Kconfig | 29 ++++----------------
lib/crypto/Makefile | 13 +++++----
lib/crypto/arm/Kconfig | 14 ----------
lib/crypto/arm/Makefile | 4 ---
lib/crypto/arm/blake2s-core.S | 5 +++-
lib/crypto/arm/blake2s-glue.c | 7 -----
lib/crypto/arm/blake2s.h | 5 ++++
lib/crypto/blake2s.c | 29 +++++++++++++-------
lib/crypto/x86/Kconfig | 13 ---------
lib/crypto/x86/Makefile | 4 ---
lib/crypto/x86/{blake2s-glue.c => blake2s.h} | 16 ++++-------
12 files changed, 47 insertions(+), 111 deletions(-)
delete mode 100644 include/crypto/internal/blake2s.h
delete mode 100644 lib/crypto/arm/Kconfig
delete mode 100644 lib/crypto/arm/Makefile
delete mode 100644 lib/crypto/arm/blake2s-glue.c
create mode 100644 lib/crypto/arm/blake2s.h
delete mode 100644 lib/crypto/x86/Kconfig
delete mode 100644 lib/crypto/x86/Makefile
rename lib/crypto/x86/{blake2s-glue.c => blake2s.h} (83%)include/crypto/internal/blake2s.h deleted +0 -19
@@ -1,19 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 OR MIT */ -/* -* Helper functions for BLAKE2s implementations. -* Keep this in sync with the corresponding BLAKE2b header. -*/ - -#ifndef _CRYPTO_INTERNAL_BLAKE2S_H -#define _CRYPTO_INTERNAL_BLAKE2S_H - -#include <crypto/blake2s.h> -#include <linux/string.h> - -void blake2s_compress_generic (struct blake2s_state *state,const u8 *block, -size_t nblocks,const u32 inc); - -void blake2s_compress (struct blake2s_state *state,const u8 *block, -size_t nblocks,const u32 inc); - -#endif /* _CRYPTO_INTERNAL_BLAKE2S_H */
lib/crypto/Kconfig +6 -23
@@ -26,25 +26,17 @@ config CRYPTO_LIB_ARC4 tristate config CRYPTO_LIB_GF128MUL tristate -config CRYPTO_ARCH_HAVE_LIB_BLAKE2S - bool - help - Declares whether the architecture provides an arch-specific - accelerated implementation of the Blake2s library interface, - either builtin or as a module. +# BLAKE2s support is always built-in, so there's no CRYPTO_LIB_BLAKE2S option. -config CRYPTO_LIB_BLAKE2S_GENERIC - def_bool !CRYPTO_ARCH_HAVE_LIB_BLAKE2S - help - This symbol can be depended upon by arch implementations of the - Blake2s library interface that require the generic code as a - fallback, e.g., for SIMD implementations. If no arch specific - implementation is enabled, this implementation serves the users - of CRYPTO_LIB_BLAKE2S. +config CRYPTO_LIB_BLAKE2S_ARCH + bool + depends on !UML + default y if ARM + default y if X86_64 config CRYPTO_LIB_CHACHA tristate select CRYPTO_LIB_UTILS help
@@ -206,15 +198,6 @@ config CRYPTO_LIB_SHA512_ARCH config CRYPTO_LIB_SM3 tristate source "lib/crypto/tests/Kconfig" -if !KMSAN # avoid false positives from assembly -if ARM -source "lib/crypto/arm/Kconfig" -endif -if X86 -source "lib/crypto/x86/Kconfig" -endif -endif - endmenu
lib/crypto/Makefile +8 -5
@@ -27,13 +27,19 @@ libaesgcm-y := aesgcm.o obj-$(CONFIG_CRYPTO_LIB_ARC4) += libarc4.o libarc4-y := arc4.o obj-$(CONFIG_CRYPTO_LIB_GF128MUL) += gf128mul.o +################################################################################ + # blake2s is used by the /dev/random driver which is always builtin -obj-y += libblake2s.o -libblake2s-y := blake2s.o +obj-y += blake2s.o +ifeq ($(CONFIG_CRYPTO_LIB_BLAKE2S_ARCH),y) +CFLAGS_blake2s.o += -I$(src)/$(SRCARCH) +obj-$(CONFIG_ARM) += arm/blake2s-core.o +obj-$(CONFIG_X86) += x86/blake2s-core.o +endif ################################################################################ # chacha20_block() is used by the /dev/random driver which is always builtin obj-y += chacha-block-generic.o
@@ -252,11 +258,8 @@ obj-$(CONFIG_MPILIB) += mpi/ obj-$(CONFIG_CRYPTO_SELFTESTS_FULL) += simd.o obj-$(CONFIG_CRYPTO_LIB_SM3) += libsm3.o libsm3-y := sm3.o -obj-$(CONFIG_ARM) += arm/ -obj-$(CONFIG_X86) += x86/ - # clean-files must be defined unconditionally clean-files += arm/sha256-core.S arm/sha512-core.S clean-files += arm64/sha256-core.S arm64/sha512-core.S
lib/crypto/arm/Kconfig deleted +0 -14
@@ -1,14 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -config CRYPTO_BLAKE2S_ARM - def_bool y - select CRYPTO_ARCH_HAVE_LIB_BLAKE2S - help - BLAKE2s cryptographic hash function (RFC 7693) - - Architecture: arm - - This is faster than the generic implementations of BLAKE2s and - BLAKE2b, but slower than the NEON implementation of BLAKE2b. - There is no NEON implementation of BLAKE2s, since NEON doesn't - really help with it.
lib/crypto/arm/Makefile deleted +0 -4
@@ -1,4 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -obj-$(CONFIG_CRYPTO_BLAKE2S_ARM) += libblake2s-arm.o -libblake2s-arm-y := blake2s-core.o blake2s-glue.o
lib/crypto/arm/blake2s-core.S +4 -1
@@ -1,8 +1,11 @@ /* SPDX-License-Identifier: GPL-2.0-or-later */ /* - * BLAKE2s digest algorithm, ARM scalar implementation + * BLAKE2s digest algorithm, ARM scalar implementation. This is faster + * than the generic implementations of BLAKE2s and BLAKE2b, but slower + * than the NEON implementation of BLAKE2b. There is no NEON + * implementation of BLAKE2s, since NEON doesn't really help with it. * * Copyright 2020 Google LLC * * Author: Eric Biggers <[email protected]> */
lib/crypto/arm/blake2s-glue.c deleted +0 -7
@@ -1,7 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later - -#include <crypto/internal/blake2s.h> -#include <linux/module.h> - -/* defined in blake2s-core.S */ -EXPORT_SYMBOL (blake2s_compress);
lib/crypto/arm/blake2s.h new file +5 -0
@@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +/* defined in blake2s-core.S */ +void blake2s_compress (struct blake2s_state *state,const u8 *block, +size_t nblocks,u32 inc);
lib/crypto/blake2s.c +19 -10
@@ -6,19 +6,18 @@* * Information: https://blake2.net/ * */ -#include <crypto/internal/blake2s.h> +#include <crypto/blake2s.h> #include <linux/bug.h> #include <linux/export.h> #include <linux/kernel.h> #include <linux/module.h> #include <linux/string.h> #include <linux/types.h> -#ifdef CONFIG_CRYPTO_LIB_BLAKE2S_GENERIC static const u8 blake2s_sigma[10 ][16 ] = { {0 ,1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 ,9 ,10 ,11 ,12 ,13 ,14 ,15 }, {14 ,10 ,4 ,8 ,9 ,15 ,13 ,6 ,1 ,12 ,0 ,2 ,11 ,7 ,5 ,3 }, {11 ,8 ,12 ,0 ,5 ,2 ,15 ,13 ,10 ,14 ,3 ,6 ,7 ,1 ,9 ,4 }, {7 ,9 ,3 ,1 ,13 ,12 ,11 ,14 ,2 ,6 ,5 ,10 ,4 ,0 ,15 ,8 },
@@ -35,16 +34,13 @@ static inline void blake2s_increment_counter(struct blake2s_state *state, { state->t[0 ] += inc; state->t[1 ] += (state->t[0 ] < inc); } -void blake2s_compress(struct blake2s_state *state, const u8 *block, - size_t nblocks, const u32 inc) - __weak __alias(blake2s_compress_generic); - -void blake2s_compress_generic (struct blake2s_state *state,const u8 *block, -size_t nblocks,const u32 inc) +static void __maybe_unused +blake2s_compress_generic (struct blake2s_state *state,const u8 *block, +size_t nblocks,const u32 inc) {u32 m[16 ];u32 v[16 ];int i;
@@ -105,12 +101,16 @@ void blake2s_compress_generic(struct blake2s_state *state, const u8 *block, block +=BLAKE2S_BLOCK_SIZE ; --nblocks; } } -EXPORT_SYMBOL (blake2s_compress_generic); -#endif /* CONFIG_CRYPTO_LIB_BLAKE2S_GENERIC */ + +#ifdef CONFIG_CRYPTO_LIB_BLAKE2S_ARCH +#include "blake2s.h" /* $(SRCARCH)/blake2s.h */ +#else +#define blake2s_compress blake2s_compress_generic +#endif static inline void blake2s_set_lastblock (struct blake2s_state *state) { state->f[0 ] =-1 ; }
@@ -150,7 +150,16 @@ void blake2s_final(struct blake2s_state *state, u8 *out)memcpy (out, state->h, state->outlen);memzero_explicit (state,sizeof (*state)); }EXPORT_SYMBOL (blake2s_final); +#ifdef blake2s_mod_init_arch +static int __init blake2s_mod_init (void ) +{ +blake2s_mod_init_arch (); +return 0 ; +} +subsys_initcall (blake2s_mod_init); +#endif +MODULE_DESCRIPTION ("BLAKE2s hash function" );MODULE_AUTHOR ("Jason A. Donenfeld <[email protected]>" );
lib/crypto/x86/Kconfig deleted +0 -13
@@ -1,13 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -config CRYPTO_BLAKE2S_X86 - def_bool y - depends on 64BIT - select CRYPTO_LIB_BLAKE2S_GENERIC - select CRYPTO_ARCH_HAVE_LIB_BLAKE2S - help - BLAKE2s cryptographic hash function (RFC 7693) - - Architecture: x86_64 using: - - SSSE3 (Supplemental SSE3) - - AVX-512 (Advanced Vector Extensions-512)
lib/crypto/x86/Makefile deleted +0 -4
@@ -1,4 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only - -obj-$(CONFIG_CRYPTO_BLAKE2S_X86) += libblake2s-x86_64.o -libblake2s-x86_64-y := blake2s-core.o blake2s-glue.o
lib/crypto/x86/blake2s.h renamed from lib/crypto/x86/blake2s-glue.c +5 -11
@@ -1,16 +1,14 @@ -// SPDX-License-Identifier: GPL-2.0 OR MIT +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ /* * Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. */ #include <asm/cpufeature.h> #include <asm/fpu/api.h> #include <asm/processor.h> #include <asm/simd.h> -#include <crypto/internal/blake2s.h> -#include <linux/init.h> #include <linux/jump_label.h> #include <linux/kernel.h> #include <linux/sizes.h> asmlinkage void blake2s_compress_ssse3(struct blake2s_state *state,
@@ -21,12 +19,12 @@ asmlinkage void blake2s_compress_avx512(struct blake2s_state *state, const u32 inc);static __ro_after_init DEFINE_STATIC_KEY_FALSE (blake2s_use_ssse3);static __ro_after_init DEFINE_STATIC_KEY_FALSE (blake2s_use_avx512); -void blake2s_compress (struct blake2s_state *state,const u8 *block, -size_t nblocks,const u32 inc) +static void blake2s_compress (struct blake2s_state *state,const u8 *block, +size_t nblocks,const u32 inc) {/* SIMD disables preemption, so relax after processing each page. */ BUILD_BUG_ON (SZ_4K /BLAKE2S_BLOCK_SIZE <8 );if (!static_branch_likely (&blake2s_use_ssse3) || !may_use_simd ()) {
@@ -47,13 +45,13 @@ void blake2s_compress(struct blake2s_state *state, const u8 *block, nblocks -= blocks; block += blocks *BLAKE2S_BLOCK_SIZE ; }while (nblocks); } -EXPORT_SYMBOL (blake2s_compress); -static int __init blake2s_mod_init (void ) +#define blake2s_mod_init_arch blake2s_mod_init_arch +static void blake2s_mod_init_arch (void ) {if (boot_cpu_has (X86_FEATURE_SSSE3 ))static_branch_enable (&blake2s_use_ssse3); if (boot_cpu_has(X86_FEATURE_AVX) &&
@@ -61,10 +59,6 @@ static int __init blake2s_mod_init(void) boot_cpu_has(X86_FEATURE_AVX512F) && boot_cpu_has(X86_FEATURE_AVX512VL) && cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM | XFEATURE_MASK_AVX512, NULL))static_branch_enable (&blake2s_use_avx512); - -return 0 ; } - -subsys_initcall (blake2s_mod_init);
Signature · 1 line
2.50.1
Eric Biggers
maintainerPATCH 12/12 lib/crypto: tests: Add KUnit tests for BLAKE2s
In reply to Eric BiggersAdd a KUnit test suite for BLAKE2s. Most of the core test logic is in the previously-added hash-test-template.h. This commit just adds the actual KUnit suite, commits the generated test vectors to the tree so that gen-hash-testvecs.py won't have to be run at build time, and adds a few BLAKE2s-specific test cases.
This is the replacement for blake2s-selftest, which an earlier commit removed. Improvements over blake2s-selftest include integration with KUnit, more comprehensive test cases, and support for benchmarking.
--- lib/crypto/tests/Kconfig | 10 ++ lib/crypto/tests/Makefile | 1 + lib/crypto/tests/blake2s-testvecs.h | 238 ++++++++++++++++++++++++++++ lib/crypto/tests/blake2s_kunit.c | 134 ++++++++++++++++ scripts/crypto/gen-hash-testvecs.py | 27 +++- 5 files changed, 407 insertions(+), 3 deletions(-) create mode 100644 lib/crypto/tests/blake2s-testvecs.h create mode 100644 lib/crypto/tests/blake2s_kunit.c
lib/crypto/tests/Kconfig +10 -0
@@ -1,7 +1,17 @@ # SPDX-License-Identifier: GPL-2.0-or-later +config CRYPTO_LIB_BLAKE2S_KUNIT_TEST + tristate "KUnit tests for BLAKE2s" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS || CRYPTO_SELFTESTS + select CRYPTO_LIB_BENCHMARK_VISIBLE + # No need to select CRYPTO_LIB_BLAKE2S here, as that option doesn't + # exist; the BLAKE2s code is always built-in for the /dev/random driver. + help + KUnit tests for the BLAKE2s cryptographic hash function. + config CRYPTO_LIB_MD5_KUNIT_TEST tristate "KUnit tests for MD5" if !KUNIT_ALL_TESTS depends on KUNIT default KUNIT_ALL_TESTS || CRYPTO_SELFTESTS select CRYPTO_LIB_BENCHMARK_VISIBLE
lib/crypto/tests/Makefile +1 -0
@@ -1,7 +1,8 @@ # SPDX-License-Identifier: GPL-2.0-or-later +obj-$(CONFIG_CRYPTO_LIB_BLAKE2S_KUNIT_TEST) += blake2s_kunit.o obj-$(CONFIG_CRYPTO_LIB_MD5_KUNIT_TEST) += md5_kunit.o obj-$(CONFIG_CRYPTO_LIB_POLY1305_KUNIT_TEST) += poly1305_kunit.o obj-$(CONFIG_CRYPTO_LIB_SHA1_KUNIT_TEST) += sha1_kunit.o obj-$(CONFIG_CRYPTO_LIB_SHA256_KUNIT_TEST) += sha224_kunit.o sha256_kunit.o obj-$(CONFIG_CRYPTO_LIB_SHA512_KUNIT_TEST) += sha384_kunit.o sha512_kunit.o
lib/crypto/tests/blake2s-testvecs.h new file +238 -0
238 lines not on this page · raw message
lib/crypto/tests/blake2s_kunit.c new file +134 -0
134 lines not on this page · raw message
scripts/crypto/gen-hash-testvecs.py +24 -3
@@ -82,15 +82,20 @@ def print_static_u8_array_definition(name, value): def print_c_struct_u8_array_field(name, value): print(f'\t\t.{name} = {{') print_bytes('\t\t\t', value, 8) print('\t\t},') +def alg_digest_size_const(alg): + if alg == 'blake2s': + return 'BLAKE2S_HASH_SIZE' + return f'{alg.upper()}_DIGEST_SIZE' + def gen_unkeyed_testvecs(alg): print('') print('static const struct {') print('\tsize_t data_len;') - print(f'\tu8 digest[{alg.upper()}_DIGEST_SIZE];') + print(f'\tu8 digest[{alg_digest_size_const(alg)}];') print('} hash_testvecs[] = {') for data_len in DATA_LENS: data = rand_bytes(data_len) print('\t{') print(f'\t\t.data_len = {data_len},')
@@ -101,11 +106,11 @@ def gen_unkeyed_testvecs(alg): data = rand_bytes(4096) ctx = hash_init(alg) for data_len in range(len(data) + 1): hash_update(ctx, compute_hash(alg, data[:data_len])) print_static_u8_array_definition( - f'hash_testvec_consolidated[{alg.upper()}_DIGEST_SIZE]', + f'hash_testvec_consolidated[{alg_digest_size_const(alg)}]', hash_final(ctx)) def gen_hmac_testvecs(alg): ctx = hmac.new(rand_bytes(32), digestmod=alg) data = rand_bytes(4096)
@@ -117,10 +122,24 @@ def gen_hmac_testvecs(alg): ctx.update(mac) print_static_u8_array_definition( f'hmac_testvec_consolidated[{alg.upper()}_DIGEST_SIZE]', ctx.digest()) +BLAKE2S_KEY_SIZE = 32 +BLAKE2S_HASH_SIZE = 32 + +def gen_additional_blake2s_testvecs(): + hashes = b'' + for key_len in range(BLAKE2S_KEY_SIZE + 1): + for out_len in range(1, BLAKE2S_HASH_SIZE + 1): + h = hashlib.blake2s(digest_size=out_len, key=rand_bytes(key_len)) + h.update(rand_bytes(100)) + hashes += h.digest() + print_static_u8_array_definition( + 'blake2s_keyed_testvec_consolidated[BLAKE2S_HASH_SIZE]', + compute_hash('blake2s', hashes)) + def gen_additional_poly1305_testvecs(): key = b'\xff' * POLY1305_KEY_SIZE data = b'' ctx = Poly1305(key) for _ in range(32):
@@ -139,9 +158,11 @@ if len(sys.argv) != 2: alg = sys.argv[1] print('/* SPDX-License-Identifier: GPL-2.0-or-later */') print(f'/* This file was generated by: {sys.argv[0]} {" ".join(sys.argv[1:])} */') gen_unkeyed_testvecs(alg) -if alg == 'poly1305': +if alg == 'blake2s': + gen_additional_blake2s_testvecs() +elif alg == 'poly1305': gen_additional_poly1305_testvecs() else: gen_hmac_testvecs(alg)
Signature · 1 line
2.50.1
Quoted 20 lines · Eric Biggers
This series is targeting libcrypto-next. It can also be retrieved from:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git chacha-blake2s-v1
This series consolidates how the ChaCha and BLAKE2s code is organized. This is essentially the same change that I made to the other algorithms, so this should be fairly boring by now.
These algorithms were the last two users of lib/crypto/$(SRCARCH)/{Makefile,Kconfig}. So this series removes all those files, finishing the transition to the centralized build process (at least for the algorithms supported by
lib/crypto/so far).This series also makes the arch-optimized BLAKE2s code start being enabled by default, again following the pattern of the other algorithms.
Finally, it adds a KUnit test suite for BLAKE2s and deletes the older blake2s-selftest.
Quoted 14 lines
Eric Biggers (12):
arm: configs: Remove obsolete assignments toCRYPTO_CHACHA20_NEON
crypto: chacha - register only "-lib" drivers
lib/crypto: chacha: Remove unused functionchacha_is_arch_optimized()lib/crypto: chacha: Renamechacha.ctochacha-block-generic.clib/crypto: chacha: Renamelibchacha.ctochacha.clib/crypto: chacha: Consolidate into single module lib/crypto: x86/blake2s: Reduce size ofBLAKE2S_SIGMA2lib/crypto: blake2s: Remove obsolete self-test
lib/crypto: blake2s: Always enable arch-optimized BLAKE2s code lib/crypto: blake2s: Move generic code intoblake2s.clib/crypto: blake2s: Consolidate into single C translation unit lib/crypto: tests: Add KUnit tests for BLAKE2s
Quoted 19 lines · Eric Biggers
This series is targeting libcrypto-next. It can also be retrieved from:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git chacha-blake2s-v1
This series consolidates how the ChaCha and BLAKE2s code is organized. This is essentially the same change that I made to the other algorithms, so this should be fairly boring by now.
These algorithms were the last two users of lib/crypto/$(SRCARCH)/{Makefile,Kconfig}. So this series removes all those files, finishing the transition to the centralized build process (at least for the algorithms supported by
lib/crypto/so far).This series also makes the arch-optimized BLAKE2s code start being enabled by default, again following the pattern of the other algorithms.
Finally, it adds a KUnit test suite for BLAKE2s and deletes the older blake2s-selftest.
Quoted 13 lines
Eric Biggers (12):
arm: configs: Remove obsolete assignments toCRYPTO_CHACHA20_NEON
crypto: chacha - register only "-lib" drivers
lib/crypto: chacha: Remove unused functionchacha_is_arch_optimized()lib/crypto: chacha: Renamechacha.ctochacha-block-generic.clib/crypto: chacha: Renamelibchacha.ctochacha.clib/crypto: chacha: Consolidate into single module lib/crypto: x86/blake2s: Reduce size ofBLAKE2S_SIGMA2lib/crypto: blake2s: Remove obsolete self-test
lib/crypto: blake2s: Always enable arch-optimized BLAKE2s code lib/crypto: blake2s: Move generic code intoblake2s.clib/crypto: blake2s: Consolidate into single C translation unit lib/crypto: tests: Add KUnit tests for BLAKE2s
Applied to https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=libcrypto-next
- Eric