crypto: test the !may_use_simd() fallback code
Subsystems (5)
cover crypto: test the !may_use_simd() fallback code
All crypto API algorithms are supposed to support the case where they are called in a context where SIMD instructions are unusable, e.g. IRQ context on some architectures. However, this isn't tested for by the self-tests, causing bugs to go undetected.
This patch series therefore updates the self-tests to test the no-SIMD code. It works by converting all may_use_simd() checks to a new macro crypto_simd_usable(), which also returns false when the self-tests have disabled SIMD in crypto algorithms on the current CPU for test purposes.
For now, all no-SIMD testing is limited to the extra crypto self-tests, because it might be a bit too invasive for the regular self-tests. But this could be changed later.
This already found bugs in the arm64 implementations of AES-GCM and ChaCha. These are fixed by the first two patches. Following this, the tests pass on x86, arm, and arm64.
This patch series is based on top of my other pending patch series "crypto: add SIMD helpers for AEADs". It can also be found in git at:
URL: https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git
Branch: crypto-nosimd-tests
Eric Biggers (8):
crypto: chacha-generic - fix use as arm64 no-NEON fallback
crypto: arm64/gcm-aes-ce - fix no-NEON fallback code
crypto: simd,testmgr - introduce crypto_simd_usable()
crypto: x86 - convert to use crypto_simd_usable()
crypto: arm - convert to use crypto_simd_usable()
crypto: arm64 - convert to use crypto_simd_usable()
crypto: simd - convert to use crypto_simd_usable()
crypto: testmgr - test the !may_use_simd() fallback code
arch/arm/crypto/chacha-neon-glue.c | 5 +- arch/arm/crypto/crc32-ce-glue.c | 5 +- arch/arm/crypto/crct10dif-ce-glue.c | 3 +- arch/arm/crypto/ghash-ce-glue.c | 7 +- arch/arm/crypto/nhpoly1305-neon-glue.c | 3 +- arch/arm/crypto/sha1-ce-glue.c | 5 +- arch/arm/crypto/sha1_neon_glue.c | 5 +- arch/arm/crypto/sha2-ce-glue.c | 5 +- arch/arm/crypto/sha256_neon_glue.c | 5 +- arch/arm/crypto/sha512-neon-glue.c | 5 +- arch/arm64/crypto/aes-ce-ccm-glue.c | 7 +- arch/arm64/crypto/aes-ce-glue.c | 5 +- arch/arm64/crypto/aes-glue.c | 4 +- arch/arm64/crypto/aes-neonbs-glue.c | 2 +- arch/arm64/crypto/chacha-neon-glue.c | 5 +- arch/arm64/crypto/crct10dif-ce-glue.c | 5 +- arch/arm64/crypto/ghash-ce-glue.c | 17 ++- arch/arm64/crypto/nhpoly1305-neon-glue.c | 3 +- arch/arm64/crypto/sha1-ce-glue.c | 7 +- arch/arm64/crypto/sha2-ce-glue.c | 7 +- arch/arm64/crypto/sha256-glue.c | 5 +- arch/arm64/crypto/sha3-ce-glue.c | 5 +- arch/arm64/crypto/sha512-ce-glue.c | 7 +- arch/arm64/crypto/sm3-ce-glue.c | 7 +- arch/arm64/crypto/sm4-ce-glue.c | 5 +- arch/x86/crypto/aesni-intel_glue.c | 8 +- arch/x86/crypto/chacha_glue.c | 6 +- arch/x86/crypto/crc32-pclmul_glue.c | 5 +- arch/x86/crypto/crc32c-intel_glue.c | 7 +- arch/x86/crypto/crct10dif-pclmul_glue.c | 7 +- arch/x86/crypto/ghash-clmulni-intel_glue.c | 9 +- arch/x86/crypto/nhpoly1305-avx2-glue.c | 5 +- arch/x86/crypto/nhpoly1305-sse2-glue.c | 5 +- arch/x86/crypto/poly1305_glue.c | 4 +- arch/x86/crypto/sha1_ssse3_glue.c | 7 +- arch/x86/crypto/sha256_ssse3_glue.c | 7 +- arch/x86/crypto/sha512_ssse3_glue.c | 10 +- crypto/chacha_generic.c | 2 +- crypto/simd.c | 8 +- crypto/testmgr.c | 142 +++++++++++++++++---- include/crypto/internal/simd.h | 24 ++++ 41 files changed, 272 insertions(+), 123 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/8 crypto: chacha-generic - fix use as arm64 no-NEON fallback
From: Eric Biggers <[email protected]>
The arm64 implementations of ChaCha and XChaCha are failing the extra crypto self-tests following my patches to test the !may_use_simd() code paths, which previously were untested. The problem is as follows:
When !may_use_simd(), the arm64 NEON implementations fall back to the generic implementation, which uses the skcipher_walk API to iterate through the src/dst scatterlists. Due to how the skcipher_walk API works, walk.stride is set from the skcipher_alg actually being used, which in this case is the arm64 NEON algorithm. Thus walk.stride is 5*CHACHA_BLOCK_SIZE, not CHACHA_BLOCK_SIZE.
This unnecessarily large stride shouldn't cause an actual problem. However, the generic implementation computes round_down(nbytes, walk.stride). round_down() assumes the round amount is a power of 2, which 5*CHACHA_BLOCK_SIZE is not, so it gives the wrong result.
This causes the following case in skcipher_walk_done() to be hit, causing a WARN() and failing the encryption operation:
if (WARN_ON(err)) {
/* unexpected case; didn't process all bytes */
err = -EINVAL;
goto finish;
}Fix it by rounding down to CHACHA_BLOCK_SIZE instead of walk.stride.
(Or we could replace round_down() with rounddown(), but that would add a slow division operation every time, which I think we should avoid.)
--- crypto/chacha_generic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
crypto/chacha_generic.c +1 -1
@@ -52,7 +52,7 @@ static int chacha_stream_xor(struct skcipher_request *req,unsigned int nbytes = walk.nbytes;if (nbytes < walk.total) - nbytes =round_down (nbytes, walk.stride); + nbytes =round_down (nbytes,CHACHA_BLOCK_SIZE );chacha_docrypt (state, walk.dst.virt.addr, walk.src.virt.addr, nbytes, ctx->nrounds);
Signature · 1 line
2.21.0
Re: [PATCH 1/8] crypto: chacha-generic - fix use as arm64 no-NEON fallback
Quoted 28 lines · Eric Biggers
From: Eric Biggers <[email protected]>
The arm64 implementations of ChaCha and XChaCha are failing the extra crypto self-tests following my patches to test the !may_use_simd() code paths, which previously were untested. The problem is as follows:
When !may_use_simd(), the arm64 NEON implementations fall back to the generic implementation, which uses the
skcipher_walkAPI to iterate through the src/dst scatterlists. Due to how theskcipher_walkAPI works, walk.stride is set from theskcipher_algactually being used, which in this case is the arm64 NEON algorithm. Thus walk.stride is 5*CHACHA_BLOCK_SIZE, notCHACHA_BLOCK_SIZE.This unnecessarily large stride shouldn't cause an actual problem. However, the generic implementation computes
round_down(nbytes, walk.stride).round_down()assumes the round amount is a power of 2, which 5*CHACHA_BLOCK_SIZE is not, so it gives the wrong result.This causes the following case in
skcipher_walk_done()to be hit, causing aWARN()and failing the encryption operation:if (WARN_ON(err)) { /* unexpected case; didn't process all bytes */ err = -EINVAL; goto finish; }
Quoted 8 lines
Fix it by rounding down to
CHACHA_BLOCK_SIZEinstead of walk.stride.(Or we could replace
round_down()withrounddown(), but that would add a slow division operation every time, which I think we should avoid.)Fixes 2fe55987b262 ("crypto: arm64/chacha - use combined SIMD/ALU routine for more speed")Cc <[email protected]> # v5.0+Signed-off-by Eric Biggers <[email protected]>
Quoted 20 lines
--- crypto/chacha_generic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)crypto/chacha_generic.c +1
-2@@ -52,7 +52,7 @@ static int chacha_stream_xor(struct skcipher_request *req,unsigned int nbytes = walk.nbytes;if (nbytes < walk.total) - nbytes =round_down (nbytes, walk.stride); + nbytes =round_down (nbytes,CHACHA_BLOCK_SIZE );chacha_docrypt (state, walk.dst.virt.addr, walk.src.virt.addr, nbytes, ctx->nrounds); --2.21.0
PATCH 2/8 crypto: arm64/gcm-aes-ce - fix no-NEON fallback code
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
The arm64 gcm-aes-ce algorithm is failing the extra crypto self-tests following my patches to test the !may_use_simd() code paths, which previously were untested. The problem is that in the !may_use_simd() case, an odd number of AES blocks can be processed within each step of the skcipher_walk. However, the skcipher_walk is being done with a "stride" of 2 blocks and is advanced by an even number of blocks after each step. This causes the encryption to produce the wrong ciphertext and authentication tag, and causes the decryption to incorrectly fail.
Fix it by only processing an even number of blocks per step.
--- arch/arm64/crypto/ghash-ce-glue.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
arch/arm64/crypto/ghash-ce-glue.c +6 -4
@@ -473,9 +473,11 @@ static int gcm_encrypt(struct aead_request *req)put_unaligned_be32 (2 , iv +GCM_IV_SIZE ); while (walk.nbytes >= (2 * AES_BLOCK_SIZE)) { - int blocks = walk.nbytes / AES_BLOCK_SIZE; + const int blocks = + walk.nbytes / (2 * AES_BLOCK_SIZE) * 2; u8 *dst = walk.dst.virt.addr; u8 *src = walk.src.virt.addr; + int remaining = blocks; do { __aes_arm64_encrypt(ctx->aes_key.key_enc,
@@ -485,9 +487,9 @@ static int gcm_encrypt(struct aead_request *req) dst +=AES_BLOCK_SIZE ; src +=AES_BLOCK_SIZE ; - }while (--blocks >0 ); + }while (--remaining >0 ); -ghash_do_update (walk.nbytes /AES_BLOCK_SIZE , dg, +ghash_do_update (blocks, dg, walk.dst.virt.addr, &ctx->ghash_key,NULL , pmull_ghash_update_p64);
@@ -609,7 +611,7 @@ static int gcm_decrypt(struct aead_request *req)put_unaligned_be32 (2 , iv +GCM_IV_SIZE );while (walk.nbytes >= (2 *AES_BLOCK_SIZE )) { -int blocks = walk.nbytes /AES_BLOCK_SIZE ; +int blocks = walk.nbytes / (2 *AES_BLOCK_SIZE ) *2 ;u8 *dst = walk.dst.virt.addr;u8 *src = walk.src.virt.addr;
Signature · 1 line
2.21.0
Re: [PATCH 2/8] crypto: arm64/gcm-aes-ce - fix no-NEON fallback code
Quoted 12 lines · Eric Biggers
From: Eric Biggers <[email protected]>
The arm64 gcm-aes-ce algorithm is failing the extra crypto self-tests following my patches to test the !may_use_simd() code paths, which previously were untested. The problem is that in the !may_use_simd() case, an odd number of AES blocks can be processed within each step of the
skcipher_walk. However, theskcipher_walkis being done with a "stride" of 2 blocks and is advanced by an even number of blocks after each step. This causes the encryption to produce the wrong ciphertext and authentication tag, and causes the decryption to incorrectly fail.
Quoted 6 lines
Fix it by only processing an even number of blocks per step.
Fixes c2b24c36e0a3 ("crypto: arm64/aes-gcm-ce - fix scatterwalk API violation")Fixes 71e52c278c54 ("crypto: arm64/aes-ce-gcm - operate on two input blocks at a time")Cc <[email protected]> # v4.19+Signed-off-by Eric Biggers <[email protected]>
Quoted 45 lines
--- arch/arm64/crypto/ghash-ce-glue.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)arch/arm64/crypto/ghash-ce-glue.c +6
-5@@ -473,9 +473,11 @@ static int gcm_encrypt(struct aead_request *req)put_unaligned_be32 (2 , iv +GCM_IV_SIZE ); while (walk.nbytes >= (2 * AES_BLOCK_SIZE)) { - int blocks = walk.nbytes / AES_BLOCK_SIZE; + const int blocks = + walk.nbytes / (2 * AES_BLOCK_SIZE) * 2; u8 *dst = walk.dst.virt.addr; u8 *src = walk.src.virt.addr; + int remaining = blocks; do { __aes_arm64_encrypt(ctx->aes_key.key_enc,@@ -485,9 +487,9 @@ static int gcm_encrypt(struct aead_request *req) dst +=AES_BLOCK_SIZE ; src +=AES_BLOCK_SIZE ; - }while (--blocks >0 ); + }while (--remaining >0 ); -ghash_do_update (walk.nbytes /AES_BLOCK_SIZE , dg, +ghash_do_update (blocks, dg, walk.dst.virt.addr, &ctx->ghash_key,NULL , pmull_ghash_update_p64);@@ -609,7 +611,7 @@ static int gcm_decrypt(struct aead_request *req)put_unaligned_be32 (2 , iv +GCM_IV_SIZE );while (walk.nbytes >= (2 *AES_BLOCK_SIZE )) { -int blocks = walk.nbytes /AES_BLOCK_SIZE ; +int blocks = walk.nbytes / (2 *AES_BLOCK_SIZE ) *2 ;u8 *dst = walk.dst.virt.addr;u8 *src = walk.src.virt.addr; --2.21.0
PATCH 3/8 crypto: simd,testmgr - introduce crypto_simd_usable()
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
So that the no-SIMD fallback code can be tested by the crypto self-tests, add a macro crypto_simd_usable() which wraps may_use_simd(), but also returns false if the crypto self-tests have set a per-CPU bool to disable SIMD in crypto code on the current CPU.
--- crypto/testmgr.c | 26 +++++++++++++++++++++++++- include/crypto/internal/simd.h | 24 ++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 1 deletion(-)
crypto/testmgr.c +25 -1
@@ -37,6 +37,7 @@#include <crypto/akcipher.h> #include <crypto/kpp.h> #include <crypto/acompress.h> +#include <crypto/internal/simd.h> #include "internal.h"
@@ -52,6 +53,9 @@ MODULE_PARM_DESC(noextratests, "disable expensive crypto self-tests");static unsigned int fuzz_iterations =100 ;module_param (fuzz_iterations, uint,0644 );MODULE_PARM_DESC (fuzz_iterations,"number of fuzz test iterations" ); + +DEFINE_PER_CPU (bool, crypto_simd_disabled_for_test); +EXPORT_PER_CPU_SYMBOL_GPL (crypto_simd_disabled_for_test);#endif #ifdef CONFIG_CRYPTO_MANAGER_DISABLE_TESTS
@@ -838,7 +842,27 @@ static void generate_random_testvec_config(struct testvec_config *cfg,WARN_ON_ONCE (!valid_testvec_config (cfg)); } -#endif /* CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ + +static void crypto_disable_simd_for_test (void ) +{ +preempt_disable (); +__this_cpu_write (crypto_simd_disabled_for_test, true); +} + +static void crypto_reenable_simd_for_test (void ) +{ +__this_cpu_write (crypto_simd_disabled_for_test, false); +preempt_enable (); +} +#else /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ +static void crypto_disable_simd_for_test (void ) +{ +} + +static void crypto_reenable_simd_for_test (void ) +{ +} +#endif /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ static int check_nonfinal_hash_op(const char *op, int err, u8 *result, unsigned int digestsize,
include/crypto/internal/simd.h +24 -0
@@ -6,6 +6,9 @@#ifndef _CRYPTO_INTERNAL_SIMD_H#define _CRYPTO_INTERNAL_SIMD_H +#include <linux/percpu.h> +#include <linux/types.h> +/* skcipher support */ struct simd_skcipher_alg ;
@@ -42,4 +45,25 @@ int simd_register_aeads_compat(struct aead_alg *algs, int count,void simd_unregister_aeads (struct aead_alg *algs,int count,struct simd_aead_alg **simd_algs); +/* +* crypto_simd_usable() - is it allowed at this time to use SIMD instructions or +* access the SIMD register file? +* +* This delegates to may_use_simd(), except that this also returns false if SIMD +* in crypto code has been temporarily disabled on this CPU by the crypto +* self-tests, in order to test the no-SIMD fallback code. This override is +* currently limited to configurations where the extra self-tests are enabled, +* because it might be a bit too invasive to be part of the regular self-tests. +* +* This is a macro so that <asm/simd.h>, which some architectures don't have, +* doesn't have to be included directly here. +*/ +#ifdef CONFIG_CRYPTO_MANAGER_EXTRA_TESTS +DECLARE_PER_CPU (bool, crypto_simd_disabled_for_test); +#define crypto_simd_usable () \ + (may_use_simd() && !this_cpu_read(crypto_simd_disabled_for_test)) +#else +#define crypto_simd_usable () may_use_simd() +#endif +#endif /* _CRYPTO_INTERNAL_SIMD_H */
Signature · 1 line
2.21.0
Re: [PATCH 3/8] crypto: simd, testmgr - introduce crypto_simd_usable()
Quoted 9 lines · Eric Biggers
From: Eric Biggers <[email protected]>
So that the no-SIMD fallback code can be tested by the crypto self-tests, add a macro
crypto_simd_usable()which wrapsmay_use_simd(), but also returns false if the crypto self-tests have set a per-CPU bool to disable SIMD in crypto code on the current CPU.Signed-off-by Eric Biggers <[email protected]>
Quoted 99 lines
--- crypto/testmgr.c | 26 +++++++++++++++++++++++++- include/crypto/internal/simd.h | 24 ++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 1 deletion(-)crypto/testmgr.c +25
-1@@ -37,6 +37,7 @@#include <crypto/akcipher.h> #include <crypto/kpp.h> #include <crypto/acompress.h> +#include <crypto/internal/simd.h> #include "internal.h" @@ -52,6 +53,9 @@ MODULE_PARM_DESC(noextratests, "disable expensive crypto self-tests");static unsigned int fuzz_iterations =100 ;module_param (fuzz_iterations, uint,0644 );MODULE_PARM_DESC (fuzz_iterations,"number of fuzz test iterations" ); + +DEFINE_PER_CPU (bool, crypto_simd_disabled_for_test); +EXPORT_PER_CPU_SYMBOL_GPL (crypto_simd_disabled_for_test);#endif #ifdef CONFIG_CRYPTO_MANAGER_DISABLE_TESTS @@ -838,7 +842,27 @@ static void generate_random_testvec_config(struct testvec_config *cfg,WARN_ON_ONCE (!valid_testvec_config (cfg)); } -#endif /* CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ + +static void crypto_disable_simd_for_test (void ) +{ +preempt_disable (); +__this_cpu_write (crypto_simd_disabled_for_test, true); +} + +static void crypto_reenable_simd_for_test (void ) +{ +__this_cpu_write (crypto_simd_disabled_for_test, false); +preempt_enable (); +} +#else /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ +static void crypto_disable_simd_for_test (void ) +{ +} + +static void crypto_reenable_simd_for_test (void ) +{ +} +#endif /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ static int check_nonfinal_hash_op(const char *op, int err, u8 *result, unsigned int digestsize,include/crypto/internal/simd.h +24
-1@@ -6,6 +6,9 @@#ifndef _CRYPTO_INTERNAL_SIMD_H#define _CRYPTO_INTERNAL_SIMD_H +#include <linux/percpu.h> +#include <linux/types.h> +/* skcipher support */ struct simd_skcipher_alg ;@@ -42,4 +45,25 @@ int simd_register_aeads_compat(struct aead_alg *algs, int count,void simd_unregister_aeads (struct aead_alg *algs,int count,struct simd_aead_alg **simd_algs); +/* +* crypto_simd_usable() - is it allowed at this time to use SIMD instructions or +* access the SIMD register file? +* +* This delegates to may_use_simd(), except that this also returns false if SIMD +* in crypto code has been temporarily disabled on this CPU by the crypto +* self-tests, in order to test the no-SIMD fallback code. This override is +* currently limited to configurations where the extra self-tests are enabled, +* because it might be a bit too invasive to be part of the regular self-tests. +* +* This is a macro so that <asm/simd.h>, which some architectures don't have, +* doesn't have to be included directly here. +*/ +#ifdef CONFIG_CRYPTO_MANAGER_EXTRA_TESTS +DECLARE_PER_CPU (bool, crypto_simd_disabled_for_test); +#define crypto_simd_usable () \ + (may_use_simd() && !this_cpu_read(crypto_simd_disabled_for_test)) +#else +#define crypto_simd_usable () may_use_simd() +#endif +#endif /* _CRYPTO_INTERNAL_SIMD_H */ --2.21.0
PATCH 4/8 crypto: x86 - convert to use crypto_simd_usable()
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Replace all calls to irq_fpu_usable() in the x86 crypto code with crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
--- arch/x86/crypto/aesni-intel_glue.c | 8 ++++---- arch/x86/crypto/chacha_glue.c | 6 +++--- arch/x86/crypto/crc32-pclmul_glue.c | 5 +++-- arch/x86/crypto/crc32c-intel_glue.c | 7 ++++--- arch/x86/crypto/crct10dif-pclmul_glue.c | 7 ++++--- arch/x86/crypto/ghash-clmulni-intel_glue.c | 9 +++++---- arch/x86/crypto/nhpoly1305-avx2-glue.c | 5 +++-- arch/x86/crypto/nhpoly1305-sse2-glue.c | 5 +++-- arch/x86/crypto/poly1305_glue.c | 4 ++-- arch/x86/crypto/sha1_ssse3_glue.c | 7 ++++--- arch/x86/crypto/sha256_ssse3_glue.c | 7 ++++--- arch/x86/crypto/sha512_ssse3_glue.c | 10 +++++----- 12 files changed, 44 insertions(+), 36 deletions(-)
arch/x86/crypto/aesni-intel_glue.c +4 -4
@@ -30,8 +30,8 @@#include <crypto/gcm.h> #include <crypto/xts.h> #include <asm/cpu_device_id.h> -#include <asm/fpu/api.h> #include <asm/crypto/aes.h> +#include <asm/simd.h> #include <crypto/scatterwalk.h> #include <crypto/internal/aead.h> #include <crypto/internal/simd.h>
@@ -332,7 +332,7 @@ static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx,return -EINVAL ; } -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ()) err =crypto_aes_expand_key (ctx, in_key, key_len);else {kernel_fpu_begin ();
@@ -353,7 +353,7 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) {struct crypto_aes_ctx *ctx =aes_ctx (crypto_tfm_ctx (tfm)); -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ())crypto_aes_encrypt_x86 (ctx, dst, src);else {kernel_fpu_begin ();
@@ -366,7 +366,7 @@ static void aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) {struct crypto_aes_ctx *ctx =aes_ctx (crypto_tfm_ctx (tfm)); -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ())crypto_aes_decrypt_x86 (ctx, dst, src);else {kernel_fpu_begin ();
arch/x86/crypto/chacha_glue.c +3 -3
@@ -12,10 +12,10 @@#include <crypto/algapi.h> #include <crypto/chacha.h> +#include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <linux/kernel.h> #include <linux/module.h> -#include <asm/fpu/api.h> #include <asm/simd.h> #define CHACHA_STATE_ALIGN 16
@@ -170,7 +170,7 @@ static int chacha_simd(struct skcipher_request *req)struct skcipher_walk walk;int err; -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !irq_fpu_usable ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_chacha_crypt (req); err =skcipher_walk_virt (&walk, req, true);
@@ -193,7 +193,7 @@ static int xchacha_simd(struct skcipher_request *req)u8 real_iv[16 ];int err; -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !irq_fpu_usable ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_xchacha_crypt (req); err =skcipher_walk_virt (&walk, req, true);
arch/x86/crypto/crc32-pclmul_glue.c +3 -2
@@ -32,10 +32,11 @@#include <linux/kernel.h> #include <linux/crc32.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <asm/cpufeatures.h> #include <asm/cpu_device_id.h> -#include <asm/fpu/api.h> +#include <asm/simd.h> #define CHKSUM_BLOCK_SIZE 1#define CHKSUM_DIGEST_SIZE 4
@@ -54,7 +55,7 @@ static u32 __attribute__((pure))unsigned int iremainder;unsigned int prealign; -if (len <PCLMUL_MIN_LEN +SCALE_F_MASK || !irq_fpu_usable ()) +if (len <PCLMUL_MIN_LEN +SCALE_F_MASK || !crypto_simd_usable ())return crc32_le (crc, p, len);if ((long )p &SCALE_F_MASK ) {
arch/x86/crypto/crc32c-intel_glue.c +4 -3
@@ -29,10 +29,11 @@#include <linux/string.h> #include <linux/kernel.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <asm/cpufeatures.h> #include <asm/cpu_device_id.h> -#include <asm/fpu/internal.h> +#include <asm/simd.h> #define CHKSUM_BLOCK_SIZE 1#define CHKSUM_DIGEST_SIZE 4
@@ -177,7 +178,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,* use faster PCL version if datasize is large enough to * overcome kernel fpu state save/restore overhead */ -if (len >=CRC32C_PCL_BREAKEVEN &&irq_fpu_usable ()) { +if (len >=CRC32C_PCL_BREAKEVEN &&crypto_simd_usable ()) {kernel_fpu_begin (); *crcp =crc_pcl (data, len, *crcp);kernel_fpu_end ();
@@ -189,7 +190,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,static int __crc32c_pcl_intel_finup (u32 *crcp,const u8 *data,unsigned int len,u8 *out) { -if (len >=CRC32C_PCL_BREAKEVEN &&irq_fpu_usable ()) { +if (len >=CRC32C_PCL_BREAKEVEN &&crypto_simd_usable ()) {kernel_fpu_begin (); *(__le32 *)out = ~cpu_to_le32 (crc_pcl (data, len, *crcp));kernel_fpu_end ();
arch/x86/crypto/crct10dif-pclmul_glue.c +4 -3
@@ -26,12 +26,13 @@#include <linux/module.h> #include <linux/crc-t10dif.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/init.h> #include <linux/string.h> #include <linux/kernel.h> -#include <asm/fpu/api.h> #include <asm/cpufeatures.h> #include <asm/cpu_device_id.h> +#include <asm/simd.h> asmlinkage u16 crc_t10dif_pcl(u16 init_crc, const u8 *buf, size_t len);
@@ -53,7 +54,7 @@ static int chksum_update(struct shash_desc *desc, const u8 *data, {struct chksum_desc_ctx *ctx =shash_desc_ctx (desc); -if (length >=16 &&irq_fpu_usable ()) { +if (length >=16 &&crypto_simd_usable ()) {kernel_fpu_begin (); ctx->crc =crc_t10dif_pcl (ctx->crc, data, length);kernel_fpu_end ();
@@ -73,7 +74,7 @@ static int chksum_final(struct shash_desc *desc, u8 *out)static int __chksum_finup (__u16 *crcp,const u8 *data,unsigned int len,u8 *out) { -if (len >=16 &&irq_fpu_usable ()) { +if (len >=16 &&crypto_simd_usable ()) {kernel_fpu_begin (); *(__u16 *)out =crc_t10dif_pcl (*crcp, data, len);kernel_fpu_end ();
arch/x86/crypto/ghash-clmulni-intel_glue.c +5 -4
@@ -19,8 +19,9 @@#include <crypto/cryptd.h> #include <crypto/gf128mul.h> #include <crypto/internal/hash.h> -#include <asm/fpu/api.h> +#include <crypto/internal/simd.h> #include <asm/cpu_device_id.h> +#include <asm/simd.h> #define GHASH_BLOCK_SIZE 16#define GHASH_DIGEST_SIZE 16
@@ -182,7 +183,7 @@ static int ghash_async_update(struct ahash_request *req)struct ghash_async_ctx *ctx =crypto_ahash_ctx (tfm);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);
@@ -200,7 +201,7 @@ static int ghash_async_final(struct ahash_request *req)struct ghash_async_ctx *ctx =crypto_ahash_ctx (tfm);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);
@@ -241,7 +242,7 @@ static int ghash_async_digest(struct ahash_request *req)struct ahash_request *cryptd_req =ahash_request_ctx (req);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);
arch/x86/crypto/nhpoly1305-avx2-glue.c +3 -2
@@ -7,9 +7,10 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/nhpoly1305.h> #include <linux/module.h> -#include <asm/fpu/api.h> +#include <asm/simd.h> asmlinkage void nh_avx2(const u32 *key, const u8 *message, size_t message_len,u8 hash[NH_HASH_BYTES ]);
@@ -24,7 +25,7 @@ static void _nh_avx2(const u32 *key, const u8 *message, size_t message_len,static int nhpoly1305_avx2_update (struct shash_desc *desc,const u8 *src,unsigned int srclen) { -if (srclen <64 || !irq_fpu_usable ()) +if (srclen <64 || !crypto_simd_usable ())return crypto_nhpoly1305_update (desc, src, srclen); do {
arch/x86/crypto/nhpoly1305-sse2-glue.c +3 -2
@@ -7,9 +7,10 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/nhpoly1305.h> #include <linux/module.h> -#include <asm/fpu/api.h> +#include <asm/simd.h> asmlinkage void nh_sse2(const u32 *key, const u8 *message, size_t message_len,u8 hash[NH_HASH_BYTES ]);
@@ -24,7 +25,7 @@ static void _nh_sse2(const u32 *key, const u8 *message, size_t message_len,static int nhpoly1305_sse2_update (struct shash_desc *desc,const u8 *src,unsigned int srclen) { -if (srclen <64 || !irq_fpu_usable ()) +if (srclen <64 || !crypto_simd_usable ())return crypto_nhpoly1305_update (desc, src, srclen); do {
arch/x86/crypto/poly1305_glue.c +2 -2
@@ -11,11 +11,11 @@#include <crypto/algapi.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/poly1305.h> #include <linux/crypto.h> #include <linux/kernel.h> #include <linux/module.h> -#include <asm/fpu/api.h> #include <asm/simd.h> struct poly1305_simd_desc_ctx {
@@ -126,7 +126,7 @@ static int poly1305_simd_update(struct shash_desc *desc,unsigned int bytes;/* kernel_fpu_begin/end is costly, use fallback for small updates */ -if (srclen <=288 || !may_use_simd ()) +if (srclen <=288 || !crypto_simd_usable ())return crypto_poly1305_update (desc, src, srclen);kernel_fpu_begin ();
arch/x86/crypto/sha1_ssse3_glue.c +4 -3
@@ -22,6 +22,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/init.h> #include <linux/module.h> #include <linux/mm.h>
@@ -29,7 +30,7 @@#include <linux/types.h> #include <crypto/sha.h> #include <crypto/sha1_base.h> -#include <asm/fpu/api.h> +#include <asm/simd.h> typedef void (sha1_transform_fn )(u32 *digest,const char *data,unsigned int rounds);
@@ -39,7 +40,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data, {struct sha1_state *sctx =shash_desc_ctx (desc); -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (sctx->count %SHA1_BLOCK_SIZE ) + len <SHA1_BLOCK_SIZE )return crypto_sha1_update (desc, data, len);
@@ -57,7 +58,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data,static int sha1_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out,sha1_transform_fn *sha1_xform) { -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ())return crypto_sha1_finup (desc, data, len, out);kernel_fpu_begin ();
arch/x86/crypto/sha256_ssse3_glue.c +4 -3
@@ -30,6 +30,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/init.h> #include <linux/module.h> #include <linux/mm.h>
@@ -37,8 +38,8 @@#include <linux/types.h> #include <crypto/sha.h> #include <crypto/sha256_base.h> -#include <asm/fpu/api.h> #include <linux/string.h> +#include <asm/simd.h> asmlinkage void sha256_transform_ssse3(u32 *digest, const char *data,u64 rounds);
@@ -49,7 +50,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data, {struct sha256_state *sctx =shash_desc_ctx (desc); -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (sctx->count %SHA256_BLOCK_SIZE ) + len <SHA256_BLOCK_SIZE )return crypto_sha256_update (desc, data, len);
@@ -67,7 +68,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,static int sha256_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out,sha256_transform_fn *sha256_xform) { -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ())return crypto_sha256_finup (desc, data, len, out);kernel_fpu_begin ();
arch/x86/crypto/sha512_ssse3_glue.c +5 -5
@@ -28,16 +28,16 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/init.h> #include <linux/module.h> #include <linux/mm.h> #include <linux/cryptohash.h> +#include <linux/string.h> #include <linux/types.h> #include <crypto/sha.h> #include <crypto/sha512_base.h> -#include <asm/fpu/api.h> - -#include <linux/string.h> +#include <asm/simd.h> asmlinkage void sha512_transform_ssse3(u64 *digest, const char *data,u64 rounds);
@@ -49,7 +49,7 @@ static int sha512_update(struct shash_desc *desc, const u8 *data, {struct sha512_state *sctx =shash_desc_ctx (desc); -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (sctx->count[0 ] %SHA512_BLOCK_SIZE ) + len <SHA512_BLOCK_SIZE )return crypto_sha512_update (desc, data, len);
@@ -67,7 +67,7 @@ static int sha512_update(struct shash_desc *desc, const u8 *data,static int sha512_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out,sha512_transform_fn *sha512_xform) { -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ())return crypto_sha512_finup (desc, data, len, out);kernel_fpu_begin ();
Signature · 1 line
2.21.0
Re: [PATCH 4/8] crypto: x86 - convert to use crypto_simd_usable()
Quoted 7 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Replace all calls to
irq_fpu_usable()in the x86 crypto code withcrypto_simd_usable(), in order to allow testing the no-SIMD code paths.Signed-off-by Eric Biggers <[email protected]>
Quoted 430 lines
--- arch/x86/crypto/aesni-intel_glue.c | 8 ++++---- arch/x86/crypto/chacha_glue.c | 6 +++--- arch/x86/crypto/crc32-pclmul_glue.c | 5 +++-- arch/x86/crypto/crc32c-intel_glue.c | 7 ++++--- arch/x86/crypto/crct10dif-pclmul_glue.c | 7 ++++--- arch/x86/crypto/ghash-clmulni-intel_glue.c | 9 +++++---- arch/x86/crypto/nhpoly1305-avx2-glue.c | 5 +++-- arch/x86/crypto/nhpoly1305-sse2-glue.c | 5 +++-- arch/x86/crypto/poly1305_glue.c | 4 ++-- arch/x86/crypto/sha1_ssse3_glue.c | 7 ++++--- arch/x86/crypto/sha256_ssse3_glue.c | 7 ++++--- arch/x86/crypto/sha512_ssse3_glue.c | 10 +++++----- 12 files changed, 44 insertions(+), 36 deletions(-)arch/x86/crypto/aesni-intel_glue.c +4
-4@@ -30,8 +30,8 @@#include <crypto/gcm.h> #include <crypto/xts.h> #include <asm/cpu_device_id.h> -#include <asm/fpu/api.h> #include <asm/crypto/aes.h> +#include <asm/simd.h> #include <crypto/scatterwalk.h> #include <crypto/internal/aead.h> #include <crypto/internal/simd.h> @@ -332,7 +332,7 @@ static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx,return -EINVAL ; } -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ()) err =crypto_aes_expand_key (ctx, in_key, key_len);else {kernel_fpu_begin ();@@ -353,7 +353,7 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) {struct crypto_aes_ctx *ctx =aes_ctx (crypto_tfm_ctx (tfm)); -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ())crypto_aes_encrypt_x86 (ctx, dst, src);else {kernel_fpu_begin ();@@ -366,7 +366,7 @@ static void aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) {struct crypto_aes_ctx *ctx =aes_ctx (crypto_tfm_ctx (tfm)); -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ())crypto_aes_decrypt_x86 (ctx, dst, src);else {kernel_fpu_begin ();arch/x86/crypto/chacha_glue.c +3
-3@@ -12,10 +12,10 @@#include <crypto/algapi.h> #include <crypto/chacha.h> +#include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <linux/kernel.h> #include <linux/module.h> -#include <asm/fpu/api.h> #include <asm/simd.h> #define CHACHA_STATE_ALIGN 16@@ -170,7 +170,7 @@ static int chacha_simd(struct skcipher_request *req)struct skcipher_walk walk;int err; -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !irq_fpu_usable ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_chacha_crypt (req); err =skcipher_walk_virt (&walk, req, true);@@ -193,7 +193,7 @@ static int xchacha_simd(struct skcipher_request *req)u8 real_iv[16 ];int err; -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !irq_fpu_usable ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_xchacha_crypt (req); err =skcipher_walk_virt (&walk, req, true);arch/x86/crypto/crc32-pclmul_glue.c +3
-2@@ -32,10 +32,11 @@#include <linux/kernel.h> #include <linux/crc32.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <asm/cpufeatures.h> #include <asm/cpu_device_id.h> -#include <asm/fpu/api.h> +#include <asm/simd.h> #define CHKSUM_BLOCK_SIZE 1#define CHKSUM_DIGEST_SIZE 4@@ -54,7 +55,7 @@ static u32 __attribute__((pure))unsigned int iremainder;unsigned int prealign; -if (len <PCLMUL_MIN_LEN +SCALE_F_MASK || !irq_fpu_usable ()) +if (len <PCLMUL_MIN_LEN +SCALE_F_MASK || !crypto_simd_usable ())return crc32_le (crc, p, len);if ((long )p &SCALE_F_MASK ) {arch/x86/crypto/crc32c-intel_glue.c +4
-3@@ -29,10 +29,11 @@#include <linux/string.h> #include <linux/kernel.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <asm/cpufeatures.h> #include <asm/cpu_device_id.h> -#include <asm/fpu/internal.h> +#include <asm/simd.h> #define CHKSUM_BLOCK_SIZE 1#define CHKSUM_DIGEST_SIZE 4@@ -177,7 +178,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,* use faster PCL version if datasize is large enough to * overcome kernel fpu state save/restore overhead */ -if (len >=CRC32C_PCL_BREAKEVEN &&irq_fpu_usable ()) { +if (len >=CRC32C_PCL_BREAKEVEN &&crypto_simd_usable ()) {kernel_fpu_begin (); *crcp =crc_pcl (data, len, *crcp);kernel_fpu_end ();@@ -189,7 +190,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,static int __crc32c_pcl_intel_finup (u32 *crcp,const u8 *data,unsigned int len,u8 *out) { -if (len >=CRC32C_PCL_BREAKEVEN &&irq_fpu_usable ()) { +if (len >=CRC32C_PCL_BREAKEVEN &&crypto_simd_usable ()) {kernel_fpu_begin (); *(__le32 *)out = ~cpu_to_le32 (crc_pcl (data, len, *crcp));kernel_fpu_end ();arch/x86/crypto/crct10dif-pclmul_glue.c +4
-3@@ -26,12 +26,13 @@#include <linux/module.h> #include <linux/crc-t10dif.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/init.h> #include <linux/string.h> #include <linux/kernel.h> -#include <asm/fpu/api.h> #include <asm/cpufeatures.h> #include <asm/cpu_device_id.h> +#include <asm/simd.h> asmlinkage u16 crc_t10dif_pcl(u16 init_crc, const u8 *buf, size_t len);@@ -53,7 +54,7 @@ static int chksum_update(struct shash_desc *desc, const u8 *data, {struct chksum_desc_ctx *ctx =shash_desc_ctx (desc); -if (length >=16 &&irq_fpu_usable ()) { +if (length >=16 &&crypto_simd_usable ()) {kernel_fpu_begin (); ctx->crc =crc_t10dif_pcl (ctx->crc, data, length);kernel_fpu_end ();@@ -73,7 +74,7 @@ static int chksum_final(struct shash_desc *desc, u8 *out)static int __chksum_finup (__u16 *crcp,const u8 *data,unsigned int len,u8 *out) { -if (len >=16 &&irq_fpu_usable ()) { +if (len >=16 &&crypto_simd_usable ()) {kernel_fpu_begin (); *(__u16 *)out =crc_t10dif_pcl (*crcp, data, len);kernel_fpu_end ();arch/x86/crypto/ghash-clmulni-intel_glue.c +5
-4@@ -19,8 +19,9 @@#include <crypto/cryptd.h> #include <crypto/gf128mul.h> #include <crypto/internal/hash.h> -#include <asm/fpu/api.h> +#include <crypto/internal/simd.h> #include <asm/cpu_device_id.h> +#include <asm/simd.h> #define GHASH_BLOCK_SIZE 16#define GHASH_DIGEST_SIZE 16@@ -182,7 +183,7 @@ static int ghash_async_update(struct ahash_request *req)struct ghash_async_ctx *ctx =crypto_ahash_ctx (tfm);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);@@ -200,7 +201,7 @@ static int ghash_async_final(struct ahash_request *req)struct ghash_async_ctx *ctx =crypto_ahash_ctx (tfm);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);@@ -241,7 +242,7 @@ static int ghash_async_digest(struct ahash_request *req)struct ahash_request *cryptd_req =ahash_request_ctx (req);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);arch/x86/crypto/nhpoly1305-avx2-glue.c +3
-2@@ -7,9 +7,10 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/nhpoly1305.h> #include <linux/module.h> -#include <asm/fpu/api.h> +#include <asm/simd.h> asmlinkage void nh_avx2(const u32 *key, const u8 *message, size_t message_len,u8 hash[NH_HASH_BYTES ]);@@ -24,7 +25,7 @@ static void _nh_avx2(const u32 *key, const u8 *message, size_t message_len,static int nhpoly1305_avx2_update (struct shash_desc *desc,const u8 *src,unsigned int srclen) { -if (srclen <64 || !irq_fpu_usable ()) +if (srclen <64 || !crypto_simd_usable ())return crypto_nhpoly1305_update (desc, src, srclen); do {arch/x86/crypto/nhpoly1305-sse2-glue.c +3
-2@@ -7,9 +7,10 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/nhpoly1305.h> #include <linux/module.h> -#include <asm/fpu/api.h> +#include <asm/simd.h> asmlinkage void nh_sse2(const u32 *key, const u8 *message, size_t message_len,u8 hash[NH_HASH_BYTES ]);@@ -24,7 +25,7 @@ static void _nh_sse2(const u32 *key, const u8 *message, size_t message_len,static int nhpoly1305_sse2_update (struct shash_desc *desc,const u8 *src,unsigned int srclen) { -if (srclen <64 || !irq_fpu_usable ()) +if (srclen <64 || !crypto_simd_usable ())return crypto_nhpoly1305_update (desc, src, srclen); do {arch/x86/crypto/poly1305_glue.c +2
-2@@ -11,11 +11,11 @@#include <crypto/algapi.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/poly1305.h> #include <linux/crypto.h> #include <linux/kernel.h> #include <linux/module.h> -#include <asm/fpu/api.h> #include <asm/simd.h> struct poly1305_simd_desc_ctx {@@ -126,7 +126,7 @@ static int poly1305_simd_update(struct shash_desc *desc,unsigned int bytes;/* kernel_fpu_begin/end is costly, use fallback for small updates */ -if (srclen <=288 || !may_use_simd ()) +if (srclen <=288 || !crypto_simd_usable ())return crypto_poly1305_update (desc, src, srclen);kernel_fpu_begin ();arch/x86/crypto/sha1_ssse3_glue.c +4
-3@@ -22,6 +22,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/init.h> #include <linux/module.h> #include <linux/mm.h> @@ -29,7 +30,7 @@#include <linux/types.h> #include <crypto/sha.h> #include <crypto/sha1_base.h> -#include <asm/fpu/api.h> +#include <asm/simd.h> typedef void (sha1_transform_fn )(u32 *digest,const char *data,unsigned int rounds);@@ -39,7 +40,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data, {struct sha1_state *sctx =shash_desc_ctx (desc); -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (sctx->count %SHA1_BLOCK_SIZE ) + len <SHA1_BLOCK_SIZE )return crypto_sha1_update (desc, data, len);@@ -57,7 +58,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data,static int sha1_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out,sha1_transform_fn *sha1_xform) { -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ())return crypto_sha1_finup (desc, data, len, out);kernel_fpu_begin ();arch/x86/crypto/sha256_ssse3_glue.c +4
-3@@ -30,6 +30,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/init.h> #include <linux/module.h> #include <linux/mm.h> @@ -37,8 +38,8 @@#include <linux/types.h> #include <crypto/sha.h> #include <crypto/sha256_base.h> -#include <asm/fpu/api.h> #include <linux/string.h> +#include <asm/simd.h> asmlinkage void sha256_transform_ssse3(u32 *digest, const char *data,u64 rounds);@@ -49,7 +50,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data, {struct sha256_state *sctx =shash_desc_ctx (desc); -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (sctx->count %SHA256_BLOCK_SIZE ) + len <SHA256_BLOCK_SIZE )return crypto_sha256_update (desc, data, len);@@ -67,7 +68,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,static int sha256_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out,sha256_transform_fn *sha256_xform) { -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ())return crypto_sha256_finup (desc, data, len, out);kernel_fpu_begin ();arch/x86/crypto/sha512_ssse3_glue.c +5
-6@@ -28,16 +28,16 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/init.h> #include <linux/module.h> #include <linux/mm.h> #include <linux/cryptohash.h> +#include <linux/string.h> #include <linux/types.h> #include <crypto/sha.h> #include <crypto/sha512_base.h> -#include <asm/fpu/api.h> - -#include <linux/string.h> +#include <asm/simd.h> asmlinkage void sha512_transform_ssse3(u64 *digest, const char *data,u64 rounds);@@ -49,7 +49,7 @@ static int sha512_update(struct shash_desc *desc, const u8 *data, {struct sha512_state *sctx =shash_desc_ctx (desc); -if (!irq_fpu_usable () || +if (!crypto_simd_usable () || (sctx->count[0 ] %SHA512_BLOCK_SIZE ) + len <SHA512_BLOCK_SIZE )return crypto_sha512_update (desc, data, len);@@ -67,7 +67,7 @@ static int sha512_update(struct shash_desc *desc, const u8 *data,static int sha512_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out,sha512_transform_fn *sha512_xform) { -if (!irq_fpu_usable ()) +if (!crypto_simd_usable ())return crypto_sha512_finup (desc, data, len, out);kernel_fpu_begin (); --2.21.0
PATCH 5/8 crypto: arm - convert to use crypto_simd_usable()
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Replace all calls to may_use_simd() in the arm crypto code with crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
--- arch/arm/crypto/chacha-neon-glue.c | 5 +++-- arch/arm/crypto/crc32-ce-glue.c | 5 +++-- arch/arm/crypto/crct10dif-ce-glue.c | 3 ++- arch/arm/crypto/ghash-ce-glue.c | 7 ++++--- arch/arm/crypto/nhpoly1305-neon-glue.c | 3 ++- arch/arm/crypto/sha1-ce-glue.c | 5 +++-- arch/arm/crypto/sha1_neon_glue.c | 5 +++-- arch/arm/crypto/sha2-ce-glue.c | 5 +++-- arch/arm/crypto/sha256_neon_glue.c | 5 +++-- arch/arm/crypto/sha512-neon-glue.c | 5 +++-- 10 files changed, 29 insertions(+), 19 deletions(-)
arch/arm/crypto/chacha-neon-glue.c +3 -2
@@ -21,6 +21,7 @@#include <crypto/algapi.h> #include <crypto/chacha.h> +#include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <linux/kernel.h> #include <linux/module.h>
@@ -93,7 +94,7 @@ static int chacha_neon(struct skcipher_request *req)struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm); -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_chacha_crypt (req);return chacha_neon_stream_xor (req, ctx, req->iv);
@@ -107,7 +108,7 @@ static int xchacha_neon(struct skcipher_request *req)u32 state[16 ];u8 real_iv[16 ]; -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_xchacha_crypt (req);crypto_chacha_init (state, ctx, req->iv);
arch/arm/crypto/crc32-ce-glue.c +3 -2
@@ -16,6 +16,7 @@#include <linux/string.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <asm/hwcap.h> #include <asm/neon.h>
@@ -113,7 +114,7 @@ static int crc32_pmull_update(struct shash_desc *desc, const u8 *data,u32 *crc =shash_desc_ctx (desc);unsigned int l; -if (may_use_simd ()) { +if (crypto_simd_usable ()) {if ((u32 )data %SCALE_F ) { l =min_t (u32, length,SCALE_F - ((u32 )data %SCALE_F ));
@@ -147,7 +148,7 @@ static int crc32c_pmull_update(struct shash_desc *desc, const u8 *data,u32 *crc =shash_desc_ctx (desc);unsigned int l; -if (may_use_simd ()) { +if (crypto_simd_usable ()) {if ((u32 )data %SCALE_F ) { l =min_t (u32, length,SCALE_F - ((u32 )data %SCALE_F ));
arch/arm/crypto/crct10dif-ce-glue.c +2 -1
@@ -15,6 +15,7 @@#include <linux/string.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <asm/neon.h> #include <asm/simd.h>
@@ -36,7 +37,7 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data, {u16 *crc =shash_desc_ctx (desc); -if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&may_use_simd ()) { +if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&crypto_simd_usable ()) {kernel_neon_begin (); *crc =crc_t10dif_pmull (*crc, data, length);kernel_neon_end ();
arch/arm/crypto/ghash-ce-glue.c +4 -3
@@ -14,6 +14,7 @@#include <asm/unaligned.h> #include <crypto/cryptd.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/gf128mul.h> #include <linux/cpufeature.h> #include <linux/crypto.h>
@@ -196,7 +197,7 @@ static int ghash_async_update(struct ahash_request *req)struct ghash_async_ctx *ctx =crypto_ahash_ctx (tfm);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);
@@ -214,7 +215,7 @@ static int ghash_async_final(struct ahash_request *req)struct ghash_async_ctx *ctx =crypto_ahash_ctx (tfm);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);
@@ -232,7 +233,7 @@ static int ghash_async_digest(struct ahash_request *req)struct ahash_request *cryptd_req =ahash_request_ctx (req);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);
arch/arm/crypto/nhpoly1305-neon-glue.c +2 -1
@@ -9,6 +9,7 @@#include <asm/neon.h> #include <asm/simd.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/nhpoly1305.h> #include <linux/module.h>
@@ -25,7 +26,7 @@ static void _nh_neon(const u32 *key, const u8 *message, size_t message_len,static int nhpoly1305_neon_update (struct shash_desc *desc,const u8 *src,unsigned int srclen) { -if (srclen <64 || !may_use_simd ()) +if (srclen <64 || !crypto_simd_usable ())return crypto_nhpoly1305_update (desc, src, srclen); do {
arch/arm/crypto/sha1-ce-glue.c +3 -2
@@ -9,6 +9,7 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha1_base.h> #include <linux/cpufeature.h>
@@ -33,7 +34,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data, {struct sha1_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd () || +if (!crypto_simd_usable () || (sctx->count %SHA1_BLOCK_SIZE ) + len <SHA1_BLOCK_SIZE )return sha1_update_arm (desc, data, len);
@@ -47,7 +48,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data,static int sha1_ce_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha1_finup_arm (desc, data, len, out);kernel_neon_begin ();
arch/arm/crypto/sha1_neon_glue.c +3 -2
@@ -19,6 +19,7 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/init.h> #include <linux/module.h> #include <linux/mm.h>
@@ -39,7 +40,7 @@ static int sha1_neon_update(struct shash_desc *desc, const u8 *data, {struct sha1_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd () || +if (!crypto_simd_usable () || (sctx->count %SHA1_BLOCK_SIZE ) + len <SHA1_BLOCK_SIZE )return sha1_update_arm (desc, data, len);
@@ -54,7 +55,7 @@ static int sha1_neon_update(struct shash_desc *desc, const u8 *data,static int sha1_neon_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha1_finup_arm (desc, data, len, out);kernel_neon_begin ();
arch/arm/crypto/sha2-ce-glue.c +3 -2
@@ -9,6 +9,7 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha256_base.h> #include <linux/cpufeature.h>
@@ -34,7 +35,7 @@ static int sha2_ce_update(struct shash_desc *desc, const u8 *data, {struct sha256_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd () || +if (!crypto_simd_usable () || (sctx->count %SHA256_BLOCK_SIZE ) + len <SHA256_BLOCK_SIZE )return crypto_sha256_arm_update (desc, data, len);
@@ -49,7 +50,7 @@ static int sha2_ce_update(struct shash_desc *desc, const u8 *data,static int sha2_ce_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha256_arm_finup (desc, data, len, out);kernel_neon_begin ();
arch/arm/crypto/sha256_neon_glue.c +3 -2
@@ -15,6 +15,7 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/cryptohash.h> #include <linux/types.h> #include <linux/string.h>
@@ -34,7 +35,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data, {struct sha256_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd () || +if (!crypto_simd_usable () || (sctx->count %SHA256_BLOCK_SIZE ) + len <SHA256_BLOCK_SIZE )return crypto_sha256_arm_update (desc, data, len);
@@ -49,7 +50,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,static int sha256_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha256_arm_finup (desc, data, len, out);kernel_neon_begin ();
arch/arm/crypto/sha512-neon-glue.c +3 -2
@@ -9,6 +9,7 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha512_base.h> #include <linux/crypto.h>
@@ -30,7 +31,7 @@ static int sha512_neon_update(struct shash_desc *desc, const u8 *data, {struct sha512_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd () || +if (!crypto_simd_usable () || (sctx->count[0 ] %SHA512_BLOCK_SIZE ) + len <SHA512_BLOCK_SIZE )return sha512_arm_update (desc, data, len);
@@ -45,7 +46,7 @@ static int sha512_neon_update(struct shash_desc *desc, const u8 *data,static int sha512_neon_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha512_arm_finup (desc, data, len, out);kernel_neon_begin ();
Signature · 1 line
2.21.0
Re: [PATCH 5/8] crypto: arm - convert to use crypto_simd_usable()
Quoted 7 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Replace all calls to
may_use_simd()in the arm crypto code withcrypto_simd_usable(), in order to allow testing the no-SIMD code paths.Signed-off-by Eric Biggers <[email protected]>
Quoted 307 lines
--- arch/arm/crypto/chacha-neon-glue.c | 5 +++-- arch/arm/crypto/crc32-ce-glue.c | 5 +++-- arch/arm/crypto/crct10dif-ce-glue.c | 3 ++- arch/arm/crypto/ghash-ce-glue.c | 7 ++++--- arch/arm/crypto/nhpoly1305-neon-glue.c | 3 ++- arch/arm/crypto/sha1-ce-glue.c | 5 +++-- arch/arm/crypto/sha1_neon_glue.c | 5 +++-- arch/arm/crypto/sha2-ce-glue.c | 5 +++-- arch/arm/crypto/sha256_neon_glue.c | 5 +++-- arch/arm/crypto/sha512-neon-glue.c | 5 +++-- 10 files changed, 29 insertions(+), 19 deletions(-)arch/arm/crypto/chacha-neon-glue.c +3
-2@@ -21,6 +21,7 @@#include <crypto/algapi.h> #include <crypto/chacha.h> +#include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <linux/kernel.h> #include <linux/module.h> @@ -93,7 +94,7 @@ static int chacha_neon(struct skcipher_request *req)struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm); -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_chacha_crypt (req);return chacha_neon_stream_xor (req, ctx, req->iv);@@ -107,7 +108,7 @@ static int xchacha_neon(struct skcipher_request *req)u32 state[16 ];u8 real_iv[16 ]; -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_xchacha_crypt (req);crypto_chacha_init (state, ctx, req->iv);arch/arm/crypto/crc32-ce-glue.c +3
-2@@ -16,6 +16,7 @@#include <linux/string.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <asm/hwcap.h> #include <asm/neon.h> @@ -113,7 +114,7 @@ static int crc32_pmull_update(struct shash_desc *desc, const u8 *data,u32 *crc =shash_desc_ctx (desc);unsigned int l; -if (may_use_simd ()) { +if (crypto_simd_usable ()) {if ((u32 )data %SCALE_F ) { l =min_t (u32, length,SCALE_F - ((u32 )data %SCALE_F ));@@ -147,7 +148,7 @@ static int crc32c_pmull_update(struct shash_desc *desc, const u8 *data,u32 *crc =shash_desc_ctx (desc);unsigned int l; -if (may_use_simd ()) { +if (crypto_simd_usable ()) {if ((u32 )data %SCALE_F ) { l =min_t (u32, length,SCALE_F - ((u32 )data %SCALE_F ));arch/arm/crypto/crct10dif-ce-glue.c +2
-1@@ -15,6 +15,7 @@#include <linux/string.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <asm/neon.h> #include <asm/simd.h> @@ -36,7 +37,7 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data, {u16 *crc =shash_desc_ctx (desc); -if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&may_use_simd ()) { +if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&crypto_simd_usable ()) {kernel_neon_begin (); *crc =crc_t10dif_pmull (*crc, data, length);kernel_neon_end ();arch/arm/crypto/ghash-ce-glue.c +4
-3@@ -14,6 +14,7 @@#include <asm/unaligned.h> #include <crypto/cryptd.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/gf128mul.h> #include <linux/cpufeature.h> #include <linux/crypto.h> @@ -196,7 +197,7 @@ static int ghash_async_update(struct ahash_request *req)struct ghash_async_ctx *ctx =crypto_ahash_ctx (tfm);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);@@ -214,7 +215,7 @@ static int ghash_async_final(struct ahash_request *req)struct ghash_async_ctx *ctx =crypto_ahash_ctx (tfm);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);@@ -232,7 +233,7 @@ static int ghash_async_digest(struct ahash_request *req)struct ahash_request *cryptd_req =ahash_request_ctx (req);struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) {memcpy (cryptd_req, req,sizeof (*req));ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base);arch/arm/crypto/nhpoly1305-neon-glue.c +2
-1@@ -9,6 +9,7 @@#include <asm/neon.h> #include <asm/simd.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/nhpoly1305.h> #include <linux/module.h> @@ -25,7 +26,7 @@ static void _nh_neon(const u32 *key, const u8 *message, size_t message_len,static int nhpoly1305_neon_update (struct shash_desc *desc,const u8 *src,unsigned int srclen) { -if (srclen <64 || !may_use_simd ()) +if (srclen <64 || !crypto_simd_usable ())return crypto_nhpoly1305_update (desc, src, srclen); do {arch/arm/crypto/sha1-ce-glue.c +3
-2@@ -9,6 +9,7 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha1_base.h> #include <linux/cpufeature.h> @@ -33,7 +34,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data, {struct sha1_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd () || +if (!crypto_simd_usable () || (sctx->count %SHA1_BLOCK_SIZE ) + len <SHA1_BLOCK_SIZE )return sha1_update_arm (desc, data, len);@@ -47,7 +48,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data,static int sha1_ce_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha1_finup_arm (desc, data, len, out);kernel_neon_begin ();arch/arm/crypto/sha1_neon_glue.c +3
-2@@ -19,6 +19,7 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/init.h> #include <linux/module.h> #include <linux/mm.h> @@ -39,7 +40,7 @@ static int sha1_neon_update(struct shash_desc *desc, const u8 *data, {struct sha1_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd () || +if (!crypto_simd_usable () || (sctx->count %SHA1_BLOCK_SIZE ) + len <SHA1_BLOCK_SIZE )return sha1_update_arm (desc, data, len);@@ -54,7 +55,7 @@ static int sha1_neon_update(struct shash_desc *desc, const u8 *data,static int sha1_neon_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha1_finup_arm (desc, data, len, out);kernel_neon_begin ();arch/arm/crypto/sha2-ce-glue.c +3
-2@@ -9,6 +9,7 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha256_base.h> #include <linux/cpufeature.h> @@ -34,7 +35,7 @@ static int sha2_ce_update(struct shash_desc *desc, const u8 *data, {struct sha256_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd () || +if (!crypto_simd_usable () || (sctx->count %SHA256_BLOCK_SIZE ) + len <SHA256_BLOCK_SIZE )return crypto_sha256_arm_update (desc, data, len);@@ -49,7 +50,7 @@ static int sha2_ce_update(struct shash_desc *desc, const u8 *data,static int sha2_ce_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha256_arm_finup (desc, data, len, out);kernel_neon_begin ();arch/arm/crypto/sha256_neon_glue.c +3
-2@@ -15,6 +15,7 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <linux/cryptohash.h> #include <linux/types.h> #include <linux/string.h> @@ -34,7 +35,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data, {struct sha256_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd () || +if (!crypto_simd_usable () || (sctx->count %SHA256_BLOCK_SIZE ) + len <SHA256_BLOCK_SIZE )return crypto_sha256_arm_update (desc, data, len);@@ -49,7 +50,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,static int sha256_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha256_arm_finup (desc, data, len, out);kernel_neon_begin ();arch/arm/crypto/sha512-neon-glue.c +3
-3@@ -9,6 +9,7 @@*/ #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha512_base.h> #include <linux/crypto.h> @@ -30,7 +31,7 @@ static int sha512_neon_update(struct shash_desc *desc, const u8 *data, {struct sha512_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd () || +if (!crypto_simd_usable () || (sctx->count[0 ] %SHA512_BLOCK_SIZE ) + len <SHA512_BLOCK_SIZE )return sha512_arm_update (desc, data, len);@@ -45,7 +46,7 @@ static int sha512_neon_update(struct shash_desc *desc, const u8 *data,static int sha512_neon_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha512_arm_finup (desc, data, len, out);kernel_neon_begin (); --2.21.0
PATCH 6/8 crypto: arm64 - convert to use crypto_simd_usable()
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Replace all calls to may_use_simd() in the arm64 crypto code with crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
--- arch/arm64/crypto/aes-ce-ccm-glue.c | 7 ++++--- arch/arm64/crypto/aes-ce-glue.c | 5 +++-- arch/arm64/crypto/aes-glue.c | 4 ++-- arch/arm64/crypto/aes-neonbs-glue.c | 2 +- arch/arm64/crypto/chacha-neon-glue.c | 5 +++-- arch/arm64/crypto/crct10dif-ce-glue.c | 5 +++-- arch/arm64/crypto/ghash-ce-glue.c | 7 ++++--- arch/arm64/crypto/nhpoly1305-neon-glue.c | 3 ++- arch/arm64/crypto/sha1-ce-glue.c | 7 ++++--- arch/arm64/crypto/sha2-ce-glue.c | 7 ++++--- arch/arm64/crypto/sha256-glue.c | 5 +++-- arch/arm64/crypto/sha3-ce-glue.c | 5 +++-- arch/arm64/crypto/sha512-ce-glue.c | 7 ++++--- arch/arm64/crypto/sm3-ce-glue.c | 7 ++++--- arch/arm64/crypto/sm4-ce-glue.c | 5 +++-- 15 files changed, 47 insertions(+), 34 deletions(-)
arch/arm64/crypto/aes-ce-ccm-glue.c +4 -3
@@ -14,6 +14,7 @@#include <crypto/aes.h> #include <crypto/scatterwalk.h> #include <crypto/internal/aead.h> +#include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <linux/module.h>
@@ -109,7 +110,7 @@ static int ccm_init_mac(struct aead_request *req, u8 maciv[], u32 msglen)static void ccm_update_mac (struct crypto_aes_ctx *key,u8 mac[],u8 const in[],u32 abytes,u32 *macp) { -if (may_use_simd ()) { +if (crypto_simd_usable ()) {kernel_neon_begin ();ce_aes_ccm_auth_data (mac, in, abytes, macp, key->key_enc,num_rounds (key));
@@ -255,7 +256,7 @@ static int ccm_encrypt(struct aead_request *req) err =skcipher_walk_aead_encrypt (&walk, req, false); -if (may_use_simd ()) { +if (crypto_simd_usable ()) {while (walk.nbytes) {u32 tail = walk.nbytes %AES_BLOCK_SIZE ;
@@ -313,7 +314,7 @@ static int ccm_decrypt(struct aead_request *req) err =skcipher_walk_aead_decrypt (&walk, req, false); -if (may_use_simd ()) { +if (crypto_simd_usable ()) {while (walk.nbytes) {u32 tail = walk.nbytes %AES_BLOCK_SIZE ;
arch/arm64/crypto/aes-ce-glue.c +3 -2
@@ -12,6 +12,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/aes.h> +#include <crypto/internal/simd.h> #include <linux/cpufeature.h> #include <linux/crypto.h> #include <linux/module.h>
@@ -52,7 +53,7 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) {struct crypto_aes_ctx *ctx =crypto_tfm_ctx (tfm); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {__aes_arm64_encrypt (ctx->key_enc, dst, src,num_rounds (ctx));return ; }
@@ -66,7 +67,7 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) {struct crypto_aes_ctx *ctx =crypto_tfm_ctx (tfm); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {__aes_arm64_decrypt (ctx->key_dec, dst, src,num_rounds (ctx));return ; }
arch/arm64/crypto/aes-glue.c +2 -2
@@ -405,7 +405,7 @@ static int ctr_encrypt_sync(struct skcipher_request *req)struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct crypto_aes_ctx *ctx =crypto_skcipher_ctx (tfm); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return aes_ctr_encrypt_fallback (ctx, req);return ctr_encrypt (req);
@@ -642,7 +642,7 @@ static void mac_do_update(struct crypto_aes_ctx *ctx, u8 const in[], int blocks, {int rounds =6 + ctx->key_length /4 ; -if (may_use_simd ()) { +if (crypto_simd_usable ()) {kernel_neon_begin ();aes_mac_update (in, ctx->key_enc, rounds, blocks, dg, enc_before, enc_after);
arch/arm64/crypto/aes-neonbs-glue.c +1 -1
@@ -288,7 +288,7 @@ static int ctr_encrypt_sync(struct skcipher_request *req)struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct aesbs_ctr_ctx *ctx =crypto_skcipher_ctx (tfm); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return aes_ctr_encrypt_fallback (&ctx->fallback, req);return ctr_encrypt (req);
arch/arm64/crypto/chacha-neon-glue.c +3 -2
@@ -21,6 +21,7 @@#include <crypto/algapi.h> #include <crypto/chacha.h> +#include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <linux/kernel.h> #include <linux/module.h>
@@ -90,7 +91,7 @@ static int chacha_neon(struct skcipher_request *req)struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm); -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_chacha_crypt (req);return chacha_neon_stream_xor (req, ctx, req->iv);
@@ -104,7 +105,7 @@ static int xchacha_neon(struct skcipher_request *req)u32 state[16 ];u8 real_iv[16 ]; -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_xchacha_crypt (req);crypto_chacha_init (state, ctx, req->iv);
arch/arm64/crypto/crct10dif-ce-glue.c +3 -2
@@ -16,6 +16,7 @@#include <linux/string.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <asm/neon.h> #include <asm/simd.h>
@@ -38,7 +39,7 @@ static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data, {u16 *crc =shash_desc_ctx (desc); -if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&may_use_simd ()) { +if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&crypto_simd_usable ()) {kernel_neon_begin (); *crc =crc_t10dif_pmull_p8 (*crc, data, length);kernel_neon_end ();
@@ -54,7 +55,7 @@ static int crct10dif_update_pmull_p64(struct shash_desc *desc, const u8 *data, {u16 *crc =shash_desc_ctx (desc); -if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&may_use_simd ()) { +if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&crypto_simd_usable ()) {kernel_neon_begin (); *crc =crc_t10dif_pmull_p64 (*crc, data, length);kernel_neon_end ();
arch/arm64/crypto/ghash-ce-glue.c +4 -3
@@ -17,6 +17,7 @@#include <crypto/gf128mul.h> #include <crypto/internal/aead.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <crypto/scatterwalk.h> #include <linux/cpufeature.h>
@@ -89,7 +90,7 @@ static void ghash_do_update(int blocks, u64 dg[], const char *src,struct ghash_key const *k, const char *head)) { - if (likely(may_use_simd())) { + if (likely(crypto_simd_usable())) {kernel_neon_begin ();simd_update (blocks, dg, src, key, head);kernel_neon_end ();
@@ -441,7 +442,7 @@ static int gcm_encrypt(struct aead_request *req) err =skcipher_walk_aead_encrypt (&walk, req, false); -if (likely (may_use_simd () && walk.total >=2 *AES_BLOCK_SIZE )) { +if (likely (crypto_simd_usable () && walk.total >=2 *AES_BLOCK_SIZE )) {u32 const *rk =NULL ;kernel_neon_begin ();
@@ -565,7 +566,7 @@ static int gcm_decrypt(struct aead_request *req) err =skcipher_walk_aead_decrypt (&walk, req, false); -if (likely (may_use_simd () && walk.total >=2 *AES_BLOCK_SIZE )) { +if (likely (crypto_simd_usable () && walk.total >=2 *AES_BLOCK_SIZE )) {u32 const *rk =NULL ;kernel_neon_begin ();
arch/arm64/crypto/nhpoly1305-neon-glue.c +2 -1
@@ -9,6 +9,7 @@#include <asm/neon.h> #include <asm/simd.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/nhpoly1305.h> #include <linux/module.h>
@@ -25,7 +26,7 @@ static void _nh_neon(const u32 *key, const u8 *message, size_t message_len,static int nhpoly1305_neon_update (struct shash_desc *desc,const u8 *src,unsigned int srclen) { -if (srclen <64 || !may_use_simd ()) +if (srclen <64 || !crypto_simd_usable ())return crypto_nhpoly1305_update (desc, src, srclen); do {
arch/arm64/crypto/sha1-ce-glue.c +4 -3
@@ -12,6 +12,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha1_base.h> #include <linux/cpufeature.h>
@@ -38,7 +39,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data, {struct sha1_ce_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha1_update (desc, data, len); sctx->finalize =0 ;
@@ -56,7 +57,7 @@ static int sha1_ce_finup(struct shash_desc *desc, const u8 *data,struct sha1_ce_state *sctx =shash_desc_ctx (desc);bool finalize = !sctx->sst.count && !(len %SHA1_BLOCK_SIZE ); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha1_finup (desc, data, len, out);/*
@@ -78,7 +79,7 @@ static int sha1_ce_final(struct shash_desc *desc, u8 *out) {struct sha1_ce_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha1_finup (desc,NULL ,0 , out); sctx->finalize =0 ;
arch/arm64/crypto/sha2-ce-glue.c +4 -3
@@ -12,6 +12,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha256_base.h> #include <linux/cpufeature.h>
@@ -42,7 +43,7 @@ static int sha256_ce_update(struct shash_desc *desc, const u8 *data, {struct sha256_ce_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha256_base_do_update (desc, data, len, (sha256_block_fn *)sha256_block_data_order);
@@ -61,7 +62,7 @@ static int sha256_ce_finup(struct shash_desc *desc, const u8 *data,struct sha256_ce_state *sctx =shash_desc_ctx (desc);bool finalize = !sctx->sst.count && !(len %SHA256_BLOCK_SIZE ); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {if (len)sha256_base_do_update (desc, data, len, (sha256_block_fn *)sha256_block_data_order);
@@ -90,7 +91,7 @@ static int sha256_ce_final(struct shash_desc *desc, u8 *out) {struct sha256_ce_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {sha256_base_do_finalize (desc, (sha256_block_fn *)sha256_block_data_order);return sha256_base_finish (desc, out);
arch/arm64/crypto/sha256-glue.c +3 -2
@@ -14,6 +14,7 @@#include <asm/neon.h> #include <asm/simd.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha256_base.h> #include <linux/cryptohash.h>
@@ -89,7 +90,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data, {struct sha256_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha256_base_do_update (desc, data, len, (sha256_block_fn *)sha256_block_data_order);
@@ -119,7 +120,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data,static int sha256_finup_neon (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {if (len)sha256_base_do_update (desc, data, len, (sha256_block_fn *)sha256_block_data_order);
arch/arm64/crypto/sha3-ce-glue.c +3 -2
@@ -14,6 +14,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha3.h> #include <linux/cpufeature.h> #include <linux/crypto.h>
@@ -32,7 +33,7 @@ static int sha3_update(struct shash_desc *desc, const u8 *data,struct sha3_state *sctx =shash_desc_ctx (desc);unsigned int digest_size =crypto_shash_digestsize (desc->tfm); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha3_update (desc, data, len);if ((sctx->partial + len) >= sctx->rsiz) {
@@ -76,7 +77,7 @@ static int sha3_final(struct shash_desc *desc, u8 *out)__le64 *digest = (__le64 *)out;int i; -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha3_final (desc, out); sctx->buf[sctx->partial++] =0x06 ;
arch/arm64/crypto/sha512-ce-glue.c +4 -3
@@ -13,6 +13,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha512_base.h> #include <linux/cpufeature.h>
@@ -31,7 +32,7 @@ asmlinkage void sha512_block_data_order(u64 *digest, u8 const *src, int blocks);static int sha512_ce_update (struct shash_desc *desc,const u8 *data,unsigned int len) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha512_base_do_update (desc, data, len, (sha512_block_fn *)sha512_block_data_order);
@@ -46,7 +47,7 @@ static int sha512_ce_update(struct shash_desc *desc, const u8 *data,static int sha512_ce_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {if (len)sha512_base_do_update (desc, data, len, (sha512_block_fn *)sha512_block_data_order);
@@ -65,7 +66,7 @@ static int sha512_ce_finup(struct shash_desc *desc, const u8 *data,static int sha512_ce_final (struct shash_desc *desc,u8 *out) { -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {sha512_base_do_finalize (desc, (sha512_block_fn *)sha512_block_data_order);return sha512_base_finish (desc, out);
arch/arm64/crypto/sm3-ce-glue.c +4 -3
@@ -12,6 +12,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sm3.h> #include <crypto/sm3_base.h> #include <linux/cpufeature.h>
@@ -28,7 +29,7 @@ asmlinkage void sm3_ce_transform(struct sm3_state *sst, u8 const *src,static int sm3_ce_update (struct shash_desc *desc,const u8 *data,unsigned int len) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sm3_update (desc, data, len);kernel_neon_begin ();
@@ -40,7 +41,7 @@ static int sm3_ce_update(struct shash_desc *desc, const u8 *data,static int sm3_ce_final (struct shash_desc *desc,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sm3_finup (desc,NULL ,0 , out);kernel_neon_begin ();
@@ -53,7 +54,7 @@ static int sm3_ce_final(struct shash_desc *desc, u8 *out)static int sm3_ce_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sm3_finup (desc, data, len, out);kernel_neon_begin ();
arch/arm64/crypto/sm4-ce-glue.c +3 -2
@@ -3,6 +3,7 @@#include <asm/neon.h> #include <asm/simd.h> #include <crypto/sm4.h> +#include <crypto/internal/simd.h> #include <linux/module.h> #include <linux/cpufeature.h> #include <linux/crypto.h>
@@ -20,7 +21,7 @@ static void sm4_ce_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) {const struct crypto_sm4_ctx *ctx =crypto_tfm_ctx (tfm); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {crypto_sm4_encrypt (tfm, out, in); }else {kernel_neon_begin ();
@@ -33,7 +34,7 @@ static void sm4_ce_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) {const struct crypto_sm4_ctx *ctx =crypto_tfm_ctx (tfm); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {crypto_sm4_decrypt (tfm, out, in); }else {kernel_neon_begin ();
Signature · 1 line
2.21.0
Re: [PATCH 6/8] crypto: arm64 - convert to use crypto_simd_usable()
Quoted 7 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Replace all calls to
may_use_simd()in the arm64 crypto code withcrypto_simd_usable(), in order to allow testing the no-SIMD code paths.Signed-off-by Eric Biggers <[email protected]>
Quoted 491 lines
--- arch/arm64/crypto/aes-ce-ccm-glue.c | 7 ++++--- arch/arm64/crypto/aes-ce-glue.c | 5 +++-- arch/arm64/crypto/aes-glue.c | 4 ++-- arch/arm64/crypto/aes-neonbs-glue.c | 2 +- arch/arm64/crypto/chacha-neon-glue.c | 5 +++-- arch/arm64/crypto/crct10dif-ce-glue.c | 5 +++-- arch/arm64/crypto/ghash-ce-glue.c | 7 ++++--- arch/arm64/crypto/nhpoly1305-neon-glue.c | 3 ++- arch/arm64/crypto/sha1-ce-glue.c | 7 ++++--- arch/arm64/crypto/sha2-ce-glue.c | 7 ++++--- arch/arm64/crypto/sha256-glue.c | 5 +++-- arch/arm64/crypto/sha3-ce-glue.c | 5 +++-- arch/arm64/crypto/sha512-ce-glue.c | 7 ++++--- arch/arm64/crypto/sm3-ce-glue.c | 7 ++++--- arch/arm64/crypto/sm4-ce-glue.c | 5 +++-- 15 files changed, 47 insertions(+), 34 deletions(-)arch/arm64/crypto/aes-ce-ccm-glue.c +4
-3@@ -14,6 +14,7 @@#include <crypto/aes.h> #include <crypto/scatterwalk.h> #include <crypto/internal/aead.h> +#include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <linux/module.h> @@ -109,7 +110,7 @@ static int ccm_init_mac(struct aead_request *req, u8 maciv[], u32 msglen)static void ccm_update_mac (struct crypto_aes_ctx *key,u8 mac[],u8 const in[],u32 abytes,u32 *macp) { -if (may_use_simd ()) { +if (crypto_simd_usable ()) {kernel_neon_begin ();ce_aes_ccm_auth_data (mac, in, abytes, macp, key->key_enc,num_rounds (key));@@ -255,7 +256,7 @@ static int ccm_encrypt(struct aead_request *req) err =skcipher_walk_aead_encrypt (&walk, req, false); -if (may_use_simd ()) { +if (crypto_simd_usable ()) {while (walk.nbytes) {u32 tail = walk.nbytes %AES_BLOCK_SIZE ;@@ -313,7 +314,7 @@ static int ccm_decrypt(struct aead_request *req) err =skcipher_walk_aead_decrypt (&walk, req, false); -if (may_use_simd ()) { +if (crypto_simd_usable ()) {while (walk.nbytes) {u32 tail = walk.nbytes %AES_BLOCK_SIZE ;arch/arm64/crypto/aes-ce-glue.c +3
-2@@ -12,6 +12,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/aes.h> +#include <crypto/internal/simd.h> #include <linux/cpufeature.h> #include <linux/crypto.h> #include <linux/module.h> @@ -52,7 +53,7 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) {struct crypto_aes_ctx *ctx =crypto_tfm_ctx (tfm); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {__aes_arm64_encrypt (ctx->key_enc, dst, src,num_rounds (ctx));return ; }@@ -66,7 +67,7 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) {struct crypto_aes_ctx *ctx =crypto_tfm_ctx (tfm); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {__aes_arm64_decrypt (ctx->key_dec, dst, src,num_rounds (ctx));return ; }arch/arm64/crypto/aes-glue.c +2
-2@@ -405,7 +405,7 @@ static int ctr_encrypt_sync(struct skcipher_request *req)struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct crypto_aes_ctx *ctx =crypto_skcipher_ctx (tfm); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return aes_ctr_encrypt_fallback (ctx, req);return ctr_encrypt (req);@@ -642,7 +642,7 @@ static void mac_do_update(struct crypto_aes_ctx *ctx, u8 const in[], int blocks, {int rounds =6 + ctx->key_length /4 ; -if (may_use_simd ()) { +if (crypto_simd_usable ()) {kernel_neon_begin ();aes_mac_update (in, ctx->key_enc, rounds, blocks, dg, enc_before, enc_after);arch/arm64/crypto/aes-neonbs-glue.c +1
-1@@ -288,7 +288,7 @@ static int ctr_encrypt_sync(struct skcipher_request *req)struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct aesbs_ctr_ctx *ctx =crypto_skcipher_ctx (tfm); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return aes_ctr_encrypt_fallback (&ctx->fallback, req);return ctr_encrypt (req);arch/arm64/crypto/chacha-neon-glue.c +3
-2@@ -21,6 +21,7 @@#include <crypto/algapi.h> #include <crypto/chacha.h> +#include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <linux/kernel.h> #include <linux/module.h> @@ -90,7 +91,7 @@ static int chacha_neon(struct skcipher_request *req)struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm); -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_chacha_crypt (req);return chacha_neon_stream_xor (req, ctx, req->iv);@@ -104,7 +105,7 @@ static int xchacha_neon(struct skcipher_request *req)u32 state[16 ];u8 real_iv[16 ]; -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !crypto_simd_usable ())return crypto_xchacha_crypt (req);crypto_chacha_init (state, ctx, req->iv);arch/arm64/crypto/crct10dif-ce-glue.c +3
-2@@ -16,6 +16,7 @@#include <linux/string.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <asm/neon.h> #include <asm/simd.h> @@ -38,7 +39,7 @@ static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data, {u16 *crc =shash_desc_ctx (desc); -if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&may_use_simd ()) { +if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&crypto_simd_usable ()) {kernel_neon_begin (); *crc =crc_t10dif_pmull_p8 (*crc, data, length);kernel_neon_end ();@@ -54,7 +55,7 @@ static int crct10dif_update_pmull_p64(struct shash_desc *desc, const u8 *data, {u16 *crc =shash_desc_ctx (desc); -if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&may_use_simd ()) { +if (length >=CRC_T10DIF_PMULL_CHUNK_SIZE &&crypto_simd_usable ()) {kernel_neon_begin (); *crc =crc_t10dif_pmull_p64 (*crc, data, length);kernel_neon_end ();arch/arm64/crypto/ghash-ce-glue.c +4
-3@@ -17,6 +17,7 @@#include <crypto/gf128mul.h> #include <crypto/internal/aead.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/internal/skcipher.h> #include <crypto/scatterwalk.h> #include <linux/cpufeature.h> @@ -89,7 +90,7 @@ static void ghash_do_update(int blocks, u64 dg[], const char *src,struct ghash_key const *k, const char *head)) { - if (likely(may_use_simd())) { + if (likely(crypto_simd_usable())) {kernel_neon_begin ();simd_update (blocks, dg, src, key, head);kernel_neon_end ();@@ -441,7 +442,7 @@ static int gcm_encrypt(struct aead_request *req) err =skcipher_walk_aead_encrypt (&walk, req, false); -if (likely (may_use_simd () && walk.total >=2 *AES_BLOCK_SIZE )) { +if (likely (crypto_simd_usable () && walk.total >=2 *AES_BLOCK_SIZE )) {u32 const *rk =NULL ;kernel_neon_begin ();@@ -565,7 +566,7 @@ static int gcm_decrypt(struct aead_request *req) err =skcipher_walk_aead_decrypt (&walk, req, false); -if (likely (may_use_simd () && walk.total >=2 *AES_BLOCK_SIZE )) { +if (likely (crypto_simd_usable () && walk.total >=2 *AES_BLOCK_SIZE )) {u32 const *rk =NULL ;kernel_neon_begin ();arch/arm64/crypto/nhpoly1305-neon-glue.c +2
-1@@ -9,6 +9,7 @@#include <asm/neon.h> #include <asm/simd.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/nhpoly1305.h> #include <linux/module.h> @@ -25,7 +26,7 @@ static void _nh_neon(const u32 *key, const u8 *message, size_t message_len,static int nhpoly1305_neon_update (struct shash_desc *desc,const u8 *src,unsigned int srclen) { -if (srclen <64 || !may_use_simd ()) +if (srclen <64 || !crypto_simd_usable ())return crypto_nhpoly1305_update (desc, src, srclen); do {arch/arm64/crypto/sha1-ce-glue.c +4
-3@@ -12,6 +12,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha1_base.h> #include <linux/cpufeature.h> @@ -38,7 +39,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data, {struct sha1_ce_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha1_update (desc, data, len); sctx->finalize =0 ;@@ -56,7 +57,7 @@ static int sha1_ce_finup(struct shash_desc *desc, const u8 *data,struct sha1_ce_state *sctx =shash_desc_ctx (desc);bool finalize = !sctx->sst.count && !(len %SHA1_BLOCK_SIZE ); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha1_finup (desc, data, len, out);/* @@ -78,7 +79,7 @@ static int sha1_ce_final(struct shash_desc *desc, u8 *out) {struct sha1_ce_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha1_finup (desc,NULL ,0 , out); sctx->finalize =0 ;arch/arm64/crypto/sha2-ce-glue.c +4
-3@@ -12,6 +12,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha256_base.h> #include <linux/cpufeature.h> @@ -42,7 +43,7 @@ static int sha256_ce_update(struct shash_desc *desc, const u8 *data, {struct sha256_ce_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha256_base_do_update (desc, data, len, (sha256_block_fn *)sha256_block_data_order);@@ -61,7 +62,7 @@ static int sha256_ce_finup(struct shash_desc *desc, const u8 *data,struct sha256_ce_state *sctx =shash_desc_ctx (desc);bool finalize = !sctx->sst.count && !(len %SHA256_BLOCK_SIZE ); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {if (len)sha256_base_do_update (desc, data, len, (sha256_block_fn *)sha256_block_data_order);@@ -90,7 +91,7 @@ static int sha256_ce_final(struct shash_desc *desc, u8 *out) {struct sha256_ce_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {sha256_base_do_finalize (desc, (sha256_block_fn *)sha256_block_data_order);return sha256_base_finish (desc, out);arch/arm64/crypto/sha256-glue.c +3
-2@@ -14,6 +14,7 @@#include <asm/neon.h> #include <asm/simd.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha256_base.h> #include <linux/cryptohash.h> @@ -89,7 +90,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data, {struct sha256_state *sctx =shash_desc_ctx (desc); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha256_base_do_update (desc, data, len, (sha256_block_fn *)sha256_block_data_order);@@ -119,7 +120,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data,static int sha256_finup_neon (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {if (len)sha256_base_do_update (desc, data, len, (sha256_block_fn *)sha256_block_data_order);arch/arm64/crypto/sha3-ce-glue.c +3
-2@@ -14,6 +14,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha3.h> #include <linux/cpufeature.h> #include <linux/crypto.h> @@ -32,7 +33,7 @@ static int sha3_update(struct shash_desc *desc, const u8 *data,struct sha3_state *sctx =shash_desc_ctx (desc);unsigned int digest_size =crypto_shash_digestsize (desc->tfm); -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha3_update (desc, data, len);if ((sctx->partial + len) >= sctx->rsiz) {@@ -76,7 +77,7 @@ static int sha3_final(struct shash_desc *desc, u8 *out)__le64 *digest = (__le64 *)out;int i; -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sha3_final (desc, out); sctx->buf[sctx->partial++] =0x06 ;arch/arm64/crypto/sha512-ce-glue.c +4
-3@@ -13,6 +13,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sha.h> #include <crypto/sha512_base.h> #include <linux/cpufeature.h> @@ -31,7 +32,7 @@ asmlinkage void sha512_block_data_order(u64 *digest, u8 const *src, int blocks);static int sha512_ce_update (struct shash_desc *desc,const u8 *data,unsigned int len) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return sha512_base_do_update (desc, data, len, (sha512_block_fn *)sha512_block_data_order);@@ -46,7 +47,7 @@ static int sha512_ce_update(struct shash_desc *desc, const u8 *data,static int sha512_ce_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {if (len)sha512_base_do_update (desc, data, len, (sha512_block_fn *)sha512_block_data_order);@@ -65,7 +66,7 @@ static int sha512_ce_finup(struct shash_desc *desc, const u8 *data,static int sha512_ce_final (struct shash_desc *desc,u8 *out) { -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {sha512_base_do_finalize (desc, (sha512_block_fn *)sha512_block_data_order);return sha512_base_finish (desc, out);arch/arm64/crypto/sm3-ce-glue.c +4
-3@@ -12,6 +12,7 @@#include <asm/simd.h> #include <asm/unaligned.h> #include <crypto/internal/hash.h> +#include <crypto/internal/simd.h> #include <crypto/sm3.h> #include <crypto/sm3_base.h> #include <linux/cpufeature.h> @@ -28,7 +29,7 @@ asmlinkage void sm3_ce_transform(struct sm3_state *sst, u8 const *src,static int sm3_ce_update (struct shash_desc *desc,const u8 *data,unsigned int len) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sm3_update (desc, data, len);kernel_neon_begin ();@@ -40,7 +41,7 @@ static int sm3_ce_update(struct shash_desc *desc, const u8 *data,static int sm3_ce_final (struct shash_desc *desc,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sm3_finup (desc,NULL ,0 , out);kernel_neon_begin ();@@ -53,7 +54,7 @@ static int sm3_ce_final(struct shash_desc *desc, u8 *out)static int sm3_ce_finup (struct shash_desc *desc,const u8 *data,unsigned int len,u8 *out) { -if (!may_use_simd ()) +if (!crypto_simd_usable ())return crypto_sm3_finup (desc, data, len, out);kernel_neon_begin ();arch/arm64/crypto/sm4-ce-glue.c +3
-3@@ -3,6 +3,7 @@#include <asm/neon.h> #include <asm/simd.h> #include <crypto/sm4.h> +#include <crypto/internal/simd.h> #include <linux/module.h> #include <linux/cpufeature.h> #include <linux/crypto.h> @@ -20,7 +21,7 @@ static void sm4_ce_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) {const struct crypto_sm4_ctx *ctx =crypto_tfm_ctx (tfm); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {crypto_sm4_encrypt (tfm, out, in); }else {kernel_neon_begin ();@@ -33,7 +34,7 @@ static void sm4_ce_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) {const struct crypto_sm4_ctx *ctx =crypto_tfm_ctx (tfm); -if (!may_use_simd ()) { +if (!crypto_simd_usable ()) {crypto_sm4_decrypt (tfm, out, in); }else {kernel_neon_begin (); --2.21.0
PATCH 7/8 crypto: simd - convert to use crypto_simd_usable()
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Replace all calls to may_use_simd() in the shared SIMD helpers with crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
--- crypto/simd.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
crypto/simd.c +4 -4
@@ -85,7 +85,7 @@ static int simd_skcipher_encrypt(struct skcipher_request *req) subreq =skcipher_request_ctx (req); *subreq = *req; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_skcipher_queued (ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base;else
@@ -106,7 +106,7 @@ static int simd_skcipher_decrypt(struct skcipher_request *req) subreq =skcipher_request_ctx (req); *subreq = *req; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_skcipher_queued (ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base;else
@@ -336,7 +336,7 @@ static int simd_aead_encrypt(struct aead_request *req) subreq =aead_request_ctx (req); *subreq = *req; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_aead_queued (ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base;else
@@ -357,7 +357,7 @@ static int simd_aead_decrypt(struct aead_request *req) subreq =aead_request_ctx (req); *subreq = *req; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_aead_queued (ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base;else
Signature · 1 line
2.21.0
Re: [PATCH 7/8] crypto: simd - convert to use crypto_simd_usable()
Quoted 7 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Replace all calls to
may_use_simd()in the shared SIMD helpers withcrypto_simd_usable(), in order to allow testing the no-SIMD code paths.Signed-off-by Eric Biggers <[email protected]>
Quoted 47 lines
--- crypto/simd.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)crypto/simd.c +4
-5@@ -85,7 +85,7 @@ static int simd_skcipher_encrypt(struct skcipher_request *req) subreq =skcipher_request_ctx (req); *subreq = *req; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_skcipher_queued (ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base;else @@ -106,7 +106,7 @@ static int simd_skcipher_decrypt(struct skcipher_request *req) subreq =skcipher_request_ctx (req); *subreq = *req; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_skcipher_queued (ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base;else @@ -336,7 +336,7 @@ static int simd_aead_encrypt(struct aead_request *req) subreq =aead_request_ctx (req); *subreq = *req; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_aead_queued (ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base;else @@ -357,7 +357,7 @@ static int simd_aead_decrypt(struct aead_request *req) subreq =aead_request_ctx (req); *subreq = *req; -if (!may_use_simd () || +if (!crypto_simd_usable () || (in_atomic () &&cryptd_aead_queued (ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base;else --2.21.0
PATCH 8/8 crypto: testmgr - test the !may_use_simd() fallback code
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
All crypto API algorithms are supposed to support the case where they are called in a context where SIMD instructions are unusable, e.g. IRQ context on some architectures. However, this isn't tested for by the self-tests, causing bugs to go undetected.
Now that all algorithms have been converted to use crypto_simd_usable(), update the self-tests to test the no-SIMD case. First, a bool testvec_config::nosimd is added. When set, the crypto operation is executed with preemption disabled and with crypto_simd_usable() mocked out to return false on the current CPU.
A bool test_sg_division::nosimd is also added. For hash algorithms it's honored by the corresponding ->update(). By setting just a subset of these bools, the case where some ->update()s are done in SIMD context and some are done in no-SIMD context is also tested.
These bools are then randomly set by generate_random_testvec_config().
For now, all no-SIMD testing is limited to the extra crypto self-tests, because it might be a bit too invasive for the regular self-tests. But this could be changed later.
This has already found bugs in the arm64 AES-GCM and ChaCha algorithms. This would have found some past bugs as well.
--- crypto/testmgr.c | 116 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 92 insertions(+), 24 deletions(-)
crypto/testmgr.c +92 -24
@@ -234,12 +234,14 @@ enum finalization_type {* @offset * @flush_type: for hashes, whether an update() should be done now vs. * continuing to accumulate data +* @nosimd: if doing the pending update(), do it with SIMD disabled? */ struct test_sg_division {unsigned int proportion_of_total;unsigned int offset;bool offset_relative_to_alignmask;enum flush_type flush_type; +bool nosimd; };/**
@@ -259,6 +261,7 @@ struct test_sg_division {* @iv_offset_relative_to_alignmask: if true, add the algorithm's alignmask to * the @iv_offset * @finalization_type: what finalization function to use for hashes +* @nosimd: execute with SIMD disabled? Requires !CRYPTO_TFM_REQ_MAY_SLEEP. */ struct testvec_config {const char *name;
@@ -269,6 +272,7 @@ struct testvec_config {unsigned int iv_offset;bool iv_offset_relative_to_alignmask;enum finalization_type finalization_type; +bool nosimd; };#define TESTVEC_CONFIG_NAMELEN 192
@@ -420,8 +424,11 @@ static unsigned int count_test_sg_divisions(const struct test_sg_division *divs)return ndivs; } +#define SGDIVS_HAVE_FLUSHES BIT(0) +#define SGDIVS_HAVE_NOSIMD BIT(1) +static bool valid_sg_divisions (const struct test_sg_division *divs, -unsigned int count,bool *any_flushes_ret) +unsigned int count,int *flags_ret) {unsigned int total =0 ;unsigned int i;
@@ -432,7 +439,9 @@ static bool valid_sg_divisions(const struct test_sg_division *divs,return false; total += divs[i].proportion_of_total;if (divs[i].flush_type !=FLUSH_TYPE_NONE ) - *any_flushes_ret = true; + *flags_ret |=SGDIVS_HAVE_FLUSHES ; +if (divs[i].nosimd) + *flags_ret |=SGDIVS_HAVE_NOSIMD ; }return total ==TEST_SG_TOTAL &&memchr_inv (&divs[i],0 , (count - i) *sizeof (divs[0 ])) ==NULL ;
@@ -445,19 +454,18 @@ static bool valid_sg_divisions(const struct test_sg_division *divs,*/ static bool valid_testvec_config (const struct testvec_config *cfg) { -bool any_flushes = false; +int flags =0 ;if (cfg->name ==NULL )return false;if (!valid_sg_divisions (cfg->src_divs,ARRAY_SIZE (cfg->src_divs), - &any_flushes)) + &flags))return false;if (cfg->dst_divs[0 ].proportion_of_total) {if (!valid_sg_divisions (cfg->dst_divs, -ARRAY_SIZE (cfg->dst_divs), - &any_flushes)) +ARRAY_SIZE (cfg->dst_divs), &flags))return false; }else {if (memchr_inv (cfg->dst_divs,0 ,sizeof (cfg->dst_divs)))
@@ -470,7 +478,12 @@ static bool valid_testvec_config(const struct testvec_config *cfg) MAX_ALGAPI_ALIGNMASK + 1)return false; -if (any_flushes && cfg->finalization_type ==FINALIZATION_TYPE_DIGEST ) +if ((flags & (SGDIVS_HAVE_FLUSHES |SGDIVS_HAVE_NOSIMD )) && + cfg->finalization_type ==FINALIZATION_TYPE_DIGEST ) +return false; + +if ((cfg->nosimd || (flags &SGDIVS_HAVE_NOSIMD )) && + (cfg->req_flags &CRYPTO_TFM_REQ_MAY_SLEEP ))return false;return true;
@@ -731,13 +744,14 @@ static int build_cipher_test_sglists(struct cipher_test_sglists *tsgls,#ifdef CONFIG_CRYPTO_MANAGER_EXTRA_TESTS static char *generate_random_sgl_divisions (struct test_sg_division *divs,size_t max_divs,char *p,char *end, -bool gen_flushes) +bool gen_flushes,u32 req_flags) {struct test_sg_division *div = divs;unsigned int remaining =TEST_SG_TOTAL ; do { unsigned int this_len; + const char *flushtype_str; if (div == &divs[max_divs - 1] || prandom_u32() % 2 == 0) this_len = remaining;
@@ -766,11 +780,31 @@ static char *generate_random_sgl_divisions(struct test_sg_division *divs, } } +if (div->flush_type !=FLUSH_TYPE_NONE && + !(req_flags &CRYPTO_TFM_REQ_MAY_SLEEP ) && +prandom_u32 () %2 ==0 ) + div->nosimd = true; + +switch (div->flush_type) { +case FLUSH_TYPE_FLUSH : +if (div->nosimd) + flushtype_str ="<flush,nosimd>" ; +else + flushtype_str ="<flush>" ; +break ; +case FLUSH_TYPE_REIMPORT : +if (div->nosimd) + flushtype_str ="<reimport,nosimd>" ; +else + flushtype_str ="<reimport>" ; +break ; +default : + flushtype_str ="" ; +break ; + } +BUILD_BUG_ON (TEST_SG_TOTAL !=10000 );/* for "%u.%u%%" */ - p += scnprintf(p, end - p, "%s%u.%u%%@%s+%u%s", - div->flush_type == FLUSH_TYPE_NONE ? "" : - div->flush_type == FLUSH_TYPE_FLUSH ? - "<flush> " : "<reimport> ", + p += scnprintf(p, end - p, "%s%u.%u%%@%s+%u%s", flushtype_str, this_len / 100, this_len % 100, div->offset_relative_to_alignmask ? "alignmask" : "",
@@ -820,18 +854,26 @@ static void generate_random_testvec_config(struct testvec_config *cfg,break ; } +if (!(cfg->req_flags &CRYPTO_TFM_REQ_MAY_SLEEP ) && +prandom_u32 () %2 ==0 ) { + cfg->nosimd = true; + p +=scnprintf (p, end - p," nosimd" ); + } + p +=scnprintf (p, end - p," src_divs=[" ); p =generate_random_sgl_divisions (cfg->src_divs,ARRAY_SIZE (cfg->src_divs), p, end, (cfg->finalization_type != -FINALIZATION_TYPE_DIGEST )); +FINALIZATION_TYPE_DIGEST ), + cfg->req_flags); p +=scnprintf (p, end - p,"]" );if (!cfg->inplace &&prandom_u32 () %2 ==0 ) { p +=scnprintf (p, end - p," dst_divs=[" ); p =generate_random_sgl_divisions (cfg->dst_divs,ARRAY_SIZE (cfg->dst_divs), - p, end, false); + p, end, false, + cfg->req_flags); p +=scnprintf (p, end - p,"]" ); }
@@ -864,6 +906,23 @@ static void crypto_reenable_simd_for_test(void) }#endif /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ +static int do_ahash_op (int (*op)(struct ahash_request *req), +struct ahash_request *req, +struct crypto_wait *wait,bool nosimd) +{ +int err; + +if (nosimd) +crypto_disable_simd_for_test (); + + err =op (req); + +if (nosimd) +crypto_reenable_simd_for_test (); + +return crypto_wait_req (err, wait); +} + static int check_nonfinal_hash_op(const char *op, int err, u8 *result, unsigned int digestsize, const char *driver, unsigned int vec_num,
@@ -938,7 +997,7 @@ static int test_hash_vec_cfg(const char *driver,ahash_request_set_callback (req, req_flags, crypto_req_done, &wait);ahash_request_set_crypt (req, tsgl->sgl, result, vec->psize); - err =crypto_wait_req (crypto_ahash_digest (req), &wait); + err =do_ahash_op (crypto_ahash_digest, req, &wait, cfg->nosimd);if (err) {pr_err ("alg: hash: %s digest() failed with err %d on test vector %u, cfg=\"%s\"\n" , driver, err, vec_num, cfg->name);
@@ -951,7 +1010,7 @@ static int test_hash_vec_cfg(const char *driver,ahash_request_set_callback (req, req_flags, crypto_req_done, &wait);ahash_request_set_crypt (req,NULL , result,0 ); - err =crypto_wait_req (crypto_ahash_init (req), &wait); + err =do_ahash_op (crypto_ahash_init, req, &wait, cfg->nosimd); err =check_nonfinal_hash_op ("init" , err, result, digestsize, driver, vec_num, cfg);if (err)
@@ -967,7 +1026,8 @@ static int test_hash_vec_cfg(const char *driver, crypto_req_done, &wait);ahash_request_set_crypt (req, pending_sgl, result, pending_len); - err =crypto_wait_req (crypto_ahash_update (req), &wait); + err =do_ahash_op (crypto_ahash_update, req, &wait, + divs[i]->nosimd); err =check_nonfinal_hash_op ("update" , err, result, digestsize, driver, vec_num, cfg);
@@ -1010,12 +1070,12 @@ static int test_hash_vec_cfg(const char *driver,ahash_request_set_crypt (req, pending_sgl, result, pending_len);if (cfg->finalization_type ==FINALIZATION_TYPE_FINAL ) {/* finish with update() and final() */ - err =crypto_wait_req (crypto_ahash_update (req), &wait); + err =do_ahash_op (crypto_ahash_update, req, &wait, cfg->nosimd); err =check_nonfinal_hash_op ("update" , err, result, digestsize, driver, vec_num, cfg);if (err)return err; - err =crypto_wait_req (crypto_ahash_final (req), &wait); + err =do_ahash_op (crypto_ahash_final, req, &wait, cfg->nosimd);if (err) {pr_err ("alg: hash: %s final() failed with err %d on test vector %u, cfg=\"%s\"\n" , driver, err, vec_num, cfg->name);
@@ -1023,7 +1083,7 @@ static int test_hash_vec_cfg(const char *driver, } }else {/* finish with finup() */ - err =crypto_wait_req (crypto_ahash_finup (req), &wait); + err =do_ahash_op (crypto_ahash_finup, req, &wait, cfg->nosimd);if (err) {pr_err ("alg: hash: %s finup() failed with err %d on test vector %u, cfg=\"%s\"\n" , driver, err, vec_num, cfg->name);
@@ -1259,8 +1319,12 @@ static int test_aead_vec_cfg(const char *driver, int enc,aead_request_set_crypt (req, tsgls->src.sgl_ptr, tsgls->dst.sgl_ptr, enc ? vec->plen : vec->clen, iv);aead_request_set_ad (req, vec->alen); - err =crypto_wait_req (enc ?crypto_aead_encrypt (req) : -crypto_aead_decrypt (req), &wait); +if (cfg->nosimd) +crypto_disable_simd_for_test (); + err = enc ?crypto_aead_encrypt (req) :crypto_aead_decrypt (req); +if (cfg->nosimd) +crypto_reenable_simd_for_test (); + err =crypto_wait_req (err, &wait);if (err) {if (err == -EBADMSG && vec->novrfy)return 0 ;
@@ -1594,8 +1658,12 @@ static int test_skcipher_vec_cfg(const char *driver, int enc,skcipher_request_set_callback (req, req_flags, crypto_req_done, &wait);skcipher_request_set_crypt (req, tsgls->src.sgl_ptr, tsgls->dst.sgl_ptr, vec->len, iv); - err =crypto_wait_req (enc ?crypto_skcipher_encrypt (req) : -crypto_skcipher_decrypt (req), &wait); +if (cfg->nosimd) +crypto_disable_simd_for_test (); + err = enc ?crypto_skcipher_encrypt (req) :crypto_skcipher_decrypt (req); +if (cfg->nosimd) +crypto_reenable_simd_for_test (); + err =crypto_wait_req (err, &wait);if (err) {pr_err ("alg: skcipher: %s %s failed with err %d on test vector %u, cfg=\"%s\"\n" , driver, op, err, vec_num, cfg->name);
Signature · 1 line
2.21.0
Re: [PATCH 8/8] crypto: testmgr - test the !may_use_simd() fallback code
Quoted 21 lines · Eric Biggers
From: Eric Biggers <[email protected]>
All crypto API algorithms are supposed to support the case where they are called in a context where SIMD instructions are unusable, e.g. IRQ context on some architectures. However, this isn't tested for by the self-tests, causing bugs to go undetected.
Now that all algorithms have been converted to use
crypto_simd_usable(), update the self-tests to test the no-SIMD case. First, a booltestvec_config::nosimdis added. When set, the crypto operation is executed with preemption disabled and withcrypto_simd_usable()mocked out to return false on the current CPU.A bool
test_sg_division::nosimdis also added. For hash algorithms it's honored by the corresponding ->update(). By setting just a subset of these bools, the case where some ->update()s are done in SIMD context and some are done in no-SIMD context is also tested.These bools are then randomly set by
generate_random_testvec_config().
Quoted 8 lines
For now, all no-SIMD testing is limited to the extra crypto self-tests, because it might be a bit too invasive for the regular self-tests. But this could be changed later.
This has already found bugs in the arm64 AES-GCM and ChaCha algorithms. This would have found some past bugs as well.
Signed-off-by Eric Biggers <[email protected]>
Quoted 290 lines
--- crypto/testmgr.c | 116 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 92 insertions(+), 24 deletions(-)crypto/testmgr.c +92
-25@@ -234,12 +234,14 @@ enum finalization_type {* @offset * @flush_type: for hashes, whether an update() should be done now vs. * continuing to accumulate data +* @nosimd: if doing the pending update(), do it with SIMD disabled? */ struct test_sg_division {unsigned int proportion_of_total;unsigned int offset;bool offset_relative_to_alignmask;enum flush_type flush_type; +bool nosimd; };/** @@ -259,6 +261,7 @@ struct test_sg_division {* @iv_offset_relative_to_alignmask: if true, add the algorithm's alignmask to * the @iv_offset * @finalization_type: what finalization function to use for hashes +* @nosimd: execute with SIMD disabled? Requires !CRYPTO_TFM_REQ_MAY_SLEEP. */ struct testvec_config {const char *name;@@ -269,6 +272,7 @@ struct testvec_config {unsigned int iv_offset;bool iv_offset_relative_to_alignmask;enum finalization_type finalization_type; +bool nosimd; };#define TESTVEC_CONFIG_NAMELEN 192@@ -420,8 +424,11 @@ static unsigned int count_test_sg_divisions(const struct test_sg_division *divs)return ndivs; } +#define SGDIVS_HAVE_FLUSHES BIT(0) +#define SGDIVS_HAVE_NOSIMD BIT(1) +static bool valid_sg_divisions (const struct test_sg_division *divs, -unsigned int count,bool *any_flushes_ret) +unsigned int count,int *flags_ret) {unsigned int total =0 ;unsigned int i;@@ -432,7 +439,9 @@ static bool valid_sg_divisions(const struct test_sg_division *divs,return false; total += divs[i].proportion_of_total;if (divs[i].flush_type !=FLUSH_TYPE_NONE ) - *any_flushes_ret = true; + *flags_ret |=SGDIVS_HAVE_FLUSHES ; +if (divs[i].nosimd) + *flags_ret |=SGDIVS_HAVE_NOSIMD ; }return total ==TEST_SG_TOTAL &&memchr_inv (&divs[i],0 , (count - i) *sizeof (divs[0 ])) ==NULL ;@@ -445,19 +454,18 @@ static bool valid_sg_divisions(const struct test_sg_division *divs,*/ static bool valid_testvec_config (const struct testvec_config *cfg) { -bool any_flushes = false; +int flags =0 ;if (cfg->name ==NULL )return false;if (!valid_sg_divisions (cfg->src_divs,ARRAY_SIZE (cfg->src_divs), - &any_flushes)) + &flags))return false;if (cfg->dst_divs[0 ].proportion_of_total) {if (!valid_sg_divisions (cfg->dst_divs, -ARRAY_SIZE (cfg->dst_divs), - &any_flushes)) +ARRAY_SIZE (cfg->dst_divs), &flags))return false; }else {if (memchr_inv (cfg->dst_divs,0 ,sizeof (cfg->dst_divs)))@@ -470,7 +478,12 @@ static bool valid_testvec_config(const struct testvec_config *cfg) MAX_ALGAPI_ALIGNMASK + 1)return false; -if (any_flushes && cfg->finalization_type ==FINALIZATION_TYPE_DIGEST ) +if ((flags & (SGDIVS_HAVE_FLUSHES |SGDIVS_HAVE_NOSIMD )) && + cfg->finalization_type ==FINALIZATION_TYPE_DIGEST ) +return false; + +if ((cfg->nosimd || (flags &SGDIVS_HAVE_NOSIMD )) && + (cfg->req_flags &CRYPTO_TFM_REQ_MAY_SLEEP ))return false;return true;@@ -731,13 +744,14 @@ static int build_cipher_test_sglists(struct cipher_test_sglists *tsgls,#ifdef CONFIG_CRYPTO_MANAGER_EXTRA_TESTS static char *generate_random_sgl_divisions (struct test_sg_division *divs,size_t max_divs,char *p,char *end, -bool gen_flushes) +bool gen_flushes,u32 req_flags) {struct test_sg_division *div = divs;unsigned int remaining =TEST_SG_TOTAL ; do { unsigned int this_len; + const char *flushtype_str; if (div == &divs[max_divs - 1] || prandom_u32() % 2 == 0) this_len = remaining;@@ -766,11 +780,31 @@ static char *generate_random_sgl_divisions(struct test_sg_division *divs, } } +if (div->flush_type !=FLUSH_TYPE_NONE && + !(req_flags &CRYPTO_TFM_REQ_MAY_SLEEP ) && +prandom_u32 () %2 ==0 ) + div->nosimd = true; + +switch (div->flush_type) { +case FLUSH_TYPE_FLUSH : +if (div->nosimd) + flushtype_str ="<flush,nosimd>" ; +else + flushtype_str ="<flush>" ; +break ; +case FLUSH_TYPE_REIMPORT : +if (div->nosimd) + flushtype_str ="<reimport,nosimd>" ; +else + flushtype_str ="<reimport>" ; +break ; +default : + flushtype_str ="" ; +break ; + } +BUILD_BUG_ON (TEST_SG_TOTAL !=10000 );/* for "%u.%u%%" */ - p += scnprintf(p, end - p, "%s%u.%u%%@%s+%u%s", - div->flush_type == FLUSH_TYPE_NONE ? "" : - div->flush_type == FLUSH_TYPE_FLUSH ? - "<flush> " : "<reimport> ", + p += scnprintf(p, end - p, "%s%u.%u%%@%s+%u%s", flushtype_str, this_len / 100, this_len % 100, div->offset_relative_to_alignmask ? "alignmask" : "",@@ -820,18 +854,26 @@ static void generate_random_testvec_config(struct testvec_config *cfg,break ; } +if (!(cfg->req_flags &CRYPTO_TFM_REQ_MAY_SLEEP ) && +prandom_u32 () %2 ==0 ) { + cfg->nosimd = true; + p +=scnprintf (p, end - p," nosimd" ); + } + p +=scnprintf (p, end - p," src_divs=[" ); p =generate_random_sgl_divisions (cfg->src_divs,ARRAY_SIZE (cfg->src_divs), p, end, (cfg->finalization_type != -FINALIZATION_TYPE_DIGEST )); +FINALIZATION_TYPE_DIGEST ), + cfg->req_flags); p +=scnprintf (p, end - p,"]" );if (!cfg->inplace &&prandom_u32 () %2 ==0 ) { p +=scnprintf (p, end - p," dst_divs=[" ); p =generate_random_sgl_divisions (cfg->dst_divs,ARRAY_SIZE (cfg->dst_divs), - p, end, false); + p, end, false, + cfg->req_flags); p +=scnprintf (p, end - p,"]" ); }@@ -864,6 +906,23 @@ static void crypto_reenable_simd_for_test(void) }#endif /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ +static int do_ahash_op (int (*op)(struct ahash_request *req), +struct ahash_request *req, +struct crypto_wait *wait,bool nosimd) +{ +int err; + +if (nosimd) +crypto_disable_simd_for_test (); + + err =op (req); + +if (nosimd) +crypto_reenable_simd_for_test (); + +return crypto_wait_req (err, wait); +} + static int check_nonfinal_hash_op(const char *op, int err, u8 *result, unsigned int digestsize, const char *driver, unsigned int vec_num,@@ -938,7 +997,7 @@ static int test_hash_vec_cfg(const char *driver,ahash_request_set_callback (req, req_flags, crypto_req_done, &wait);ahash_request_set_crypt (req, tsgl->sgl, result, vec->psize); - err =crypto_wait_req (crypto_ahash_digest (req), &wait); + err =do_ahash_op (crypto_ahash_digest, req, &wait, cfg->nosimd);if (err) {pr_err ("alg: hash: %s digest() failed with err %d on test vector %u, cfg=\"%s\"\n" , driver, err, vec_num, cfg->name);@@ -951,7 +1010,7 @@ static int test_hash_vec_cfg(const char *driver,ahash_request_set_callback (req, req_flags, crypto_req_done, &wait);ahash_request_set_crypt (req,NULL , result,0 ); - err =crypto_wait_req (crypto_ahash_init (req), &wait); + err =do_ahash_op (crypto_ahash_init, req, &wait, cfg->nosimd); err =check_nonfinal_hash_op ("init" , err, result, digestsize, driver, vec_num, cfg);if (err)@@ -967,7 +1026,8 @@ static int test_hash_vec_cfg(const char *driver, crypto_req_done, &wait);ahash_request_set_crypt (req, pending_sgl, result, pending_len); - err =crypto_wait_req (crypto_ahash_update (req), &wait); + err =do_ahash_op (crypto_ahash_update, req, &wait, + divs[i]->nosimd); err =check_nonfinal_hash_op ("update" , err, result, digestsize, driver, vec_num, cfg);@@ -1010,12 +1070,12 @@ static int test_hash_vec_cfg(const char *driver,ahash_request_set_crypt (req, pending_sgl, result, pending_len);if (cfg->finalization_type ==FINALIZATION_TYPE_FINAL ) {/* finish with update() and final() */ - err =crypto_wait_req (crypto_ahash_update (req), &wait); + err =do_ahash_op (crypto_ahash_update, req, &wait, cfg->nosimd); err =check_nonfinal_hash_op ("update" , err, result, digestsize, driver, vec_num, cfg);if (err)return err; - err =crypto_wait_req (crypto_ahash_final (req), &wait); + err =do_ahash_op (crypto_ahash_final, req, &wait, cfg->nosimd);if (err) {pr_err ("alg: hash: %s final() failed with err %d on test vector %u, cfg=\"%s\"\n" , driver, err, vec_num, cfg->name);@@ -1023,7 +1083,7 @@ static int test_hash_vec_cfg(const char *driver, } }else {/* finish with finup() */ - err =crypto_wait_req (crypto_ahash_finup (req), &wait); + err =do_ahash_op (crypto_ahash_finup, req, &wait, cfg->nosimd);if (err) {pr_err ("alg: hash: %s finup() failed with err %d on test vector %u, cfg=\"%s\"\n" , driver, err, vec_num, cfg->name);@@ -1259,8 +1319,12 @@ static int test_aead_vec_cfg(const char *driver, int enc,aead_request_set_crypt (req, tsgls->src.sgl_ptr, tsgls->dst.sgl_ptr, enc ? vec->plen : vec->clen, iv);aead_request_set_ad (req, vec->alen); - err =crypto_wait_req (enc ?crypto_aead_encrypt (req) : -crypto_aead_decrypt (req), &wait); +if (cfg->nosimd) +crypto_disable_simd_for_test (); + err = enc ?crypto_aead_encrypt (req) :crypto_aead_decrypt (req); +if (cfg->nosimd) +crypto_reenable_simd_for_test (); + err =crypto_wait_req (err, &wait);if (err) {if (err == -EBADMSG && vec->novrfy)return 0 ;@@ -1594,8 +1658,12 @@ static int test_skcipher_vec_cfg(const char *driver, int enc,skcipher_request_set_callback (req, req_flags, crypto_req_done, &wait);skcipher_request_set_crypt (req, tsgls->src.sgl_ptr, tsgls->dst.sgl_ptr, vec->len, iv); - err =crypto_wait_req (enc ?crypto_skcipher_encrypt (req) : -crypto_skcipher_decrypt (req), &wait); +if (cfg->nosimd) +crypto_disable_simd_for_test (); + err = enc ?crypto_skcipher_encrypt (req) :crypto_skcipher_decrypt (req); +if (cfg->nosimd) +crypto_reenable_simd_for_test (); + err =crypto_wait_req (err, &wait);if (err) {pr_err ("alg: skcipher: %s %s failed with err %d on test vector %u, cfg=\"%s\"\n" , driver, op, err, vec_num, cfg->name); --2.21.0
Quoted 11 lines · Eric Biggers
All crypto API algorithms are supposed to support the case where they are called in a context where SIMD instructions are unusable, e.g. IRQ context on some architectures. However, this isn't tested for by the self-tests, causing bugs to go undetected.
This patch series therefore updates the self-tests to test the no-SIMD code. It works by converting all
may_use_simd()checks to a new macrocrypto_simd_usable(), which also returns false when the self-tests have disabled SIMD in crypto algorithms on the current CPU for test purposes.
Quoted 8 lines
For now, all no-SIMD testing is limited to the extra crypto self-tests, because it might be a bit too invasive for the regular self-tests. But this could be changed later.
This already found bugs in the arm64 implementations of AES-GCM and ChaCha. These are fixed by the first two patches. Following this, the tests pass on x86, arm, and arm64.
Thanks a lot for doing this Eric.
I pushed your branch to kernelci again, let's see if anything else falls out (although not as likely, given that this shouldn't affect h/w accelerators)
Quoted 62 lines
This patch series is based on top of my other pending patch series "crypto: add SIMD helpers for AEADs". It can also be found in git at:
URL: https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git
Branch: crypto-nosimd-testsEric Biggers (8):
crypto: chacha-generic - fix use as arm64 no-NEON fallback
crypto: arm64/gcm-aes-ce - fix no-NEON fallback code
crypto: simd,testmgr - introducecrypto_simd_usable()
crypto: x86 - convert to usecrypto_simd_usable()
crypto: arm - convert to usecrypto_simd_usable()
crypto: arm64 - convert to usecrypto_simd_usable()
crypto: simd - convert to usecrypto_simd_usable()
crypto: testmgr - test the !may_use_simd() fallback codearch/arm/crypto/chacha-neon-glue.c | 5 +- arch/arm/crypto/crc32-ce-glue.c | 5 +- arch/arm/crypto/crct10dif-ce-glue.c | 3 +- arch/arm/crypto/ghash-ce-glue.c | 7 +- arch/arm/crypto/nhpoly1305-neon-glue.c | 3 +- arch/arm/crypto/sha1-ce-glue.c | 5 +- arch/arm/crypto/sha1_neon_glue.c | 5 +- arch/arm/crypto/sha2-ce-glue.c | 5 +- arch/arm/crypto/sha256_neon_glue.c | 5 +- arch/arm/crypto/sha512-neon-glue.c | 5 +- arch/arm64/crypto/aes-ce-ccm-glue.c | 7 +- arch/arm64/crypto/aes-ce-glue.c | 5 +- arch/arm64/crypto/aes-glue.c | 4 +- arch/arm64/crypto/aes-neonbs-glue.c | 2 +- arch/arm64/crypto/chacha-neon-glue.c | 5 +- arch/arm64/crypto/crct10dif-ce-glue.c | 5 +- arch/arm64/crypto/ghash-ce-glue.c | 17 ++- arch/arm64/crypto/nhpoly1305-neon-glue.c | 3 +- arch/arm64/crypto/sha1-ce-glue.c | 7 +- arch/arm64/crypto/sha2-ce-glue.c | 7 +- arch/arm64/crypto/sha256-glue.c | 5 +- arch/arm64/crypto/sha3-ce-glue.c | 5 +- arch/arm64/crypto/sha512-ce-glue.c | 7 +- arch/arm64/crypto/sm3-ce-glue.c | 7 +- arch/arm64/crypto/sm4-ce-glue.c | 5 +- arch/x86/crypto/aesni-intel_glue.c | 8 +- arch/x86/crypto/chacha_glue.c | 6 +- arch/x86/crypto/crc32-pclmul_glue.c | 5 +- arch/x86/crypto/crc32c-intel_glue.c | 7 +- arch/x86/crypto/crct10dif-pclmul_glue.c | 7 +- arch/x86/crypto/ghash-clmulni-intel_glue.c | 9 +- arch/x86/crypto/nhpoly1305-avx2-glue.c | 5 +- arch/x86/crypto/nhpoly1305-sse2-glue.c | 5 +- arch/x86/crypto/poly1305_glue.c | 4 +- arch/x86/crypto/sha1_ssse3_glue.c | 7 +- arch/x86/crypto/sha256_ssse3_glue.c | 7 +- arch/x86/crypto/sha512_ssse3_glue.c | 10 +- crypto/chacha_generic.c | 2 +- crypto/simd.c | 8 +- crypto/testmgr.c | 142 +++++++++++++++++---- include/crypto/internal/simd.h | 24 ++++ 41 files changed, 272 insertions(+), 123 deletions(-)--
2.21.0
Herbert Xu
maintainerQuoted 68 lines · Eric Biggers
All crypto API algorithms are supposed to support the case where they are called in a context where SIMD instructions are unusable, e.g. IRQ context on some architectures. However, this isn't tested for by the self-tests, causing bugs to go undetected.
This patch series therefore updates the self-tests to test the no-SIMD code. It works by converting all
may_use_simd()checks to a new macrocrypto_simd_usable(), which also returns false when the self-tests have disabled SIMD in crypto algorithms on the current CPU for test purposes.For now, all no-SIMD testing is limited to the extra crypto self-tests, because it might be a bit too invasive for the regular self-tests. But this could be changed later.
This already found bugs in the arm64 implementations of AES-GCM and ChaCha. These are fixed by the first two patches. Following this, the tests pass on x86, arm, and arm64.
This patch series is based on top of my other pending patch series "crypto: add SIMD helpers for AEADs". It can also be found in git at:
URL: https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git
Branch: crypto-nosimd-testsEric Biggers (8):
crypto: chacha-generic - fix use as arm64 no-NEON fallback
crypto: arm64/gcm-aes-ce - fix no-NEON fallback code
crypto: simd,testmgr - introducecrypto_simd_usable()
crypto: x86 - convert to usecrypto_simd_usable()
crypto: arm - convert to usecrypto_simd_usable()
crypto: arm64 - convert to usecrypto_simd_usable()
crypto: simd - convert to usecrypto_simd_usable()
crypto: testmgr - test the !may_use_simd() fallback codearch/arm/crypto/chacha-neon-glue.c | 5 +- arch/arm/crypto/crc32-ce-glue.c | 5 +- arch/arm/crypto/crct10dif-ce-glue.c | 3 +- arch/arm/crypto/ghash-ce-glue.c | 7 +- arch/arm/crypto/nhpoly1305-neon-glue.c | 3 +- arch/arm/crypto/sha1-ce-glue.c | 5 +- arch/arm/crypto/sha1_neon_glue.c | 5 +- arch/arm/crypto/sha2-ce-glue.c | 5 +- arch/arm/crypto/sha256_neon_glue.c | 5 +- arch/arm/crypto/sha512-neon-glue.c | 5 +- arch/arm64/crypto/aes-ce-ccm-glue.c | 7 +- arch/arm64/crypto/aes-ce-glue.c | 5 +- arch/arm64/crypto/aes-glue.c | 4 +- arch/arm64/crypto/aes-neonbs-glue.c | 2 +- arch/arm64/crypto/chacha-neon-glue.c | 5 +- arch/arm64/crypto/crct10dif-ce-glue.c | 5 +- arch/arm64/crypto/ghash-ce-glue.c | 17 ++- arch/arm64/crypto/nhpoly1305-neon-glue.c | 3 +- arch/arm64/crypto/sha1-ce-glue.c | 7 +- arch/arm64/crypto/sha2-ce-glue.c | 7 +- arch/arm64/crypto/sha256-glue.c | 5 +- arch/arm64/crypto/sha3-ce-glue.c | 5 +- arch/arm64/crypto/sha512-ce-glue.c | 7 +- arch/arm64/crypto/sm3-ce-glue.c | 7 +- arch/arm64/crypto/sm4-ce-glue.c | 5 +- arch/x86/crypto/aesni-intel_glue.c | 8 +- arch/x86/crypto/chacha_glue.c | 6 +- arch/x86/crypto/crc32-pclmul_glue.c | 5 +- arch/x86/crypto/crc32c-intel_glue.c | 7 +- arch/x86/crypto/crct10dif-pclmul_glue.c | 7 +- arch/x86/crypto/ghash-clmulni-intel_glue.c | 9 +- arch/x86/crypto/nhpoly1305-avx2-glue.c | 5 +- arch/x86/crypto/nhpoly1305-sse2-glue.c | 5 +- arch/x86/crypto/poly1305_glue.c | 4 +-
Quoted 8 lines
arch/x86/crypto/sha1_ssse3_glue.c | 7 +- arch/x86/crypto/sha256_ssse3_glue.c | 7 +- arch/x86/crypto/sha512_ssse3_glue.c | 10 +- crypto/chacha_generic.c | 2 +- crypto/simd.c | 8 +- crypto/testmgr.c | 142 +++++++++++++++++---- include/crypto/internal/simd.h | 24 ++++ 41 files changed, 272 insertions(+), 123 deletions(-)
All applied. Thanks.
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt