ARM: NEON based fast(er) AES in CBC/CTR/XTS modes
Subsystems (3)
cover ARM: NEON based fast(er) AES in CBC/CTR/XTS modes
This implementation of the AES algorithm gives around 45% speedup on Cortex-A15 for CTR mode and for XTS in encryption mode. Both CBC and XTS in decryption mode are slightly faster (5 - 10% on Cortex-A15). [As CBC in encryption mode can only be performed sequentially, there is no speedup in this case.]
Unlike the core AES cipher (on which this module also depends), this algorithm uses bit slicing to process up to 8 blocks in parallel in constant time. This algorithm does not rely on any lookup tables so it is believed to be invulnerable to cache timing attacks.
The core code has been adopted from the OpenSSL project (in collaboration with the original author, on cc). For ease of maintenance, this version is identical to the upstream OpenSSL code, i.e., all modifications that were required to make it suitable for inclusion into the kernel have already been merged upstream.
This code passes the builtin test 'modprobe tcrypt.ko mode=10' in both ARM and Thumb-2 modes.
Note to reviewers:
Reviewing the file aesbs-core.S may be a bit overwhelming, so if there are any questions or concerns, please refer to the link below. This is the original Perl script that gets called by OpenSSL's build system during their build to generate the .S file on the fly. [In the case of OpenSSL, this is used in some cases to target different assemblers or ABIs]. This arrangement is not suitable (or required) for the kernel, so I have taken the generated .S file instead.
http://git.openssl.org/gitweb/?p=openssl.git;f=crypto/aes/asm/bsaes-armv7.pl;a=blob
Note to integrators:
While this implementation is significantly faster, especially in CTR mode, it is unclear whether the net impact on power efficiency is favorable or not, so please refrain from making any assumptions to that effect.
Ard Biesheuvel (4):
crypto: create generic version of ablk_helper
ARM: pull in <asm/simd.h> from asm-generic
ARM: move AES typedefs and function prototypes to separate header
ARM: add support for bit sliced AES using NEON instructions
arch/arm/crypto/Makefile | 6 +- arch/arm/crypto/aes_glue.c | 22 +- arch/arm/crypto/aes_glue.h | 19 + arch/arm/crypto/aesbs-core.S | 2603 ++++++++++++++++++++++++++++++++++++++++++ arch/arm/crypto/aesbs-glue.c | 449 ++++++++ arch/arm/include/asm/Kbuild | 1 + crypto/Kconfig | 20 + crypto/Makefile | 1 + crypto/ablk_helper.c | 150 +++ include/asm-generic/simd.h | 14 + include/crypto/ablk_helper.h | 31 + 11 files changed, 3298 insertions(+), 18 deletions(-) create mode 100644 arch/arm/crypto/aes_glue.h create mode 100644 arch/arm/crypto/aesbs-core.S create mode 100644 arch/arm/crypto/aesbs-glue.c create mode 100644 crypto/ablk_helper.c create mode 100644 include/asm-generic/simd.h create mode 100644 include/crypto/ablk_helper.h
Signature · 1 line
1.8.1.2
PATCH 1/4 crypto: create generic version of ablk_helper
Create a generic version of ablk_helper so it can be reused by other architectures.
--- crypto/Kconfig | 4 ++ crypto/Makefile | 1 + crypto/ablk_helper.c | 150 +++++++++++++++++++++++++++++++++++++++++++ include/asm-generic/simd.h | 14 ++++ include/crypto/ablk_helper.h | 31 +++++++++ 5 files changed, 200 insertions(+) create mode 100644 crypto/ablk_helper.c create mode 100644 include/asm-generic/simd.h create mode 100644 include/crypto/ablk_helper.h
crypto/Kconfig +4 -0
@@ -179,6 +179,10 @@ config CRYPTO_ABLK_HELPER_X86 depends on X86 select CRYPTO_CRYPTD +config CRYPTO_ABLK_HELPER + tristate + select CRYPTO_CRYPTD + config CRYPTO_GLUE_HELPER_X86 tristate depends on X86
crypto/Makefile +1 -0
@@ -104,3 +104,4 @@ obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.o obj-$(CONFIG_XOR_BLOCKS) += xor.o obj-$(CONFIG_ASYNC_CORE) += async_tx/ obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys/ +obj-$(CONFIG_CRYPTO_ABLK_HELPER) += ablk_helper.o
crypto/ablk_helper.c new file +150 -0
150 lines not on this page · raw message
include/asm-generic/simd.h new file +14 -0
@@ -0,0 +1,14 @@ + +#include <linux/hardirq.h> + +/* +* may_use_simd - whether it is allowable at this time to issue SIMD +* instructions or access the SIMD register file +* +* As architectures typically don't preserve the SIMD register file when +* taking an interrupt, !in_interrupt() should be a reasonable default. +*/ +static __must_check inline bool may_use_simd (void ) +{ +return !in_interrupt (); +}
include/crypto/ablk_helper.h new file +31 -0
@@ -0,0 +1,31 @@ +/* +* Shared async block cipher helpers +*/ + +#ifndef _CRYPTO_ABLK_HELPER_H +#define _CRYPTO_ABLK_HELPER_H + +#include <linux/crypto.h> +#include <linux/kernel.h> +#include <crypto/cryptd.h> + +struct async_helper_ctx { +struct cryptd_ablkcipher *cryptd_tfm; +}; + +int ablk_set_key (struct crypto_ablkcipher *tfm,const u8 *key, +unsigned int key_len); + +int __ablk_encrypt (struct ablkcipher_request *req); + +int ablk_encrypt (struct ablkcipher_request *req); + +int ablk_decrypt (struct ablkcipher_request *req); + +void ablk_exit (struct crypto_tfm *tfm); + +int ablk_init_common (struct crypto_tfm *tfm,const char *drv_name); + +int ablk_init (struct crypto_tfm *tfm); + +#endif /* _CRYPTO_ABLK_HELPER_H */
Signature · 1 line
1.8.1.2
PATCH 1/4 crypto: create generic version of ablk_helper
Quoted 5 lines · Ard Biesheuvel
Create a generic version of
ablk_helperso it can be reused by other architectures.Acked-by Jussi Kivilinna <[email protected]>Signed-off-by Ard Biesheuvel <[email protected]>
Why resent this patch here when this was in the earlier patchset?
http://marc.info/?l=linux-crypto-vger&m=137966378813818&w=2
-Jussi
Quoted 249 lines
--- crypto/Kconfig | 4 ++ crypto/Makefile | 1 + crypto/ablk_helper.c | 150 +++++++++++++++++++++++++++++++++++++++++++ include/asm-generic/simd.h | 14 ++++ include/crypto/ablk_helper.h | 31 +++++++++ 5 files changed, 200 insertions(+) create mode 100644 crypto/ablk_helper.c create mode 100644 include/asm-generic/simd.h create mode 100644 include/crypto/ablk_helper.hcrypto/Kconfig +4
-0@@ -179,6 +179,10 @@ config CRYPTO_ABLK_HELPER_X86 depends on X86 select CRYPTO_CRYPTD +config CRYPTO_ABLK_HELPER + tristate + select CRYPTO_CRYPTD + config CRYPTO_GLUE_HELPER_X86 tristate depends on X86crypto/Makefile +1
-0@@ -104,3 +104,4 @@ obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.o obj-$(CONFIG_XOR_BLOCKS) += xor.o obj-$(CONFIG_ASYNC_CORE) += async_tx/ obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys/ +obj-$(CONFIG_CRYPTO_ABLK_HELPER) += ablk_helper.ocrypto/ablk_helper.c new file +150
-0@@ -0,0 +1,150 @@ +/* +* Shared async block cipher helpers +* +* Copyright (c) 2012 Jussi Kivilinna <[email protected]> +* +* Based on aesni-intel_glue.c by: +* Copyright (C) 2008, Intel Corp. +* Author: Huang Ying <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 +* USA +* +*/ + +#include <linux/kernel.h> +#include <linux/crypto.h> +#include <linux/init.h> +#include <linux/module.h> +#include <linux/hardirq.h> +#include <crypto/algapi.h> +#include <crypto/cryptd.h> +#include <crypto/ablk_helper.h> +#include <asm/simd.h> + +int ablk_set_key (struct crypto_ablkcipher *tfm,const u8 *key, +unsigned int key_len) +{ +struct async_helper_ctx *ctx =crypto_ablkcipher_ctx (tfm); +struct crypto_ablkcipher *child = &ctx->cryptd_tfm->base; +int err; + +crypto_ablkcipher_clear_flags (child,CRYPTO_TFM_REQ_MASK ); +crypto_ablkcipher_set_flags (child,crypto_ablkcipher_get_flags (tfm) + &CRYPTO_TFM_REQ_MASK ); + err =crypto_ablkcipher_setkey (child, key, key_len); +crypto_ablkcipher_set_flags (tfm,crypto_ablkcipher_get_flags (child) + &CRYPTO_TFM_RES_MASK ); +return err; +} +EXPORT_SYMBOL_GPL (ablk_set_key); + +int __ablk_encrypt (struct ablkcipher_request *req) +{ +struct crypto_ablkcipher *tfm =crypto_ablkcipher_reqtfm (req); +struct async_helper_ctx *ctx =crypto_ablkcipher_ctx (tfm); +struct blkcipher_desc desc; + + desc.tfm =cryptd_ablkcipher_child (ctx->cryptd_tfm); + desc.info = req->info; + desc.flags =0 ; + +return crypto_blkcipher_crt (desc.tfm)->encrypt ( + &desc, req->dst, req->src, req->nbytes); +} +EXPORT_SYMBOL_GPL (__ablk_encrypt); + +int ablk_encrypt (struct ablkcipher_request *req) +{ +struct crypto_ablkcipher *tfm =crypto_ablkcipher_reqtfm (req); +struct async_helper_ctx *ctx =crypto_ablkcipher_ctx (tfm); + +if (!may_use_simd ()) { +struct ablkcipher_request *cryptd_req = +ablkcipher_request_ctx (req); + +memcpy (cryptd_req, req,sizeof (*req)); +ablkcipher_request_set_tfm (cryptd_req, &ctx->cryptd_tfm->base); + +return crypto_ablkcipher_encrypt (cryptd_req); + }else { +return __ablk_encrypt (req); + } +} +EXPORT_SYMBOL_GPL (ablk_encrypt); + +int ablk_decrypt (struct ablkcipher_request *req) +{ +struct crypto_ablkcipher *tfm =crypto_ablkcipher_reqtfm (req); +struct async_helper_ctx *ctx =crypto_ablkcipher_ctx (tfm); + +if (!may_use_simd ()) { +struct ablkcipher_request *cryptd_req = +ablkcipher_request_ctx (req); + +memcpy (cryptd_req, req,sizeof (*req)); +ablkcipher_request_set_tfm (cryptd_req, &ctx->cryptd_tfm->base); + +return crypto_ablkcipher_decrypt (cryptd_req); + }else { +struct blkcipher_desc desc; + + desc.tfm =cryptd_ablkcipher_child (ctx->cryptd_tfm); + desc.info = req->info; + desc.flags =0 ; + +return crypto_blkcipher_crt (desc.tfm)->decrypt ( + &desc, req->dst, req->src, req->nbytes); + } +} +EXPORT_SYMBOL_GPL (ablk_decrypt); + +void ablk_exit (struct crypto_tfm *tfm) +{ +struct async_helper_ctx *ctx =crypto_tfm_ctx (tfm); + +cryptd_free_ablkcipher (ctx->cryptd_tfm); +} +EXPORT_SYMBOL_GPL (ablk_exit); + +int ablk_init_common (struct crypto_tfm *tfm,const char *drv_name) +{ +struct async_helper_ctx *ctx =crypto_tfm_ctx (tfm); +struct cryptd_ablkcipher *cryptd_tfm; + + cryptd_tfm =cryptd_alloc_ablkcipher (drv_name,0 ,0 ); +if (IS_ERR (cryptd_tfm)) +return PTR_ERR (cryptd_tfm); + + ctx->cryptd_tfm = cryptd_tfm; + tfm->crt_ablkcipher.reqsize =sizeof (struct ablkcipher_request ) + +crypto_ablkcipher_reqsize (&cryptd_tfm->base); + +return 0 ; +} +EXPORT_SYMBOL_GPL (ablk_init_common); + +int ablk_init (struct crypto_tfm *tfm) +{ +char drv_name[CRYPTO_MAX_ALG_NAME ]; + +snprintf (drv_name,sizeof (drv_name),"__driver-%s" , +crypto_tfm_alg_driver_name (tfm)); + +return ablk_init_common (tfm, drv_name); +} +EXPORT_SYMBOL_GPL (ablk_init); + +MODULE_LICENSE ("GPL" );include/asm-generic/simd.h new file +14
-0@@ -0,0 +1,14 @@ + +#include <linux/hardirq.h> + +/* +* may_use_simd - whether it is allowable at this time to issue SIMD +* instructions or access the SIMD register file +* +* As architectures typically don't preserve the SIMD register file when +* taking an interrupt, !in_interrupt() should be a reasonable default. +*/ +static __must_check inline bool may_use_simd (void ) +{ +return !in_interrupt (); +}include/crypto/ablk_helper.h new file +31
-0@@ -0,0 +1,31 @@ +/* +* Shared async block cipher helpers +*/ + +#ifndef _CRYPTO_ABLK_HELPER_H +#define _CRYPTO_ABLK_HELPER_H + +#include <linux/crypto.h> +#include <linux/kernel.h> +#include <crypto/cryptd.h> + +struct async_helper_ctx { +struct cryptd_ablkcipher *cryptd_tfm; +}; + +int ablk_set_key (struct crypto_ablkcipher *tfm,const u8 *key, +unsigned int key_len); + +int __ablk_encrypt (struct ablkcipher_request *req); + +int ablk_encrypt (struct ablkcipher_request *req); + +int ablk_decrypt (struct ablkcipher_request *req); + +void ablk_exit (struct crypto_tfm *tfm); + +int ablk_init_common (struct crypto_tfm *tfm,const char *drv_name); + +int ablk_init (struct crypto_tfm *tfm); + +#endif /* _CRYPTO_ABLK_HELPER_H */
Re: [PATCH 1/4] crypto: create generic version of ablk_helper
Quoted 8 lines · Jussi Kivilinna
Quoted 5 lines · Ard Biesheuvel
Create a generic version of
ablk_helperso it can be reused by other architectures.
Quoted 6 lines
Quoted 2 lines
Acked-by Jussi Kivilinna <[email protected]>Signed-off-by Ard Biesheuvel <[email protected]>Why resent this patch here when this was in the earlier patchset?
http://marc.info/?l=linux-crypto-vger&m=137966378813818&w=2
Apologies for the spam.
This second submission was a cross post from linux-arm-kernel because the later patches in the set depend on this one.
Regards,
Ard.
Quoted 251 lines
Quoted 250 lines
---
crypto/Kconfig | 4 ++ crypto/Makefile | 1 + crypto/ablk_helper.c | 150 +++++++++++++++++++++++++++++++++++++++++++ include/asm-generic/simd.h | 14 ++++ include/crypto/ablk_helper.h | 31 +++++++++ 5 files changed, 200 insertions(+)create mode 100644
crypto/ablk_helper.c
create mode 100644include/asm-generic/simd.h
create mode 100644include/crypto/ablk_helper.hcrypto/Kconfig +4
-0@@ -179,6 +179,10 @@ config CRYPTO_ABLK_HELPER_X86 depends on X86 select CRYPTO_CRYPTD +config CRYPTO_ABLK_HELPER + tristate + select CRYPTO_CRYPTD +config
CRYPTO_GLUE_HELPER_X86
tristate
depends on X86crypto/Makefile +0
-0@@ -104,3 +104,4 @@ obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.oobj-$(CONFIG_XOR_BLOCKS) += xor.o obj-$(CONFIG_ASYNC_CORE) += async_tx/ obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys/
crypto/Makefile +1
-0+obj-$(CONFIG_CRYPTO_ABLK_HELPER) += ablk_helper.ocrypto/ablk_helper.c new file +150
-0@@ -0,0 +1,150 @@ +/* +* Shared async block cipher helpers +* +* Copyright (c) 2012 Jussi Kivilinna <[email protected]> +* +* Based on aesni-intel_glue.c by: +* Copyright (C) 2008, Intel Corp. +* Author: Huang Ying <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 +* USA +* +*/ + +#include <linux/kernel.h> +#include <linux/crypto.h> +#include <linux/init.h> +#include <linux/module.h> +#include <linux/hardirq.h> +#include <crypto/algapi.h> +#include <crypto/cryptd.h> +#include <crypto/ablk_helper.h> +#include <asm/simd.h> + +int ablk_set_key (struct crypto_ablkcipher *tfm,const u8 *key, +unsigned int key_len) +{ +struct async_helper_ctx *ctx =crypto_ablkcipher_ctx (tfm); +struct crypto_ablkcipher *child = &ctx->cryptd_tfm->base; +int err; + +crypto_ablkcipher_clear_flags (child,CRYPTO_TFM_REQ_MASK ); +crypto_ablkcipher_set_flags (child,crypto_ablkcipher_get_flags (tfm) + &CRYPTO_TFM_REQ_MASK ); + err =crypto_ablkcipher_setkey (child, key, key_len); +crypto_ablkcipher_set_flags (tfm,crypto_ablkcipher_get_flags (child) + &CRYPTO_TFM_RES_MASK ); +return err; +} +EXPORT_SYMBOL_GPL (ablk_set_key); + +int __ablk_encrypt (struct ablkcipher_request *req) +{ +struct crypto_ablkcipher *tfm =crypto_ablkcipher_reqtfm (req); +struct async_helper_ctx *ctx =crypto_ablkcipher_ctx (tfm); +struct blkcipher_desc desc; + + desc.tfm =cryptd_ablkcipher_child (ctx->cryptd_tfm); + desc.info = req->info; + desc.flags =0 ; + +return crypto_blkcipher_crt (desc.tfm)->encrypt ( + &desc, req->dst, req->src, req->nbytes); +} +EXPORT_SYMBOL_GPL (__ablk_encrypt); + +int ablk_encrypt (struct ablkcipher_request *req) +{ +struct crypto_ablkcipher *tfm =crypto_ablkcipher_reqtfm (req); +struct async_helper_ctx *ctx =crypto_ablkcipher_ctx (tfm); + +if (!may_use_simd ()) { +struct ablkcipher_request *cryptd_req = +ablkcipher_request_ctx (req); + +memcpy (cryptd_req, req,sizeof (*req)); +ablkcipher_request_set_tfm (cryptd_req, &ctx->cryptd_tfm->base); + +return crypto_ablkcipher_encrypt (cryptd_req); + }else { +return __ablk_encrypt (req); + } +} +EXPORT_SYMBOL_GPL (ablk_encrypt); + +int ablk_decrypt (struct ablkcipher_request *req) +{ +struct crypto_ablkcipher *tfm =crypto_ablkcipher_reqtfm (req); +struct async_helper_ctx *ctx =crypto_ablkcipher_ctx (tfm); + +if (!may_use_simd ()) { +struct ablkcipher_request *cryptd_req = +ablkcipher_request_ctx (req); + +memcpy (cryptd_req, req,sizeof (*req)); +ablkcipher_request_set_tfm (cryptd_req, &ctx->cryptd_tfm->base); + +return crypto_ablkcipher_decrypt (cryptd_req); + }else { +struct blkcipher_desc desc; + + desc.tfm =cryptd_ablkcipher_child (ctx->cryptd_tfm); + desc.info = req->info; + desc.flags =0 ; + +return crypto_blkcipher_crt (desc.tfm)->decrypt ( + &desc, req->dst, req->src, req->nbytes); + } +} +EXPORT_SYMBOL_GPL (ablk_decrypt); + +void ablk_exit (struct crypto_tfm *tfm) +{ +struct async_helper_ctx *ctx =crypto_tfm_ctx (tfm); + +cryptd_free_ablkcipher (ctx->cryptd_tfm); +} +EXPORT_SYMBOL_GPL (ablk_exit); + +int ablk_init_common (struct crypto_tfm *tfm,const char *drv_name) +{ +struct async_helper_ctx *ctx =crypto_tfm_ctx (tfm); +struct cryptd_ablkcipher *cryptd_tfm; + + cryptd_tfm =cryptd_alloc_ablkcipher (drv_name,0 ,0 ); +if (IS_ERR (cryptd_tfm)) +return PTR_ERR (cryptd_tfm); + + ctx->cryptd_tfm = cryptd_tfm; + tfm->crt_ablkcipher.reqsize =sizeof (struct ablkcipher_request ) + +crypto_ablkcipher_reqsize (&cryptd_tfm->base); + +return 0 ; +} +EXPORT_SYMBOL_GPL (ablk_init_common); + +int ablk_init (struct crypto_tfm *tfm) +{ +char drv_name[CRYPTO_MAX_ALG_NAME ]; + +snprintf (drv_name,sizeof (drv_name),"__driver-%s" , +crypto_tfm_alg_driver_name (tfm)); + +return ablk_init_common (tfm, drv_name); +} +EXPORT_SYMBOL_GPL (ablk_init); + +MODULE_LICENSE ("GPL" );include/asm-generic/simd.h new file +14
-0@@ -0,0 +1,14 @@ + +#include <linux/hardirq.h> + +/* +* may_use_simd - whether it is allowable at this time to issue SIMD +* instructions or access the SIMD register file +* +* As architectures typically don't preserve the SIMD register file when +* taking an interrupt, !in_interrupt() should be a reasonable default. +*/ +static __must_check inline bool may_use_simd (void ) +{ +return !in_interrupt (); +}include/crypto/ablk_helper.h new file +31
-0@@ -0,0 +1,31 @@ +/* +* Shared async block cipher helpers +*/ + +#ifndef _CRYPTO_ABLK_HELPER_H +#define _CRYPTO_ABLK_HELPER_H + +#include <linux/crypto.h> +#include <linux/kernel.h> +#include <crypto/cryptd.h> + +struct async_helper_ctx { +struct cryptd_ablkcipher *cryptd_tfm; +}; + +int ablk_set_key (struct crypto_ablkcipher *tfm,const u8 *key, +unsigned int key_len); + +int __ablk_encrypt (struct ablkcipher_request *req); + +int ablk_encrypt (struct ablkcipher_request *req); + +int ablk_decrypt (struct ablkcipher_request *req); + +void ablk_exit (struct crypto_tfm *tfm); + +int ablk_init_common (struct crypto_tfm *tfm,const char *drv_name); + +int ablk_init (struct crypto_tfm *tfm); + +#endif /* _CRYPTO_ABLK_HELPER_H */
PATCH 2/4 ARM: pull in <asm/simd.h> from asm-generic
In reply to Ard Biesheuvel--- arch/arm/include/asm/Kbuild | 1 + 1 file changed, 1 insertion(+)
arch/arm/include/asm/Kbuild +1 -0
@@ -24,6 +24,7 @@ generic-y += sembuf.h generic-y += serial.h generic-y += shmbuf.h generic-y += siginfo.h +generic-y += simd.h generic-y += sizes.h generic-y += socket.h generic-y += sockios.h
Signature · 1 line
1.8.1.2
PATCH 3/4 ARM: move AES typedefs and function prototypes to separate header
In reply to Ard BiesheuvelPut the struct definitions for AES keys and the asm function prototypes in a separate header and export the asm functions from the module. This allows other drivers to use them directly.
--- arch/arm/crypto/aes_glue.c | 22 ++++++---------------- arch/arm/crypto/aes_glue.h | 19 +++++++++++++++++++ 2 files changed, 25 insertions(+), 16 deletions(-) create mode 100644 arch/arm/crypto/aes_glue.h
arch/arm/crypto/aes_glue.c +6 -16
@@ -6,22 +6,12 @@#include <linux/crypto.h> #include <crypto/aes.h> -#define AES_MAXNR 14 +#include "aes_glue.h" -typedef struct { -unsigned int rd_key[4 *(AES_MAXNR +1 )]; -int rounds; -}AES_KEY ; - -struct AES_CTX { -AES_KEY enc_key; -AES_KEY dec_key; -}; - -asmlinkage void AES_encrypt(const u8 *in, u8 *out, AES_KEY *ctx); -asmlinkage void AES_decrypt(const u8 *in, u8 *out, AES_KEY *ctx); -asmlinkage int private_AES_set_decrypt_key(const unsigned char *userKey, const int bits, AES_KEY *key); -asmlinkage int private_AES_set_encrypt_key(const unsigned char *userKey, const int bits, AES_KEY *key); +EXPORT_SYMBOL (AES_encrypt); +EXPORT_SYMBOL (AES_decrypt); +EXPORT_SYMBOL (private_AES_set_encrypt_key); +EXPORT_SYMBOL (private_AES_set_decrypt_key);static void aes_encrypt (struct crypto_tfm *tfm,u8 *dst,const u8 *src) {
@@ -81,7 +71,7 @@ static struct crypto_alg aes_alg = { .cipher = { .cia_min_keysize = AES_MIN_KEY_SIZE, .cia_max_keysize = AES_MAX_KEY_SIZE, - .cia_setkey = aes_set_key, + .cia_setkey = aes_set_key, .cia_encrypt = aes_encrypt, .cia_decrypt = aes_decrypt }
arch/arm/crypto/aes_glue.h new file +19 -0
@@ -0,0 +1,19 @@ + +#define AES_MAXNR 14 + +struct AES_KEY { +unsigned int rd_key[4 * (AES_MAXNR +1 )]; +int rounds; +}; + +struct AES_CTX { +struct AES_KEY enc_key; +struct AES_KEY dec_key; +}; + +asmlinkage void AES_encrypt(const u8 *in, u8 *out, struct AES_KEY *ctx); +asmlinkage void AES_decrypt(const u8 *in, u8 *out, struct AES_KEY *ctx); +asmlinkage int private_AES_set_decrypt_key(const unsigned char *userKey, +const int bits,struct AES_KEY *key); +asmlinkage int private_AES_set_encrypt_key(const unsigned char *userKey, +const int bits,struct AES_KEY *key);
Signature · 1 line
1.8.1.2
PATCH 4/4 ARM: add support for bit sliced AES using NEON instructions
In reply to Ard BiesheuvelThis implementation of the AES algorithm gives around 45% speedup on Cortex-A15 for CTR mode and for XTS in encryption mode. Both CBC and XTS in decryption mode are slightly faster (5 - 10% on Cortex-A15). [As CBC in encryption mode can only be performed sequentially, there is no speedup in this case.]
Unlike the core AES cipher (on which this module also depends), this algorithm uses bit slicing to process up to 8 blocks in parallel in constant time. This algorithm does not rely on any lookup tables so it is believed to be invulnerable to cache timing attacks.
The core code has been adopted from the OpenSSL project (in collaboration with the original author, on cc). For ease of maintenance, this version is identical to the upstream OpenSSL code, i.e., all modifications that were required to make it suitable for inclusion into the kernel have already been merged upstream.
--- arch/arm/crypto/Makefile | 6 +- arch/arm/crypto/aesbs-core.S | 2603 ++++++++++++++++++++++++++++++++++++++++++ arch/arm/crypto/aesbs-glue.c | 449 ++++++++ crypto/Kconfig | 16 + 4 files changed, 3072 insertions(+), 2 deletions(-) create mode 100644 arch/arm/crypto/aesbs-core.S create mode 100644 arch/arm/crypto/aesbs-glue.c
arch/arm/crypto/Makefile +4 -2
@@ -3,7 +3,9 @@ # obj-$(CONFIG_CRYPTO_AES_ARM) += aes-arm.o +obj-$(CONFIG_CRYPTO_AES_ARM_BS) += aes-arm-bs.o obj-$(CONFIG_CRYPTO_SHA1_ARM) += sha1-arm.o -aes-arm-y := aes-armv4.o aes_glue.o -sha1-arm-y := sha1-armv4-large.o sha1_glue.o +aes-arm-y := aes-armv4.o aes_glue.o +aes-arm-bs-y := aesbs-core.o aesbs-glue.o +sha1-arm-y := sha1-armv4-large.o sha1_glue.o
arch/arm/crypto/aesbs-core.S new file +2603 -0
2,603 lines not on this page · raw message
arch/arm/crypto/aesbs-glue.c new file +449 -0
386 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
1.8.1.2
Part of the diff is not shown here. View the raw message.
PATCH 4/4 ARM: add support for bit sliced AES using NEON instructions
Quoted 19 lines · Ard Biesheuvel
This implementation of the AES algorithm gives around 45% speedup on Cortex-A15 for CTR mode and for XTS in encryption mode. Both CBC and XTS in decryption mode are slightly faster (5 - 10% on Cortex-A15). [As CBC in encryption mode can only be performed sequentially, there is no speedup in this case.]
Unlike the core AES cipher (on which this module also depends), this algorithm uses bit slicing to process up to 8 blocks in parallel in constant time. This algorithm does not rely on any lookup tables so it is believed to be invulnerable to cache timing attacks.
The core code has been adopted from the OpenSSL project (in collaboration with the original author, on cc). For ease of maintenance, this version is identical to the upstream OpenSSL code, i.e., all modifications that were required to make it suitable for inclusion into the kernel have already been merged upstream.
Cc Andy Polyakov <[email protected]>Signed-off-by Ard Biesheuvel <[email protected]>---
[..snip..]
arch/arm/crypto/aesbs-core.S line 363
Quoted 2 lines
arch/arm/crypto/aesbs-core.S new file +2
-0+ bcc .Ldec_done + @ multiplication by 0x0e
Decryption can probably be made faster by implementing InvMixColumns slightly differently. Instead of implementing inverse MixColumns matrix directly, use preprocessing step, followed by MixColumns as described in section "4.1.3 Decryption" of "The Design of Rijndael: AES - The Advanced Encryption Standard" (J. Daemen, V. Rijmen / 2002).
In short, the MixColumns and InvMixColumns matrixes have following relation:
| 0e 0b 0d 09 | | 02 03 01 01 | | 05 00 04 00 | | 09 0e 0b 0d | = | 01 02 03 01 | x | 00 05 00 04 | | 0d 09 0e 0b | | 01 01 02 03 | | 04 00 05 00 | | 0b 0d 09 0e | | 03 01 01 02 | | 00 04 00 05 |
Bit-sliced implementation of the 05-00-04-00 matrix much shorter than 0e-0b-0d-09 matrix, so even when combined with MixColumns total instruction count for InvMixColumns implemented this way should be nearly half of current.
Check [1] for implementation of this on AVX instruction set.
-Jussi
PATCH 4/4 ARM: add support for bit sliced AES using NEON instructions
On 22 September 2013 13:12, Jussi Kivilinna <[email protected]> wrote:
[...]
Quoted 7 lines
Decryption can probably be made faster by implementing InvMixColumns slightly differently. Instead of implementing inverse MixColumns matrix directly, use preprocessing step, followed by MixColumns as described in section "4.1.3 Decryption" of "The Design of Rijndael: AES - The Advanced Encryption Standard" (J. Daemen, V. Rijmen / 2002).
In short, the MixColumns and InvMixColumns matrixes have following relation:
Quoted 9 lines
| 0e 0b 0d 09 | | 02 03 01 01 | | 05 00 04 00 | | 09 0e 0b 0d | = | 01 02 03 01 | x | 00 05 00 04 | | 0d 09 0e 0b | | 01 01 02 03 | | 04 00 05 00 | | 0b 0d 09 0e | | 03 01 01 02 | | 00 04 00 05 |Bit-sliced implementation of the 05-00-04-00 matrix much shorter than 0e-0b-0d-09 matrix, so even when combined with MixColumns total instruction count for InvMixColumns implemented this way should be nearly half of current.
That is a very useful tip, thank you. I will have a go at it and follow up later.
Regards,
Ard.
cover ARM: NEON based fast(er) AES in CBC/CTR/XTS modes
In reply to Ard BiesheuvelQuoted 9 lines · Ard Biesheuvel
Note to reviewers:
Reviewing the fileaesbs-core.Smay be a bit overwhelming, so if there are any questions or concerns, please refer to the link below. This is the original Perl script that gets called by OpenSSL's build system during their build to generate the .S file on the fly. [In the case of OpenSSL, this is used in some cases to target different assemblers or ABIs]. This arrangement is not suitable (or required) for the kernel, so I have taken the generated .S file instead.http://git.openssl.org/gitweb/?p=openssl.git;f=crypto/aes/asm/bsaes-armv7.pl;a=blob
You should probably capture this reference in the code, or at least in the commit log for the corresponding patch.
Quoted 4 lines
Note to integrators:
While this implementation is significantly faster, especially in CTR mode, it is unclear whether the net impact on power efficiency is favorable or not, so please refrain from making any assumptions to that effect.
That definitively should be mentioned in the commit log for patch #4.
Nicolas