crypto: aria: add ARIA AES-NI/AVX/x86_64/GFNI implementation
Subsystems (2)
v4 cover crypto: aria: add ARIA AES-NI/AVX/x86_64/GFNI implementation
The purpose of this patchset is to support the implementation of ARIA-AVX. Many of the ideas in this implementation are from Camellia-avx, especially byte slicing.
Like Camellia, ARIA also uses a 16way strategy.
ARIA cipher algorithm is similar to AES.
There are four s-boxes in the ARIA spec and the first and second s-boxes are the same as AES's s-boxes.
Almost functions are based on aria-generic code except for s-box related function.
The aria-avx doesn't implement the key expanding function. it supports only encrypt() and decrypt().
Encryption and Decryption are actually the same but it should use separated keys(encryption key and decryption key). En/Decryption steps are like below:
1. Add-Round-Key
2. S-box.
3. Diffusion Layer.
There is no special thing in the Add-Round-Key step.
There are some notable things in s-box step.
Like Camellia, it doesn't use a lookup table, instead, it uses AES-NI. There are 2 implementations for that.
One is to use AES-NI and affine transformation, which is the same as Camellia, sm4, and others.
Another is to use GFNI.
GFNI implementation is faster than AES-NI implementation. So, it uses GFNI implementation if the running CPU supports GFNI.
To calculate the first s-box(S1), it just uses the aesenclast and then inverts shift_row. No more process is needed for this job because the first s-box is the same as the AES encryption s-box.
To calculate the second s-box(X1, invert of S1), it just uses the aesdeclast and then inverts shift_row. No more process is needed for this job because the second s-box is the same as the AES decryption s-box.
To calculate the third s-box(S2), it uses the aesenclast, then affine transformation, which is combined AES inverse affine and ARIA S2.
To calculate the last s-box(X2, invert of S2), it uses the aesdeclast, then affine transformation, which is combined X2 and AES forward affine.
The optimized third and last s-box logic and GFNI s-box logic are implemented by Jussi Kivilinna.
The aria-generic implementation is based on a 32-bit implementation, not an 8-bit implementation.
The aria-avx Diffusion Layer implementation is based on aria-generic implementation because 8-bit implementation is not fit for parallel implementation but 32-bit is fit for this.
The first patch in this series is to export functions for aria-avx. The aria-avx uses existing functions in the aria-generic code. The second patch is to implement aria-avx.
The last patch is to add async test for aria.
Benchmarks:
The tcrypt is used.
cpu: i3-12100
How to test:
modprobe aria-generic
tcrypt mode=610 num_mb=8192
Result:
testing speed of multibuffer ecb(aria) (ecb(aria-generic)) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 534 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2006 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 3674 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 52374 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 608 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2586 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 4707 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 69794 cycles
testing speed of multibuffer ecb(aria) (ecb(aria-generic)) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 545 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 1995 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 3673 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 52359 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 615 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2588 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 4712 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 69916 cycles
How to test:
modprobe aria
tcrypt mode=610 num_mb=8192
AVX with AES-NI:
testing speed of multibuffer ecb(aria) (ecb-aria-avx) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 629 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2060 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 1223 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 11931 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 686 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2616 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 1439 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 15488 cycles
testing speed of multibuffer ecb(aria) (ecb-aria-avx) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 609 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2027 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 1211 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 12040 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 684 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2614 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 1445 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 15478 cycles
AVX with GFNI:
testing speed of multibuffer ecb(aria) (ecb-aria-avx) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 730 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2056 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 1028 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 9223 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 685 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2603 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 1179 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 11728 cycles
testing speed of multibuffer ecb(aria) (ecb-aria-avx) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 617 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2057 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 1020 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 9280 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 687 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2599 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 1176 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 11909 cycles
v4:
- Fix sparse warning.
- Remove .align statement for .text
- https://lkml.kernel.org/r/[email protected]
v3:
- Use ECB macro instead of opencode.
- Implement ctr(aria-avx).
- Improve performance(20% ~ 30%) with combined affine transformation for S2 and X2.
- Implemented by Jussi Kivilinna.
- Improve performance( ~ 55%) with GFNI.
- Implemented by Jussi Kivilinna.
- Add aria-ctr async speed test.
- Add aria-gcm multi buffer speed test
- Rebase and fix Kconfig
v2:
- Do not call non-FPU functions(aria_{encrypt | decrypt}()) in the FPU context.
- Do not acquire FPU context for too long.
Taehee Yoo (3):
crypto: aria: prepare generic module for optimized implementations
crypto: aria-avx: add AES-NI/AVX/x86_64/GFNI assembler implementation
of aria cipher
crypto: tcrypt: add async speed test for aria cipher
arch/x86/crypto/Kconfig | 18 +
arch/x86/crypto/Makefile | 3 +
arch/x86/crypto/aria-aesni-avx-asm_64.S | 1303 +++++++++++++++++++++++
arch/x86/crypto/aria-avx.h | 16 +
arch/x86/crypto/aria_aesni_avx_glue.c | 213 ++++
crypto/Makefile | 2 +-
crypto/{aria.c => aria_generic.c} | 39 +-
crypto/tcrypt.c | 30 +
include/crypto/aria.h | 17 +-
9 files changed, 1623 insertions(+), 18 deletions(-)
create mode 100644 arch/x86/crypto/aria-aesni-avx-asm_64.S
create mode 100644 arch/x86/crypto/aria-avx.h
create mode 100644 arch/x86/crypto/aria_aesni_avx_glue.c
rename crypto/{aria.c => aria_generic.c} (86%)Signature · 1 line
2.17.1
PATCH v4 1/3 crypto: aria: prepare generic module for optimized implementations
It renames aria to aria_generic and exports some functions such as aria_set_key(), aria_encrypt(), and aria_decrypt() to be able to be used by aria-avx implementation.
---
v4:
- No changes.
v3:
- No changes.
v2:
- No changes.
crypto/Makefile | 2 +-
crypto/{aria.c => aria_generic.c} | 39 +++++++++++++++++++++++++------
include/crypto/aria.h | 17 ++++++--------
3 files changed, 40 insertions(+), 18 deletions(-)
rename crypto/{aria.c => aria_generic.c} (86%)crypto/Makefile +1 -1
@@ -149,7 +149,7 @@ obj-$(CONFIG_CRYPTO_TEA) += tea.o obj-$(CONFIG_CRYPTO_KHAZAD) += khazad.o obj-$(CONFIG_CRYPTO_ANUBIS) += anubis.o obj-$(CONFIG_CRYPTO_SEED) += seed.o -obj-$(CONFIG_CRYPTO_ARIA) += aria.o +obj-$(CONFIG_CRYPTO_ARIA) += aria_generic.o obj-$(CONFIG_CRYPTO_CHACHA20) += chacha_generic.o obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_generic.o obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o
crypto/aria_generic.c renamed from crypto/aria.c +32 -7
@@ -16,6 +16,14 @@#include <crypto/aria.h> +static const u32 key_rc[20 ] = { +0x517cc1b7 ,0x27220a94 ,0xfe13abe8 ,0xfa9a6ee0 , +0x6db14acc ,0x9e21c820 ,0xff28b1d5 ,0xef5de2b0 , +0xdb92371d ,0x2126e970 ,0x03249775 ,0x04e8c90e , +0x517cc1b7 ,0x27220a94 ,0xfe13abe8 ,0xfa9a6ee0 , +0x6db14acc ,0x9e21c820 ,0xff28b1d5 ,0xef5de2b0 +}; +static void aria_set_encrypt_key (struct aria_ctx *ctx,const u8 *in_key,unsigned int key_len) {
@@ -25,7 +33,7 @@ static void aria_set_encrypt_key(struct aria_ctx *ctx, const u8 *in_key,const u32 *ck;int rkidx =0 ; - ck = &key_rc[(key_len -16 ) /8 ][0 ]; + ck = &key_rc[(key_len -16 ) /2 ]; w0[0 ] =be32_to_cpu (key[0 ]); w0[1 ] =be32_to_cpu (key[1 ]);
@@ -163,8 +171,7 @@ static void aria_set_decrypt_key(struct aria_ctx *ctx) } } -static int aria_set_key (struct crypto_tfm *tfm,const u8 *in_key, -unsigned int key_len) +int aria_set_key (struct crypto_tfm *tfm,const u8 *in_key,unsigned int key_len) {struct aria_ctx *ctx =crypto_tfm_ctx (tfm);
@@ -179,6 +186,7 @@ static int aria_set_key(struct crypto_tfm *tfm, const u8 *in_key,return 0 ; } +EXPORT_SYMBOL_GPL (aria_set_key);static void __aria_crypt (struct aria_ctx *ctx,u8 *out,const u8 *in,u32 key[][ARIA_RD_KEY_WORDS ])
@@ -235,14 +243,30 @@ static void __aria_crypt(struct aria_ctx *ctx, u8 *out, const u8 *in, dst[3 ] =cpu_to_be32 (reg3); } -static void aria_encrypt (struct crypto_tfm *tfm,u8 *out,const u8 *in) +void aria_encrypt (void *_ctx,u8 *out,const u8 *in) +{ +struct aria_ctx *ctx = (struct aria_ctx *)_ctx; + +__aria_crypt (ctx, out, in, ctx->enc_key); +} +EXPORT_SYMBOL_GPL (aria_encrypt); + +void aria_decrypt (void *_ctx,u8 *out,const u8 *in) +{ +struct aria_ctx *ctx = (struct aria_ctx *)_ctx; + +__aria_crypt (ctx, out, in, ctx->dec_key); +} +EXPORT_SYMBOL_GPL (aria_decrypt); + +static void __aria_encrypt (struct crypto_tfm *tfm,u8 *out,const u8 *in) {struct aria_ctx *ctx =crypto_tfm_ctx (tfm);__aria_crypt (ctx, out, in, ctx->enc_key); } -static void aria_decrypt (struct crypto_tfm *tfm,u8 *out,const u8 *in) +static void __aria_decrypt (struct crypto_tfm *tfm,u8 *out,const u8 *in) {struct aria_ctx *ctx =crypto_tfm_ctx (tfm);
@@ -263,8 +287,8 @@ static struct crypto_alg aria_alg = { .cia_min_keysize = ARIA_MIN_KEY_SIZE, .cia_max_keysize = ARIA_MAX_KEY_SIZE, .cia_setkey = aria_set_key, - .cia_encrypt = aria_encrypt, - .cia_decrypt = aria_decrypt + .cia_encrypt = __aria_encrypt, + .cia_decrypt = __aria_decrypt } } };
@@ -286,3 +310,4 @@ MODULE_DESCRIPTION("ARIA Cipher Algorithm");MODULE_LICENSE ("GPL" );MODULE_AUTHOR ("Taehee Yoo <[email protected]>" );MODULE_ALIAS_CRYPTO ("aria" ); +MODULE_ALIAS_CRYPTO ("aria-generic" );
include/crypto/aria.h +7 -10
@@ -32,18 +32,10 @@#define ARIA_RD_KEY_WORDS (ARIA_BLOCK_SIZE / sizeof(u32))struct aria_ctx { -int key_length; -int rounds;u32 enc_key[ARIA_MAX_RD_KEYS ][ARIA_RD_KEY_WORDS ];u32 dec_key[ARIA_MAX_RD_KEYS ][ARIA_RD_KEY_WORDS ]; -}; - -static const u32 key_rc[5 ][4 ] = { - {0x517cc1b7 ,0x27220a94 ,0xfe13abe8 ,0xfa9a6ee0 }, - {0x6db14acc ,0x9e21c820 ,0xff28b1d5 ,0xef5de2b0 }, - {0xdb92371d ,0x2126e970 ,0x03249775 ,0x04e8c90e }, - {0x517cc1b7 ,0x27220a94 ,0xfe13abe8 ,0xfa9a6ee0 }, - {0x6db14acc ,0x9e21c820 ,0xff28b1d5 ,0xef5de2b0 } +int rounds; +int key_length; };static const u32 s1[256 ] = {
@@ -458,4 +450,9 @@ static inline void aria_gsrk(u32 *rk, u32 *x, u32 *y, u32 n) ((y[(q + 2) % 4]) << (32 - r)); } +void aria_encrypt (void *ctx,u8 *out,const u8 *in); +void aria_decrypt (void *ctx,u8 *out,const u8 *in); +int aria_set_key (struct crypto_tfm *tfm,const u8 *in_key, +unsigned int key_len); +#endif
Signature · 1 line
2.17.1
PATCH v4 2/3 crypto: aria-avx: add AES-NI/AVX/x86_64/GFNI assembler implementation of aria cipher
In reply to Taehee YooThe implementation is based on the 32-bit implementation of the aria. Also, aria-avx process steps are the similar to the camellia-avx.
1. Byteslice(16way)
2. Add-round-key.
3. Sbox
4. Diffusion layer.
Except for s-box, all steps are the same as the aria-generic implementation. s-box step is very similar to camellia and sm4 implementation.
There are 2 implementations for s-box step.
One is to use AES-NI and affine transformation, which is the same as Camellia, sm4, and others.
Another is to use GFNI.
GFNI implementation is faster than AES-NI implementation. So, it uses GFNI implementation if the running CPU supports GFNI.
There are 4 s-boxes in the ARIA and the 2 s-boxes are the same as AES's s-boxes.
To calculate the first sbox, it just uses the aesenclast and then inverts shift_row.
No more process is needed for this job because the first s-box is the same as the AES encryption s-box.
To calculate the second sbox(invert of s1), it just uses the aesdeclast and then inverts shift_row.
No more process is needed for this job because the second s-box is the same as the AES decryption s-box.
To calculate the third s-box, it uses the aesenclast, then affine transformation, which is combined AES inverse affine and ARIA S2.
To calculate the last s-box, it uses the aesdeclast,
then affine transformation, which is combined X2 and AES forward affine.
The optimized third and last s-box logic and GFNI s-box logic are implemented by Jussi Kivilinna.
The aria-generic implementation is based on a 32-bit implementation, not an 8-bit implementation. the aria-avx Diffusion Layer implementation is based on aria-generic implementation because 8-bit implementation is not fit for parallel implementation but 32-bit is enough to fit for this.
---
v4:
- Fix sparse warning.
- Remove .align statement for .text
- https://lkml.kernel.org/r/[email protected]
v3:
- Use ECB macro instead of opencode.
- Implement ctr(aria-avx).
- Improve performance(20% ~ 30%) with combined affine transformation for S2 and X2.
- Implemented by Jussi Kivilinna.
- Improve performance( ~ 55%) with GFNI.
- Implemented by Jussi Kivilinna.
- Rebase and fix Kconfig
v2:
- Do not call non-FPU functions(aria_{encrypt | decrypt}()) in the FPU context.
- Do not acquire FPU context for too long.
arch/x86/crypto/Kconfig | 18 + arch/x86/crypto/Makefile | 3 + arch/x86/crypto/aria-aesni-avx-asm_64.S | 1303 +++++++++++++++++++++++ arch/x86/crypto/aria-avx.h | 16 + arch/x86/crypto/aria_aesni_avx_glue.c | 213 ++++ 5 files changed, 1553 insertions(+) create mode 100644 arch/x86/crypto/aria-aesni-avx-asm_64.S create mode 100644 arch/x86/crypto/aria-avx.h create mode 100644 arch/x86/crypto/aria_aesni_avx_glue.c
arch/x86/crypto/Kconfig +18 -0
@@ -286,6 +286,24 @@ config CRYPTO_TWOFISH_AVX_X86_64 Processes eight blocks in parallel. +config CRYPTO_ARIA_AESNI_AVX_X86_64 + tristate "Ciphers: ARIA with modes: ECB, CTR (AES-NI/AVX/GFNI)" + depends on X86 && 64BIT + select CRYPTO_SKCIPHER + select CRYPTO_SIMD + select CRYPTO_ALGAPI + select CRYPTO_ARIA + help + Length-preserving cipher: ARIA cipher algorithms + (RFC 5794) with ECB and CTR modes + + Architecture: x86_64 using: + - AES-NI (AES New Instructions) + - AVX (Advanced Vector Extensions) + - GFNI (Galois Field New Instructions) + + Processes 16 blocks in parallel. + config CRYPTO_CHACHA20_X86_64 tristate "Ciphers: ChaCha20, XChaCha20, XChaCha12 (SSSE3/AVX2/AVX-512VL)" depends on X86 && 64BIT
arch/x86/crypto/Makefile +3 -0
@@ -100,6 +100,9 @@ sm4-aesni-avx-x86_64-y := sm4-aesni-avx-asm_64.o sm4_aesni_avx_glue.o obj-$(CONFIG_CRYPTO_SM4_AESNI_AVX2_X86_64) += sm4-aesni-avx2-x86_64.o sm4-aesni-avx2-x86_64-y := sm4-aesni-avx2-asm_64.o sm4_aesni_avx2_glue.o +obj-$(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) += aria-aesni-avx-x86_64.o +aria-aesni-avx-x86_64-y := aria-aesni-avx-asm_64.o aria_aesni_avx_glue.o + quiet_cmd_perlasm = PERLASM $@ cmd_perlasm = $(PERL) $< > $@ $(obj)/%.S: $(src)/%.pl FORCE
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +1303 -0
1,303 lines not on this page · raw message
arch/x86/crypto/aria-avx.h new file +16 -0
@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef ASM_X86_ARIA_AVX_H +#define ASM_X86_ARIA_AVX_H + +#include <linux/types.h> + +#define ARIA_AESNI_PARALLEL_BLOCKS 16 +#define ARIA_AESNI_PARALLEL_BLOCK_SIZE (ARIA_BLOCK_SIZE * 16) + +struct aria_avx_ops { +void (*aria_encrypt_16way)(const void *ctx,u8 *dst,const u8 *src); +void (*aria_decrypt_16way)(const void *ctx,u8 *dst,const u8 *src); +void (*aria_ctr_crypt_16way)(const void *ctx,u8 *dst,const u8 *src, +u8 *keystream,u8 *iv); +}; +#endif
arch/x86/crypto/aria_aesni_avx_glue.c new file +213 -0
213 lines not on this page · raw message
Signature · 1 line
2.17.1
Herbert Xu
maintainerRe: [PATCH v4 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64/GFNI assembler implementation of aria cipher
arch/x86/crypto/aria-aesni-avx-asm_64.S line 275
Quoted 6 lines · Taehee Yoo
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +5
-0+#define aria_ark_8way(x0, x1, x2, x3, \ + x4, x5, x6, x7, \ + t0, rk, idx, round) \ + /* AddRoundKey */ \ + vpbroadcastb ((round * 16) + idx + 3)(rk), t0; \
This triggers an invalid OP code on CPUs that supports AVX but not AVX2:
[ 10.857327] invalid opcode: 0000 [#1] PREEMPT SMP [ 10.857935] CPU: 1 PID: 2816 Comm: cryptomgr_test Tainted: G W 6.2.0-rc1+ #1 [ 10.858783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014 [ 10.859622] RIP: 0010:__aria_aesni_avx_crypt_16way+0x22d/0x6000 [aria_aesni_avx_x86_64] [ 10.860440] Code: c4 41 7a 7f 48 10 c4 41 7a 7f 50 20 c4 41 7a 7f 58 30 c4 41 7a 7f 60 40 c4 41 7a 7f 68 50 c4 41 7a 7f 70 60 c4 41 7a 7f 78 70 <c4> c2 79 78 41 0b c5 39 ef c0 c4 c2 79 78 41 0a c5 31 ef c8 c4 c2 [ 10.862409] RSP: 0018:ffffc900002eb728 EFLAGS: 00010206 [ 10.863011] RAX: ffff88800666b00f RBX: ffff88800666b00f RCX: ffffffffa0c02010 [ 10.863754] RDX: ffff88800666b00f RSI: ffff88800666b00f RDI: ffff88800408a420 [ 10.864500] RBP: ffffc900002eb728 R08: ffff88800666b08f R09: ffff88800408a420 [ 10.865270] R10: ffff8880064de040 R11: 00000000fffffff0 R12: ffff88800408a420 [ 10.866019] R13: 000000000000011b R14: ffff88800666b00f R15: ffff88800666b00f [ 10.866764] FS: 0000000000000000(0000) GS:ffff88801ed00000(0000) knlGS:0000000000000000 [ 10.867587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 10.868230] CR2: 00007f8c80f160a4 CR3: 0000000001e0a002 CR4: 0000000000060ee0 [ 10.868986] Call Trace: [ 10.869392] <TASK> [ 10.869744] aria_aesni_avx_encrypt_16way+0x77/0xf0 [aria_aesni_avx_x86_64] [ 10.870489] ecb_do_encrypt.constprop.0+0x100/0x140 [aria_aesni_avx_x86_64] [ 10.871233] aria_avx_ecb_encrypt+0xd/0x20 [aria_aesni_avx_x86_64] [ 10.871912] crypto_skcipher_encrypt+0x3a/0x70 [skcipher] [ 10.872527] simd_skcipher_encrypt+0x8f/0xd0 [crypto_simd] [ 10.873165] crypto_skcipher_encrypt+0x3a/0x70 [skcipher] [ 10.873789] test_skcipher_vec_cfg+0x286/0x6a0 [cryptomgr] [ 10.874432] ? number+0x305/0x350 [ 10.874900] ? skcipher_walk_next+0x524/0x570 [skcipher] [ 10.875508] ? string+0x5c/0xe0 [ 10.875942] ? vsnprintf+0x256/0x4e0 [ 10.876412] ? vsnprintf+0x385/0x4e0 [ 10.876878] ? scnprintf+0x46/0x70 [ 10.877359] ? valid_sg_divisions.constprop.0+0x8a/0xa0 [cryptomgr] [ 10.878058] ? valid_testvec_config+0xb7/0xd0 [cryptomgr] [ 10.878699] ? generate_random_testvec_config.constprop.0+0x166/0x370 [cryptomgr] [ 10.879496] test_skcipher_vs_generic_impl+0x401/0x610 [cryptomgr] [ 10.880194] alg_test_skcipher+0x19d/0x1b0 [cryptomgr] [ 10.880805] alg_test.part.0+0x335/0x3b0 [cryptomgr] [ 10.881425] ? __schedule+0x274/0xf90 [ 10.881916] ? ttwu_queue_wakelist+0xf4/0x100 [ 10.882464] ? _raw_spin_unlock_irqrestore+0xd/0x40 [ 10.883041] ? try_to_wake_up+0x1f6/0x2d0 [ 10.883537] ? __pfx_cryptomgr_test+0x10/0x10 [cryptomgr] [ 10.884145] alg_test+0x18/0x50 [cryptomgr] [ 10.884662] cryptomgr_test+0x24/0x50 [cryptomgr] [ 10.885240] kthread+0xe9/0x110 [ 10.885672] ? __pfx_kthread+0x10/0x10 [ 10.886144] ret_from_fork+0x2c/0x50 [ 10.886599] </TASK> [ 10.886939] Modules linked in: aria_aesni_avx_x86_64(+) aria_generic aegis128 aegis128_aesni chacha20poly1305 cmac camellia_aesni_avx_x86_64 camellia_generic camellia_x86_64 lzo lzo_compress lzo_decompress cast6_avx_x86_64 cast6_generic cast5_avx_x86_64 cast5_generic cast_common deflate ccm serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common twofish_avx_x86_64 twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common xcbc md5 des_generic libdes xt_CHECKSUM nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_tcpudp nft_compat bridge stp llc nf_tables libcrc32c nfnetlink intel_rapl_msr intel_rapl_common iosf_mbi kvm_intel kvm irqbypass crc32_generic crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel af_packet sha512_ssse3 sha512_generic xctr ghash_generic gf128mul gcm crypto_null xts ctr sd_mod t10_pi crc64_rocksoft_generic crc64_rocksoft crc_t10dif cts [ 10.886989] crct10dif_generic sr_mod cdrom crct10dif_pclmul crc64 cbc crct10dif_common joydev sg mousedev ppdev bochs drm_vram_helper drm_ttm_helper aes_generic ttm drm_kms_helper ecb drm drm_panel_orientation_quirks snd_pcm cfbfillrect cfbimgblt cfbcopyarea syscopyarea snd_timer sysfillrect aesni_intel sysimgblt snd fb soundcore psmouse ata_generic virtio_net libaes evdev libcryptoutils crypto_simd pata_acpi input_leds fbdev cryptd rapl led_class net_failover failover backlight pcspkr serio_raw ata_piix floppy libata intel_agp i2c_piix4 intel_gtt scsi_mod agpgart rtc_cmos parport_pc i2c_core parport scsi_common tiny_power_button button qemu_fw_cfg ip_tables x_tables sha256_ssse3 sha256_generic libsha256 sha1_ssse3 sha1_generic hmac ipv6 autofs4 ext4 crc16 mbcache jbd2 dm_snapshot dm_bufio dm_mod virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio_blk virtio_ring virtio unix crc32c_generic crc32c_intel cryptomgr kpp crypto_acompress akcipher rng aead crypto_hash skcipher [ 10.895975] crypto_algapi crypto [ 10.905842] ---[ end trace 0000000000000000 ]---
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [PATCH v4 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64/GFNI assembler implementation of aria cipher
Hi Herbert,
Thank you so much for the report!
Quoted 10 lines · Herbert Xu
arch/x86/crypto/aria-aesni-avx-asm_64.S line 275
Quoted 6 lines · Taehee Yoo
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +5
-0+#define aria_ark_8way(x0, x1, x2, x3, \ + x4, x5, x6, x7, \ + t0, rk, idx, round) \ + /* AddRoundKey */ \ + vpbroadcastb ((round * 16) + idx + 3)(rk), t0; \This triggers an invalid OP code on CPUs that supports AVX but not AVX2:
Thanks for it, vpbroadcastX instructions are not supported by AVX. So, I think vbroadcastss should be used instead.
I will send the fix patch for it soon.
Quoted 3 lines
[ 10.857327] invalid opcode: 0000 [#1] PREEMPT SMP [ 10.857935] CPU: 1 PID: 2816 Comm: cryptomgr_test Tainted: G
W 6.2.0-rc1+ #1
Quoted 49 lines
[ 10.858783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014 [ 10.859622] RIP: 0010:__aria_aesni_avx_crypt_16way+0x22d/0x6000 [aria_aesni_avx_x86_64] [ 10.860440] Code: c4 41 7a 7f 48 10 c4 41 7a 7f 50 20 c4 41 7a 7f 58 30 c4 41 7a 7f 60 40 c4 41 7a 7f 68 50 c4 41 7a 7f 70 60 c4 41 7a 7f 78 70 <c4> c2 79 78 41 0b c5 39 ef c0 c4 c2 79 78 41 0a c5 31 ef c8 c4 c2 [ 10.862409] RSP: 0018:ffffc900002eb728 EFLAGS: 00010206 [ 10.863011] RAX: ffff88800666b00f RBX: ffff88800666b00f RCX: ffffffffa0c02010 [ 10.863754] RDX: ffff88800666b00f RSI: ffff88800666b00f RDI: ffff88800408a420 [ 10.864500] RBP: ffffc900002eb728 R08: ffff88800666b08f R09: ffff88800408a420 [ 10.865270] R10: ffff8880064de040 R11: 00000000fffffff0 R12: ffff88800408a420 [ 10.866019] R13: 000000000000011b R14: ffff88800666b00f R15: ffff88800666b00f [ 10.866764] FS: 0000000000000000(0000) GS:ffff88801ed00000(0000) knlGS:0000000000000000 [ 10.867587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 10.868230] CR2: 00007f8c80f160a4 CR3: 0000000001e0a002 CR4: 0000000000060ee0 [ 10.868986] Call Trace: [ 10.869392] <TASK> [ 10.869744] aria_aesni_avx_encrypt_16way+0x77/0xf0 [aria_aesni_avx_x86_64] [ 10.870489] ecb_do_encrypt.constprop.0+0x100/0x140 [aria_aesni_avx_x86_64] [ 10.871233] aria_avx_ecb_encrypt+0xd/0x20 [aria_aesni_avx_x86_64] [ 10.871912] crypto_skcipher_encrypt+0x3a/0x70 [skcipher] [ 10.872527] simd_skcipher_encrypt+0x8f/0xd0 [crypto_simd] [ 10.873165] crypto_skcipher_encrypt+0x3a/0x70 [skcipher] [ 10.873789] test_skcipher_vec_cfg+0x286/0x6a0 [cryptomgr] [ 10.874432] ? number+0x305/0x350 [ 10.874900] ? skcipher_walk_next+0x524/0x570 [skcipher] [ 10.875508] ? string+0x5c/0xe0 [ 10.875942] ? vsnprintf+0x256/0x4e0 [ 10.876412] ? vsnprintf+0x385/0x4e0 [ 10.876878] ? scnprintf+0x46/0x70 [ 10.877359] ? valid_sg_divisions.constprop.0+0x8a/0xa0 [cryptomgr] [ 10.878058] ? valid_testvec_config+0xb7/0xd0 [cryptomgr] [ 10.878699] ? generate_random_testvec_config.constprop.0+0x166/0x370 [cryptomgr] [ 10.879496] test_skcipher_vs_generic_impl+0x401/0x610 [cryptomgr] [ 10.880194] alg_test_skcipher+0x19d/0x1b0 [cryptomgr] [ 10.880805] alg_test.part.0+0x335/0x3b0 [cryptomgr] [ 10.881425] ? __schedule+0x274/0xf90 [ 10.881916] ? ttwu_queue_wakelist+0xf4/0x100 [ 10.882464] ? _raw_spin_unlock_irqrestore+0xd/0x40 [ 10.883041] ? try_to_wake_up+0x1f6/0x2d0 [ 10.883537] ? __pfx_cryptomgr_test+0x10/0x10 [cryptomgr] [ 10.884145] alg_test+0x18/0x50 [cryptomgr] [ 10.884662] cryptomgr_test+0x24/0x50 [cryptomgr] [ 10.885240] kthread+0xe9/0x110 [ 10.885672] ? __pfx_kthread+0x10/0x10 [ 10.886144] ret_from_fork+0x2c/0x50 [ 10.886599] </TASK> [ 10.886939] Modules linked in: aria_aesni_avx_x86_64(+) aria_generic aegis128 aegis128_aesni chacha20poly1305 cmac camellia_aesni_avx_x86_64 camellia_generic camellia_x86_64 lzo lzo_compress lzo_decompress cast6_avx_x86_64 cast6_generic cast5_avx_x86_64 cast5_generic cast_common deflate ccm serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common twofish_avx_x86_64 twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common xcbc md5 des_generic libdes xt_CHECKSUM nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_tcpudp nft_compat bridge stp llc nf_tables libcrc32c nfnetlink intel_rapl_msr intel_rapl_common iosf_mbi kvm_intel kvm irqbypass crc32_generic crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel af_packet sha512_ssse3 sha512_generic xctr ghash_generic gf128mul gcm crypto_null xts ctr sd_mod t10_pi crc64_rocksoft_generic crc64_rocksoft crc_t10dif cts [ 10.886989] crct10dif_generic sr_mod cdrom crct10dif_pclmul crc64 cbc crct10dif_common joydev sg mousedev ppdev bochs drm_vram_helper drm_ttm_helper aes_generic ttm drm_kms_helper ecb drm drm_panel_orientation_quirks snd_pcm cfbfillrect cfbimgblt cfbcopyarea syscopyarea snd_timer sysfillrect aesni_intel sysimgblt snd fb soundcore psmouse ata_generic virtio_net libaes evdev libcryptoutils crypto_simd pata_acpi input_leds fbdev cryptd rapl led_class net_failover failover backlight pcspkr serio_raw ata_piix floppy libata intel_agp i2c_piix4 intel_gtt scsi_mod agpgart rtc_cmos parport_pc i2c_core parport scsi_common tiny_power_button button qemu_fw_cfg ip_tables x_tables sha256_ssse3 sha256_generic libsha256 sha1_ssse3 sha1_generic hmac ipv6 autofs4 ext4 crc16 mbcache jbd2 dm_snapshot dm_bufio dm_mod virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio_blk virtio_ring virtio unix crc32c_generic crc32c_intel cryptomgr kpp crypto_acompress akcipher rng aead crypto_hash skcipher [ 10.895975] crypto_algapi crypto [ 10.905842] ---[ end trace 0000000000000000 ]---
Quoted 1 line
Cheers,
Thank you so much,
Taehee Yoo
PATCH v4 3/3 crypto: tcrypt: add async speed test for aria cipher
In reply to Taehee YooIn order to test for the performance of aria-avx implementation, it needs an async speed test.
So, it adds async speed tests to the tcrypt.
---
v4:
- No changes.
v3:
- Add aria-ctr async speed test.
- Add aria-gcm multi buffer speed test
v2:
- No changes.
crypto/tcrypt.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
crypto/tcrypt.c +30 -0
@@ -2205,6 +2205,13 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb) NULL, 0, 16, 8, speed_template_16_24_32);break ; +case 229 : +test_mb_aead_speed ("gcm(aria)" ,ENCRYPT , sec,NULL ,0 ,16 ,8 , + speed_template_16, num_mb); +test_mb_aead_speed ("gcm(aria)" ,DECRYPT , sec,NULL ,0 ,16 ,8 , + speed_template_16, num_mb); +break ; +case 300 :if (alg) {test_hash_speed (alg, sec, generic_hash_speed_template);
@@ -2625,6 +2632,17 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb) speed_template_16);break ; +case 519 : +test_acipher_speed ("ecb(aria)" ,ENCRYPT , sec,NULL ,0 , + speed_template_16_24_32); +test_acipher_speed ("ecb(aria)" ,DECRYPT , sec,NULL ,0 , + speed_template_16_24_32); +test_acipher_speed ("ctr(aria)" ,ENCRYPT , sec,NULL ,0 , + speed_template_16_24_32); +test_acipher_speed ("ctr(aria)" ,DECRYPT , sec,NULL ,0 , + speed_template_16_24_32); +break ; +case 600 :test_mb_skcipher_speed ("ecb(aes)" ,ENCRYPT , sec,NULL ,0 , speed_template_16_24_32, num_mb);
@@ -2836,6 +2854,18 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)test_mb_skcipher_speed ("ctr(blowfish)" ,DECRYPT , sec,NULL ,0 , speed_template_8_32, num_mb);break ; + +case 610 : +test_mb_skcipher_speed ("ecb(aria)" ,ENCRYPT , sec,NULL ,0 , + speed_template_16_32, num_mb); +test_mb_skcipher_speed ("ecb(aria)" ,DECRYPT , sec,NULL ,0 , + speed_template_16_32, num_mb); +test_mb_skcipher_speed ("ctr(aria)" ,ENCRYPT , sec,NULL ,0 , + speed_template_16_32, num_mb); +test_mb_skcipher_speed ("ctr(aria)" ,DECRYPT , sec,NULL ,0 , + speed_template_16_32, num_mb); +break ; + }return ret;
Signature · 1 line
2.17.1
Herbert Xu
maintainerQuoted 164 lines · Taehee Yoo
The purpose of this patchset is to support the implementation of ARIA-AVX. Many of the ideas in this implementation are from Camellia-avx, especially byte slicing.
Like Camellia, ARIA also uses a 16way strategy.ARIA cipher algorithm is similar to AES.
There are four s-boxes in the ARIA spec and the first and second s-boxes are the same as AES's s-boxes.
Almost functions are based on aria-generic code except for s-box related function.
The aria-avx doesn't implement the key expanding function. it supports onlyencrypt()anddecrypt().Encryption and Decryption are actually the same but it should use separated keys(encryption key and decryption key). En/Decryption steps are like below:
1. Add-Round-Key
2. S-box.
3. Diffusion Layer.There is no special thing in the Add-Round-Key step.
There are some notable things in s-box step.
Like Camellia, it doesn't use a lookup table, instead, it uses AES-NI. There are 2 implementations for that.
One is to use AES-NI and affine transformation, which is the same as Camellia, sm4, and others.
Another is to use GFNI.
GFNI implementation is faster than AES-NI implementation. So, it uses GFNI implementation if the running CPU supports GFNI.To calculate the first s-box(S1), it just uses the aesenclast and then inverts
shift_row. No more process is needed for this job because the first s-box is the same as the AES encryption s-box.To calculate the second s-box(X1, invert of S1), it just uses the aesdeclast and then inverts
shift_row. No more process is needed for this job because the second s-box is the same as the AES decryption s-box.To calculate the third s-box(S2), it uses the aesenclast, then affine transformation, which is combined AES inverse affine and ARIA S2.
To calculate the last s-box(X2, invert of S2), it uses the aesdeclast, then affine transformation, which is combined X2 and AES forward affine.
The optimized third and last s-box logic and GFNI s-box logic are implemented by Jussi Kivilinna.
The aria-generic implementation is based on a 32-bit implementation, not an 8-bit implementation.
The aria-avx Diffusion Layer implementation is based on aria-generic implementation because 8-bit implementation is not fit for parallel implementation but 32-bit is fit for this.The first patch in this series is to export functions for aria-avx. The aria-avx uses existing functions in the aria-generic code. The second patch is to implement aria-avx.
The last patch is to add async test for aria.Benchmarks:
The tcrypt is used.
cpu: i3-12100How to test:
modprobe aria-generic tcrypt mode=610 num_mb=8192Result:
testing speed of multibuffer ecb(aria) (ecb(aria-generic)) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 534 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2006 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 3674 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 52374 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 608 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2586 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 4707 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 69794 cyclestesting speed of multibuffer ecb(aria) (ecb(aria-generic)) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 545 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 1995 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 3673 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 52359 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 615 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2588 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 4712 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 69916 cyclesHow to test:
modprobe aria tcrypt mode=610 num_mb=8192AVX with AES-NI:
testing speed of multibuffer ecb(aria) (ecb-aria-avx) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 629 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2060 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 1223 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 11931 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 686 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2616 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 1439 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 15488 cyclestesting speed of multibuffer ecb(aria) (ecb-aria-avx) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 609 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2027 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 1211 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 12040 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 684 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2614 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 1445 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 15478 cyclesAVX with GFNI:
testing speed of multibuffer ecb(aria) (ecb-aria-avx) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 730 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2056 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 1028 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 9223 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 685 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2603 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 1179 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 11728 cyclestesting speed of multibuffer ecb(aria) (ecb-aria-avx) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 617 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2057 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 1020 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 9280 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 687 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2599 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 1176 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 11909 cyclesv4:
- Fix sparse warning.
- Remove .align statement for .text
- https://lkml.kernel.org/r/[email protected]v3:
- Use ECB macro instead of opencode.
- Implement ctr(aria-avx).
- Improve performance(20% ~ 30%) with combined affine transformation for S2 and X2.
- Implemented by Jussi Kivilinna.
- Improve performance( ~ 55%) with GFNI.
- Implemented by Jussi Kivilinna.
- Add aria-ctr async speed test.
- Add aria-gcm multi buffer speed test
- Rebase and fixKconfigv2:
- Do not call non-FPUfunctions(aria_{encrypt | decrypt}())in the FPU context.
- Do not acquire FPU context for too long.Taehee Yoo (3):
crypto: aria: prepare generic module for optimized implementations
crypto: aria-avx: add AES-NI/AVX/x86_64/GFNI assembler implementation
of aria cipher
crypto: tcrypt: add async speed test for aria cipher
Quoted 17 lines
arch/x86/crypto/Kconfig | 18 + arch/x86/crypto/Makefile | 3 + arch/x86/crypto/aria-aesni-avx-asm_64.S | 1303 +++++++++++++++++++++++ arch/x86/crypto/aria-avx.h | 16 + arch/x86/crypto/aria_aesni_avx_glue.c | 213 ++++ crypto/Makefile | 2 +- crypto/{aria.c => aria_generic.c} | 39 +- crypto/tcrypt.c | 30 + include/crypto/aria.h | 17 +- 9 files changed, 1623 insertions(+), 18 deletions(-) create mode 100644 arch/x86/crypto/aria-aesni-avx-asm_64.S create mode 100644 arch/x86/crypto/aria-avx.h create mode 100644 arch/x86/crypto/aria_aesni_avx_glue.c rename crypto/{aria.c => aria_generic.c} (86%)Signature · 1 line
2.17.1
All applied. Thanks.
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt