crypto: aria: add ARIA AES-NI/AVX/x86_64 implementation
Subsystems (2)
v2 cover crypto: aria: add ARIA AES-NI/AVX/x86_64 implementation
The purpose of this patchset is to support the implementation of ARIA-AVX. Many of the ideas in this implementation are from Camellia-avx, especially byte slicing.
Like Camellia, ARIA also uses a 16way strategy.
ARIA cipher algorithm is similar to AES.
There are four s-boxes in the ARIA spec and the first and second s-boxes are the same as AES's s-boxes.
Almost functions are based on aria-generic code except for s-box related function.
The aria-avx doesn't implement the key expanding function. it only support encrypt() and decrypt().
Encryption and Decryption logic is actually the same but it should use separated keys(encryption key and decryption key). En/Decryption steps are like below:
1. Add-Round-Key
2. S-box.
3. Diffusion Layer.
There is no special thing in the Add-Round-Key step.
There are some notable things in s-box step.
Like Camellia, it doesn't use a lookup table, instead, it uses aes-ni.
To calculate the first s-box, it just uses the aesenclast and then inverts shift_row. No more process is needed for this job because the first s-box is the same as the AES encryption s-box.
To calculate a second s-box(invert of s1), it just uses the aesdeclast and then inverts shift_row. No more process is needed for this job because the second s-box is the same as the AES decryption s-box.
To calculate a third and fourth s-boxes, it uses the aesenclast, then inverts shift_row, and affine transformation.
The aria-generic implementation is based on a 32-bit implementation, not an 8-bit implementation.
The aria-avx Diffusion Layer implementation is based on aria-generic implementation because 8-bit implementation is not fit for parallel implementation but 32-bit is fit for this.
The first patch in this series is to export functions for aria-avx. The aria-avx uses existing functions in the aria-generic code. The second patch is to implement aria-avx.
The last patch is to add async test for aria.
Benchmarks:
The tcrypt is used.
cpu: i3-12100
How to test:
modprobe aria-generic
tcrypt mode=610 num_mb=8192
Result:
testing speed of multibuffer ecb(aria) (ecb(aria-generic)) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 534 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2006 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 3674 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 52374 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 608 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2586 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 4707 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 69794 cycles
testing speed of multibuffer ecb(aria) (ecb(aria-generic)) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 545 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 1995 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 3673 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 52359 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 615 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2588 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 4712 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 69916 cycles
How to test:
modprobe aria
tcrypt mode=610 num_mb=8192
Result:
testing speed of multibuffer ecb(aria) (ecb-aria-avx) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 727 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2040 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 1399 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 14758 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 702 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2615 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 1677 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 19454 cycles
testing speed of multibuffer ecb(aria) (ecb-aria-avx) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 638 cycles test 2 (128 bit key, 128 byte blocks): 1 operation in 2090 cycles test 3 (128 bit key, 256 byte blocks): 1 operation in 1394 cycles test 6 (128 bit key, 4096 byte blocks): 1 operation in 14824 cycles test 7 (256 bit key, 16 byte blocks): 1 operation in 719 cycles test 9 (256 bit key, 128 byte blocks): 1 operation in 2633 cycles test 10 (256 bit key, 256 byte blocks): 1 operation in 1684 cycles test 13 (256 bit key, 4096 byte blocks): 1 operation in 19457 cycles
v2:
- Do not call non-FPU functions(aria_{encrypt | decrypt}() in the FPU context.
- Do not acquire FPU context for too long.
Taehee Yoo (3):
crypto: aria: prepare generic module for optimized implementations
crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of
aria cipher
crypto: tcrypt: add async speed test for aria cipher
arch/x86/crypto/Makefile | 3 +
arch/x86/crypto/aria-aesni-avx-asm_64.S | 648 ++++++++++++++++++++++++
arch/x86/crypto/aria_aesni_avx_glue.c | 165 ++++++
crypto/Kconfig | 21 +
crypto/Makefile | 2 +-
crypto/{aria.c => aria_generic.c} | 39 +-
crypto/tcrypt.c | 13 +
include/crypto/aria.h | 14 +-
8 files changed, 889 insertions(+), 16 deletions(-)
create mode 100644 arch/x86/crypto/aria-aesni-avx-asm_64.S
create mode 100644 arch/x86/crypto/aria_aesni_avx_glue.c
rename crypto/{aria.c => aria_generic.c} (86%)Signature · 1 line
2.17.1
PATCH v2 1/3 crypto: aria: prepare generic module for optimized implementations
It renames aria to aria_generic and exports some functions such as aria_set_key(), aria_encrypt(), and aria_decrypt() to be able to be used by aria-avx implementation.
---
v2:
- No changes
crypto/Makefile | 2 +-
crypto/{aria.c => aria_generic.c} | 39 +++++++++++++++++++++++++------
include/crypto/aria.h | 14 +++++------
3 files changed, 39 insertions(+), 16 deletions(-)
rename crypto/{aria.c => aria_generic.c} (86%)crypto/Makefile +1 -1
@@ -149,7 +149,7 @@ obj-$(CONFIG_CRYPTO_TEA) += tea.o obj-$(CONFIG_CRYPTO_KHAZAD) += khazad.o obj-$(CONFIG_CRYPTO_ANUBIS) += anubis.o obj-$(CONFIG_CRYPTO_SEED) += seed.o -obj-$(CONFIG_CRYPTO_ARIA) += aria.o +obj-$(CONFIG_CRYPTO_ARIA) += aria_generic.o obj-$(CONFIG_CRYPTO_CHACHA20) += chacha_generic.o obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_generic.o obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o
crypto/aria_generic.c renamed from crypto/aria.c +32 -7
@@ -16,6 +16,14 @@#include <crypto/aria.h> +static const u32 key_rc[20 ] = { +0x517cc1b7 ,0x27220a94 ,0xfe13abe8 ,0xfa9a6ee0 , +0x6db14acc ,0x9e21c820 ,0xff28b1d5 ,0xef5de2b0 , +0xdb92371d ,0x2126e970 ,0x03249775 ,0x04e8c90e , +0x517cc1b7 ,0x27220a94 ,0xfe13abe8 ,0xfa9a6ee0 , +0x6db14acc ,0x9e21c820 ,0xff28b1d5 ,0xef5de2b0 +}; +static void aria_set_encrypt_key (struct aria_ctx *ctx,const u8 *in_key,unsigned int key_len) {
@@ -25,7 +33,7 @@ static void aria_set_encrypt_key(struct aria_ctx *ctx, const u8 *in_key,const u32 *ck;int rkidx =0 ; - ck = &key_rc[(key_len -16 ) /8 ][0 ]; + ck = &key_rc[(key_len -16 ) /2 ]; w0[0 ] =be32_to_cpu (key[0 ]); w0[1 ] =be32_to_cpu (key[1 ]);
@@ -163,8 +171,7 @@ static void aria_set_decrypt_key(struct aria_ctx *ctx) } } -static int aria_set_key (struct crypto_tfm *tfm,const u8 *in_key, -unsigned int key_len) +int aria_set_key (struct crypto_tfm *tfm,const u8 *in_key,unsigned int key_len) {struct aria_ctx *ctx =crypto_tfm_ctx (tfm);
@@ -179,6 +186,7 @@ static int aria_set_key(struct crypto_tfm *tfm, const u8 *in_key,return 0 ; } +EXPORT_SYMBOL_GPL (aria_set_key);static void __aria_crypt (struct aria_ctx *ctx,u8 *out,const u8 *in,u32 key[][ARIA_RD_KEY_WORDS ])
@@ -235,14 +243,30 @@ static void __aria_crypt(struct aria_ctx *ctx, u8 *out, const u8 *in, dst[3 ] =cpu_to_be32 (reg3); } -static void aria_encrypt (struct crypto_tfm *tfm,u8 *out,const u8 *in) +void aria_encrypt (void *_ctx,u8 *out,const u8 *in) +{ +struct aria_ctx *ctx = (struct aria_ctx *)_ctx; + +__aria_crypt (ctx, out, in, ctx->enc_key); +} +EXPORT_SYMBOL_GPL (aria_encrypt); + +void aria_decrypt (void *_ctx,u8 *out,const u8 *in) +{ +struct aria_ctx *ctx = (struct aria_ctx *)_ctx; + +__aria_crypt (ctx, out, in, ctx->dec_key); +} +EXPORT_SYMBOL_GPL (aria_decrypt); + +static void __aria_encrypt (struct crypto_tfm *tfm,u8 *out,const u8 *in) {struct aria_ctx *ctx =crypto_tfm_ctx (tfm);__aria_crypt (ctx, out, in, ctx->enc_key); } -static void aria_decrypt (struct crypto_tfm *tfm,u8 *out,const u8 *in) +static void __aria_decrypt (struct crypto_tfm *tfm,u8 *out,const u8 *in) {struct aria_ctx *ctx =crypto_tfm_ctx (tfm);
@@ -263,8 +287,8 @@ static struct crypto_alg aria_alg = { .cia_min_keysize = ARIA_MIN_KEY_SIZE, .cia_max_keysize = ARIA_MAX_KEY_SIZE, .cia_setkey = aria_set_key, - .cia_encrypt = aria_encrypt, - .cia_decrypt = aria_decrypt + .cia_encrypt = __aria_encrypt, + .cia_decrypt = __aria_decrypt } } };
@@ -286,3 +310,4 @@ MODULE_DESCRIPTION("ARIA Cipher Algorithm");MODULE_LICENSE ("GPL" );MODULE_AUTHOR ("Taehee Yoo <[email protected]>" );MODULE_ALIAS_CRYPTO ("aria" ); +MODULE_ALIAS_CRYPTO ("aria-generic" );
include/crypto/aria.h +6 -8
@@ -28,6 +28,7 @@#define ARIA_MIN_KEY_SIZE 16#define ARIA_MAX_KEY_SIZE 32#define ARIA_BLOCK_SIZE 16 +#define ARIA_AVX_BLOCK_SIZE (ARIA_BLOCK_SIZE * 16)#define ARIA_MAX_RD_KEYS 17#define ARIA_RD_KEY_WORDS (ARIA_BLOCK_SIZE / sizeof(u32))
@@ -38,14 +39,6 @@ struct aria_ctx {u32 dec_key[ARIA_MAX_RD_KEYS ][ARIA_RD_KEY_WORDS ]; }; -static const u32 key_rc[5 ][4 ] = { - {0x517cc1b7 ,0x27220a94 ,0xfe13abe8 ,0xfa9a6ee0 }, - {0x6db14acc ,0x9e21c820 ,0xff28b1d5 ,0xef5de2b0 }, - {0xdb92371d ,0x2126e970 ,0x03249775 ,0x04e8c90e }, - {0x517cc1b7 ,0x27220a94 ,0xfe13abe8 ,0xfa9a6ee0 }, - {0x6db14acc ,0x9e21c820 ,0xff28b1d5 ,0xef5de2b0 } -}; -static const u32 s1[256 ] = {0x00636363 ,0x007c7c7c ,0x00777777 ,0x007b7b7b ,0x00f2f2f2 ,0x006b6b6b ,0x006f6f6f ,0x00c5c5c5 ,
@@ -458,4 +451,9 @@ static inline void aria_gsrk(u32 *rk, u32 *x, u32 *y, u32 n) ((y[(q + 2) % 4]) << (32 - r)); } +void aria_encrypt (void *ctx,u8 *out,const u8 *in); +void aria_decrypt (void *ctx,u8 *out,const u8 *in); +int aria_set_key (struct crypto_tfm *tfm,const u8 *in_key, +unsigned int key_len); +#endif
Signature · 1 line
2.17.1
PATCH v2 2/3 crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of aria cipher
In reply to Taehee YooThe implementation is based on the 32-bit implementation of the aria. Also, aria-avx process steps are the similar to the camellia-avx.
1. Byteslice(16way)
2. Add-round-key.
3. Sbox
4. Diffusion layer.
Except for s-box, all steps are the same as the aria-generic implementation. s-box step is very similar to camellia and sm4 implementation.
There are 4 s-boxes in the ARIA and the 2 s-boxes are the same as AES's s-boxes. The basic strategy is to use the aes-ni.
To calculate a first sbox, it just uses the aesenclast and then inverts shift_row.
No more process is needed for this job because the first s-box is the same as the AES encryption s-box.
To calculate a second sbox(invert of s1), it just uses the aesdeclast and then inverts shift_row.
No more process is needed for this job because the second s-box is the same as the AES decryption s-box.
To calculate a third and fourth s-boxes, it uses the aesenclast, then inverts shift_row, and affine transformation.
The aria-generic implementation is based on a 32-bit implementation, not an 8-bit implementation. the aria-avx Diffusion Layer implementation is based on aria-generic implementation because 8-bit implementation is not fit for parallel implementation but 32-bit is enough to fit for this.
---
v2:
- Do not call non-FPU functions(aria_{encrypt | decrypt}() in the FPU context.
- Do not acquire FPU context for too long.
arch/x86/crypto/Makefile | 3 + arch/x86/crypto/aria-aesni-avx-asm_64.S | 648 ++++++++++++++++++++++++ arch/x86/crypto/aria_aesni_avx_glue.c | 165 ++++++ crypto/Kconfig | 21 + 4 files changed, 837 insertions(+) create mode 100644 arch/x86/crypto/aria-aesni-avx-asm_64.S create mode 100644 arch/x86/crypto/aria_aesni_avx_glue.c
arch/x86/crypto/Makefile +3 -0
@@ -100,6 +100,9 @@ sm4-aesni-avx-x86_64-y := sm4-aesni-avx-asm_64.o sm4_aesni_avx_glue.o obj-$(CONFIG_CRYPTO_SM4_AESNI_AVX2_X86_64) += sm4-aesni-avx2-x86_64.o sm4-aesni-avx2-x86_64-y := sm4-aesni-avx2-asm_64.o sm4_aesni_avx2_glue.o +obj-$(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) += aria-aesni-avx-x86_64.o +aria-aesni-avx-x86_64-y := aria-aesni-avx-asm_64.o aria_aesni_avx_glue.o + quiet_cmd_perlasm = PERLASM $@ cmd_perlasm = $(PERL) $< > $@ $(obj)/%.S: $(src)/%.pl FORCE
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +648 -0
648 lines not on this page · raw message
arch/x86/crypto/aria_aesni_avx_glue.c new file +165 -0
165 lines not on this page · raw message
crypto/Kconfig +21 -0
@@ -1659,6 +1659,27 @@ config CRYPTO_ARIA See also: <https://seed.kisa.or.kr/kisa/algorithm/EgovAriaInfo.do> +config CRYPTO_ARIA_AESNI_AVX_X86_64 + tristate "ARIA cipher algorithm (x86_64/AES-NI/AVX)" + depends on X86 && 64BIT + select CRYPTO_SKCIPHER + select CRYPTO_ARIA + select CRYPTO_SIMD + help + ARIA cipher algorithm (RFC5794). + + ARIA is a standard encryption algorithm of the Republic of Korea. + The ARIA specifies three key sizes and rounds. + 128-bit: 12 rounds. + 192-bit: 14 rounds. + 256-bit: 16 rounds. + + This module provides the ARIA cipher algorithm that processes + sixteen blocks parallel using the AVX instruction set. + + See also: + <https://seed.kisa.or.kr/kisa/algorithm/EgovAriaInfo.do> + config CRYPTO_SERPENT tristate "Serpent cipher algorithm" select CRYPTO_ALGAPI
Signature · 1 line
2.17.1
RE: [PATCH v2 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of aria cipher
Quoted 12 lines
-----Original Message-----
From: Taehee Yoo <[email protected]>
Sent: Friday, August 26, 2022 12:32 AM
Subject: [PATCH v2 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of aria cipherv2:
- Do not call non-FPU functions(aria_{encrypt | decrypt}() in the FPU context.
- Do not acquire FPU context for too long.
...
Quoted 2 lines
(quoted diff) +2
-0+static int ecb_do_encrypt(struct skcipher_request *req, const u32 *rkey) +{
...
arch/x86/crypto/aria_aesni_avx_glue.c line 41
Quoted 12 lines
arch/x86/crypto/aria_aesni_avx_glue.c new file +12
-0+while ((nbytes = walk.nbytes) >0 ) { +const u8 *src = walk.src.virt.addr; +u8 *dst = walk.dst.virt.addr; + +kernel_fpu_begin (); +while (nbytes >=ARIA_AVX_BLOCK_SIZE ) { +aria_aesni_avx_crypt_16way (rkey, dst, src, ctx->rounds); + dst +=ARIA_AVX_BLOCK_SIZE ; + src +=ARIA_AVX_BLOCK_SIZE ; + nbytes -=ARIA_AVX_BLOCK_SIZE ; + } +kernel_fpu_end ();
Per Herbert's reply on the sha512-avx RCU stall issue, another nesting level might be necessary limiting the amount of data processed between each kernel_fpu_begin() to kernel_fpu_end() pair to 4 KiB.
If you modify this driver to use the ECB_WALK_START, ECB_BLOCK, and ECB_WALK_END macros from ecb_cbc_helpers.h and incorporate that fix, then your fix would be easy to replicate into the other users (camellia, cast5, cast6, serpent, and twofish).
Re: [PATCH v2 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of aria cipher
Hi Elliott, Robert
Thanks for your review!
2022. 8. 27. 오전 12:12에 Elliott, Robert (Servers) 이(가) 쓴 글:
Quoted 34 lines
Quoted 12 lines
-----Original Message-----
From: Taehee Yoo <[email protected]>
Sent: Friday, August 26, 2022 12:32 AM
Subject: [PATCH v2 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of aria cipherv2:
- Do not call non-FPU functions(aria_{encrypt | decrypt}() in the FPU context.
- Do not acquire FPU context for too long....
Quoted 2 lines
(quoted diff) +2
-0+static int ecb_do_encrypt(struct skcipher_request *req, const u32 *rkey) +{...
arch/x86/crypto/aria_aesni_avx_glue.c line 41
Quoted 12 lines
arch/x86/crypto/aria_aesni_avx_glue.c new file +12
-0+while ((nbytes = walk.nbytes) >0 ) { +const u8 *src = walk.src.virt.addr; +u8 *dst = walk.dst.virt.addr; + +kernel_fpu_begin (); +while (nbytes >=ARIA_AVX_BLOCK_SIZE ) { +aria_aesni_avx_crypt_16way (rkey, dst, src, ctx->rounds); + dst +=ARIA_AVX_BLOCK_SIZE ; + src +=ARIA_AVX_BLOCK_SIZE ; + nbytes -=ARIA_AVX_BLOCK_SIZE ; + } +kernel_fpu_end ();Per Herbert's reply on the sha512-avx RCU stall issue, another nesting level might be necessary limiting the amount of data processed between each
kernel_fpu_begin()tokernel_fpu_end()pair to 4 KiB.
Quoted 5 lines
If you modify this driver to use the
ECB_WALK_START,ECB_BLOCK, andECB_WALK_ENDmacros fromecb_cbc_helpers.hand incorporate that fix, then your fix would be easy to replicate into the other users (camellia, cast5, cast6, serpent, and twofish).
Now I understand why you suggested using ECB macro instead of open code. I think your idea is nice for many users such as camellia, cast5, etc.
I will use ECB macro in the v3 patch.
Then, I will send a fixing the RCU stall issue patch separately instead of containing it.
Thanks a lot!
Taehee Yoo
Re: [PATCH v2 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of aria cipher
In reply to Taehee YooQuoted 6 lines · Taehee Yoo
arch/x86/crypto/aria_aesni_avx_glue.c new file +6
-0+static struct skcipher_alg aria_algs[] = { + { + .base.cra_name ="__ecb(aria)" , + .base.cra_driver_name ="__ecb-aria-avx" , + .base.cra_priority =400 , + .base.cra_flags =CRYPTO_ALG_INTERNAL ,
arch/x86/crypto/aria_aesni_avx_glue.c line 126
Quoted 10 lines
arch/x86/crypto/aria_aesni_avx_glue.c new file +10
-0+ .base.cra_blocksize = ARIA_BLOCK_SIZE, + .base.cra_ctxsize = sizeof(struct aria_ctx), + .base.cra_module = THIS_MODULE, + .min_keysize = ARIA_MIN_KEY_SIZE, + .max_keysize = ARIA_MAX_KEY_SIZE, + .setkey = aria_avx_set_key, + .encrypt = aria_avx_ecb_encrypt, + .decrypt = aria_avx_ecb_decrypt, + } +};
Why do you want ECB mode and nothing else? At
https://lore.kernel.org/r/[email protected]
you claimed that the use case for ARIA support in the kernel is kTLS.
So you are using ECB mode in TLS?
- Eric
Re: [PATCH v2 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of aria cipher
Hi Eric,
Thanks for your review!
2022. 8. 27. 오전 11:46에 Eric Biggers 이(가) 쓴 글:
Quoted 22 lines · Taehee Yoo
Quoted 6 lines
arch/x86/crypto/aria_aesni_avx_glue.c new file +6
-0+static struct skcipher_alg aria_algs[] = { + { + .base.cra_name ="__ecb(aria)" , + .base.cra_driver_name ="__ecb-aria-avx" , + .base.cra_priority =400 , + .base.cra_flags =CRYPTO_ALG_INTERNAL ,arch/x86/crypto/aria_aesni_avx_glue.c line 126
Quoted 10 lines
arch/x86/crypto/aria_aesni_avx_glue.c new file +10
-0+ .base.cra_blocksize = ARIA_BLOCK_SIZE, + .base.cra_ctxsize = sizeof(struct aria_ctx), + .base.cra_module = THIS_MODULE, + .min_keysize = ARIA_MIN_KEY_SIZE, + .max_keysize = ARIA_MAX_KEY_SIZE, + .setkey = aria_avx_set_key, + .encrypt = aria_avx_ecb_encrypt, + .decrypt = aria_avx_ecb_decrypt, + } +};Why do you want ECB mode and nothing else? At
https://lore.kernel.org/r/[email protected]
you claimed that the use case for ARIA support in the kernel is kTLS.
Quoted 2 lines
So you are using ECB mode in TLS?
aria-ktls only uses GCM mode.
So, ECB will not be used by ktls.
My plan is to implement the GCM aria-avx eventually. ECB implementation will be a basic block of aria-avx. I think it can be used by gcm(aria).
So, I will implement gcm mode of aria with this implementation.
If this plan is not good, please let me know. If so, I will change my plan :)
Thanks a lot!
Taehee Yoo
Quoted 1 line
- Eric
Re: [PATCH v2 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of aria cipher
Quoted 32 lines · Taehee Yoo
Hi Eric,
Thanks for your review!2022. 8. 27. 오전 11:46에 Eric Biggers 이(가) 쓴 글:
Quoted 22 lines · Taehee Yoo
Quoted 6 lines
arch/x86/crypto/aria_aesni_avx_glue.c new file +6
-0+static struct skcipher_alg aria_algs[] = { + { + .base.cra_name ="__ecb(aria)" , + .base.cra_driver_name ="__ecb-aria-avx" , + .base.cra_priority =400 , + .base.cra_flags =CRYPTO_ALG_INTERNAL ,arch/x86/crypto/aria_aesni_avx_glue.c line 126
Quoted 10 lines
arch/x86/crypto/aria_aesni_avx_glue.c new file +10
-0+ .base.cra_blocksize = ARIA_BLOCK_SIZE, + .base.cra_ctxsize = sizeof(struct aria_ctx), + .base.cra_module = THIS_MODULE, + .min_keysize = ARIA_MIN_KEY_SIZE, + .max_keysize = ARIA_MAX_KEY_SIZE, + .setkey = aria_avx_set_key, + .encrypt = aria_avx_ecb_encrypt, + .decrypt = aria_avx_ecb_decrypt, + } +};Why do you want ECB mode and nothing else? At
https://lore.kernel.org/r/[email protected]
you claimed that the use case for ARIA support in the kernel is kTLS.Quoted 2 lines
So you are using ECB mode in TLS?
aria-ktls only uses GCM mode.
So, ECB will not be used by ktls.
Quoted 7 lines
My plan is to implement the GCM aria-avx eventually. ECB implementation will be a basic block of aria-avx. I think it can be used by gcm(aria).
So, I will implement gcm mode of aria with this implementation.If this plan is not good, please let me know. If so, I will change my plan :)
GCM uses CTR mode, not ECB mode.
- Eric
Re: [PATCH v2 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of aria cipher
2022. 8. 27. 오후 3:35에 Eric Biggers 이(가) 쓴 글:
Quoted 38 lines · Taehee Yoo
Quoted 37 lines
Hi Eric,
Thanks for your review!2022. 8. 27. 오전 11:46에 Eric Biggers 이(가) 쓴 글:
Quoted 22 lines · Taehee Yoo
Quoted 6 lines
arch/x86/crypto/aria_aesni_avx_glue.c new file +6
-0+static struct skcipher_alg aria_algs[] = { + { + .base.cra_name ="__ecb(aria)" , + .base.cra_driver_name ="__ecb-aria-avx" , + .base.cra_priority =400 , + .base.cra_flags =CRYPTO_ALG_INTERNAL ,arch/x86/crypto/aria_aesni_avx_glue.c line 126
Quoted 10 lines
arch/x86/crypto/aria_aesni_avx_glue.c new file +10
-0+ .base.cra_blocksize = ARIA_BLOCK_SIZE, + .base.cra_ctxsize = sizeof(struct aria_ctx), + .base.cra_module = THIS_MODULE, + .min_keysize = ARIA_MIN_KEY_SIZE, + .max_keysize = ARIA_MAX_KEY_SIZE, + .setkey = aria_avx_set_key, + .encrypt = aria_avx_ecb_encrypt, + .decrypt = aria_avx_ecb_decrypt, + } +};Why do you want ECB mode and nothing else? At
https://lore.kernel.org/r/[email protected]
you claimed that the use case for ARIA support in the kernel is kTLS.Quoted 2 lines
So you are using ECB mode in TLS?
aria-ktls only uses GCM mode.
So, ECB will not be used by ktls.My plan is to implement the GCM aria-avx eventually. ECB implementation will be a basic block of aria-avx. I think it can be used by gcm(aria).
So, I will implement gcm mode of aria with this implementation.
Quoted 5 lines
Quoted 2 lines
If this plan is not good, please let me know. If so, I will change my plan :)
GCM uses CTR mode, not ECB mode.
Thanks for it,
I will implement CTR and includes it in the v3 patch.
Thanks a lot!
Taehee Yoo
Re: [PATCH v2 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of aria cipher
In reply to Taehee YooHello,
Quoted 26 lines · Taehee Yoo
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +26
-0+#define aria_sbox_8way(x0, x1, x2, x3, \ + x4, x5, x6, x7, \ + t0, t1, t2, t3, \ + t4, t5, t6, t7) \ + vpxor t0, t0, t0; \ + vaesenclast t0, x0, x0; \ + vaesenclast t0, x4, x4; \ + vaesenclast t0, x1, x1; \ + vaesenclast t0, x5, x5; \ + vaesdeclast t0, x2, x2; \ + vaesdeclast t0, x6, x6; \ + \ + /* AES inverse shift rows */ \ + vmovdqa .Linv_shift_row, t0; \ + vmovdqa .Lshift_row, t1; \ + vpshufb t0, x0, x0; \ + vpshufb t0, x4, x4; \ + vpshufb t0, x1, x1; \ + vpshufb t0, x5, x5; \ + vpshufb t0, x3, x3; \ + vpshufb t0, x7, x7; \ + vpshufb t1, x2, x2; \ + vpshufb t1, x6, x6; \ + \ + vmovdqa .Linv_lo, t0; \ + vmovdqa .Linv_hi, t1; \
arch/x86/crypto/aria-aesni-avx-asm_64.S line 298
Quoted 10 lines
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +10
-0+ vmovdqa .Ltf_lo_s2, t2; \ + vmovdqa .Ltf_hi_s2, t3; \ + vmovdqa .Ltf_lo_x2, t4; \ + vmovdqa .Ltf_hi_x2, t5; \ + vbroadcastss .L0f0f0f0f, t6; \ + \ + /* extract multiplicative inverse */ \ + filter_8bit(x1, t0, t1, t6, t7); \ + /* affine transformation for S2 */ \ + filter_8bit(x1, t2, t3, t6, t7); \
Here's room for improvement. These two affine transformations could be combined into single filter_8bit...
Quoted 7 lines
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +7
-0+ /* extract multiplicative inverse */ \ + filter_8bit(x5, t0, t1, t6, t7); \ + /* affine transformation for S2 */ \ + filter_8bit(x5, t2, t3, t6, t7); \ + \ + /* affine transformation for X2 */ \ + filter_8bit(x3, t4, t5, t6, t7); \
arch/x86/crypto/aria-aesni-avx-asm_64.S line 315
Quoted 10 lines
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +10
-0+ vpxor t7, t7, t7; \ + vaesenclast t7, x3, x3; \ + /* extract multiplicative inverse */ \ + filter_8bit(x3, t0, t1, t6, t7); \ + /* affine transformation for X2 */ \ + filter_8bit(x7, t4, t5, t6, t7); \ + vpxor t7, t7, t7; \ + vaesenclast t7, x7, x7; \ + /* extract multiplicative inverse */ \ + filter_8bit(x7, t0, t1, t6, t7);
... as well as these two filter_8bit could be replaced with one operation if 'vaesenclast' would be changed to 'vaesdeclast'.
With these optimizations, 'aria_sbox_8way' would look like this:
/////////////////////////////////////////////////////////
#define aria_sbox_8way(x0, x1, x2, x3, \ x4, x5, x6, x7, \ t0, t1, t2, t3, \ t4, t5, t6, t7) \ vpxor t7, t7, t7; \ vmovdqa .Linv_shift_row, t0; \ vmovdqa .Lshift_row, t1; \ vpbroadcastd .L0f0f0f0f, t6; \ vmovdqa .Ltf_lo__inv_aff__and__s2, t2; \ vmovdqa .Ltf_hi__inv_aff__and__s2, t3; \ vmovdqa .Ltf_lo__x2__and__fwd_aff, t4; \ vmovdqa .Ltf_hi__x2__and__fwd_aff, t5; \ \ vaesenclast t7, x0, x0; \ vaesenclast t7, x4, x4; \ vaesenclast t7, x1, x1; \ vaesenclast t7, x5, x5; \ vaesdeclast t7, x2, x2; \ vaesdeclast t7, x6, x6; \ \ /* AES inverse shift rows */ \ vpshufb t0, x0, x0; \ vpshufb t0, x4, x4; \ vpshufb t0, x1, x1; \ vpshufb t0, x5, x5; \ vpshufb t1, x3, x3; \ vpshufb t1, x7, x7; \ vpshufb t1, x2, x2; \ vpshufb t1, x6, x6; \ \ /* affine transformation for S2 */ \ filter_8bit(x1, t2, t3, t6, t0); \ /* affine transformation for S2 */ \ filter_8bit(x5, t2, t3, t6, t0); \ \ /* affine transformation for X2 */ \ filter_8bit(x3, t4, t5, t6, t0); \ /* affine transformation for X2 */ \ filter_8bit(x7, t4, t5, t6, t0); \ vaesdeclast t7, x3, x3; \ vaesdeclast t7, x7, x7;
/* AES inverse affine and S2 combined:
* 1 1 0 0 0 0 0 1 x0 0 * 0 1 0 0 1 0 0 0 x1 0 * 1 1 0 0 1 1 1 1 x2 0 * 0 1 1 0 1 0 0 1 x3 1 * 0 1 0 0 1 1 0 0 * x4 + 0 * 0 1 0 1 1 0 0 0 x5 0 * 0 0 0 0 0 1 0 1 x6 0 * 1 1 1 0 0 1 1 1 x7 1 */ .Ltf_lo__inv_aff__and__s2: .octa 0x92172DA81A9FA520B2370D883ABF8500 .Ltf_hi__inv_aff__and__s2: .octa 0x2B15FFC1AF917B45E6D8320C625CB688
/* X2 and AES forward affine combined:
* 1 0 1 1 0 0 0 1 x0 0 * 0 1 1 1 1 0 1 1 x1 0 * 0 0 0 1 1 0 1 0 x2 1 * 0 1 0 0 0 1 0 0 x3 0 * 0 0 1 1 1 0 1 1 * x4 + 0 * 0 1 0 0 1 0 0 0 x5 0 * 1 1 0 1 0 0 1 1 x6 0 * 0 1 0 0 1 0 1 0 x7 0 */ .Ltf_lo__x2__and__fwd_aff: .octa 0xEFAE0544FCBD1657B8F95213ABEA4100 .Ltf_hi__x2__and__fwd_aff: .octa 0x3F893781E95FE1576CDA64D2BA0CB204
/////////////////////////////////////////////////////////
I tested above quickly in userspace against aria-generic and your original aria-avx implementation and output matches to these references. In quick and dirty benchmark, function execution time was ~30% faster on AMD Zen3 and ~20% faster on Intel tiger-lake.
-Jussi
Re: [PATCH v2 2/3] crypto: aria-avx: add AES-NI/AVX/x86_64 assembler implementation of aria cipher
Hi Jussi,
Thank you so much for this work!
Quoted 138 lines · Jussi Kivilinna
Hello,
Quoted 26 lines · Taehee Yoo
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +26
-0+#define aria_sbox_8way(x0, x1, x2, x3, \ + x4, x5, x6, x7, \ + t0, t1, t2, t3, \ + t4, t5, t6, t7) \ + vpxor t0, t0, t0; \ + vaesenclast t0, x0, x0; \ + vaesenclast t0, x4, x4; \ + vaesenclast t0, x1, x1; \ + vaesenclast t0, x5, x5; \ + vaesdeclast t0, x2, x2; \ + vaesdeclast t0, x6, x6; \ + \ + /* AES inverse shift rows */ \ + vmovdqa .Linv_shift_row, t0; \ + vmovdqa .Lshift_row, t1; \ + vpshufb t0, x0, x0; \ + vpshufb t0, x4, x4; \ + vpshufb t0, x1, x1; \ + vpshufb t0, x5, x5; \ + vpshufb t0, x3, x3; \ + vpshufb t0, x7, x7; \ + vpshufb t1, x2, x2; \ + vpshufb t1, x6, x6; \ + \ + vmovdqa .Linv_lo, t0; \ + vmovdqa .Linv_hi, t1; \arch/x86/crypto/aria-aesni-avx-asm_64.S line 298
Quoted 10 lines
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +10
-0+ vmovdqa .Ltf_lo_s2, t2; \ + vmovdqa .Ltf_hi_s2, t3; \ + vmovdqa .Ltf_lo_x2, t4; \ + vmovdqa .Ltf_hi_x2, t5; \ + vbroadcastss .L0f0f0f0f, t6; \ + \ + /* extract multiplicative inverse */ \ + filter_8bit(x1, t0, t1, t6, t7); \ + /* affine transformation for S2 */ \ + filter_8bit(x1, t2, t3, t6, t7); \Here's room for improvement. These two affine transformations could be combined into single
filter_8bit...Quoted 7 lines
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +7
-0+ /* extract multiplicative inverse */ \ + filter_8bit(x5, t0, t1, t6, t7); \ + /* affine transformation for S2 */ \ + filter_8bit(x5, t2, t3, t6, t7); \ + \ + /* affine transformation for X2 */ \ + filter_8bit(x3, t4, t5, t6, t7); \arch/x86/crypto/aria-aesni-avx-asm_64.S line 315
Quoted 10 lines
arch/x86/crypto/aria-aesni-avx-asm_64.S new file +10
-0+ vpxor t7, t7, t7; \ + vaesenclast t7, x3, x3; \ + /* extract multiplicative inverse */ \ + filter_8bit(x3, t0, t1, t6, t7); \ + /* affine transformation for X2 */ \ + filter_8bit(x7, t4, t5, t6, t7); \ + vpxor t7, t7, t7; \ + vaesenclast t7, x7, x7; \ + /* extract multiplicative inverse */ \ + filter_8bit(x7, t0, t1, t6, t7);... as well as these two
filter_8bitcould be replaced with one operation if 'vaesenclast' would be changed to 'vaesdeclast'.With these optimizations, '
aria_sbox_8way' would look like this://///////////////////////////////////////////////////////
#define aria_sbox_8way(x0, x1, x2, x3, \ x4, x5, x6, x7, \ t0, t1, t2, t3, \ t4, t5, t6, t7) \ vpxor t7, t7, t7; \ vmovdqa .Linv_shift_row, t0; \ vmovdqa .Lshift_row, t1; \ vpbroadcastd .L0f0f0f0f, t6; \ vmovdqa .Ltf_lo__inv_aff__and__s2, t2; \ vmovdqa .Ltf_hi__inv_aff__and__s2, t3; \ vmovdqa .Ltf_lo__x2__and__fwd_aff, t4; \ vmovdqa .Ltf_hi__x2__and__fwd_aff, t5; \ \ vaesenclast t7, x0, x0; \ vaesenclast t7, x4, x4; \ vaesenclast t7, x1, x1; \ vaesenclast t7, x5, x5; \ vaesdeclast t7, x2, x2; \ vaesdeclast t7, x6, x6; \ \ /* AES inverse shift rows */ \ vpshufb t0, x0, x0; \ vpshufb t0, x4, x4; \ vpshufb t0, x1, x1; \ vpshufb t0, x5, x5; \ vpshufb t1, x3, x3; \ vpshufb t1, x7, x7; \ vpshufb t1, x2, x2; \ vpshufb t1, x6, x6; \ \ /* affine transformation for S2 */ \ filter_8bit(x1, t2, t3, t6, t0); \ /* affine transformation for S2 */ \ filter_8bit(x5, t2, t3, t6, t0); \ \ /* affine transformation for X2 */ \ filter_8bit(x3, t4, t5, t6, t0); \ /* affine transformation for X2 */ \ filter_8bit(x7, t4, t5, t6, t0); \ vaesdeclast t7, x3, x3; \ vaesdeclast t7, x7, x7;/* AES inverse affine and S2 combined:
* 1 1 0 0 0 0 0 1 x0 0 * 0 1 0 0 1 0 0 0 x1 0 * 1 1 0 0 1 1 1 1 x2 0 * 0 1 1 0 1 0 0 1 x3 1 * 0 1 0 0 1 1 0 0 * x4 + 0 * 0 1 0 1 1 0 0 0 x5 0 * 0 0 0 0 0 1 0 1 x6 0 * 1 1 1 0 0 1 1 1 x7 1 */ .Ltf_lo__inv_aff__and__s2: .octa 0x92172DA81A9FA520B2370D883ABF8500 .Ltf_hi__inv_aff__and__s2: .octa 0x2B15FFC1AF917B45E6D8320C625CB688/* X2 and AES forward affine combined:
* 1 0 1 1 0 0 0 1 x0 0 * 0 1 1 1 1 0 1 1 x1 0 * 0 0 0 1 1 0 1 0 x2 1 * 0 1 0 0 0 1 0 0 x3 0 * 0 0 1 1 1 0 1 1 * x4 + 0 * 0 1 0 0 1 0 0 0 x5 0 * 1 1 0 1 0 0 1 1 x6 0 * 0 1 0 0 1 0 1 0 x7 0 */ .Ltf_lo__x2__and__fwd_aff: .octa 0xEFAE0544FCBD1657B8F95213ABEA4100 .Ltf_hi__x2__and__fwd_aff: .octa 0x3F893781E95FE1576CDA64D2BA0CB204
Quoted 7 lines
/////////////////////////////////////////////////////////
I tested above quickly in userspace against aria-generic and your original aria-avx implementation and output matches to these references. In quick and dirty benchmark, function execution time was ~30% faster on AMD Zen3 and ~20% faster on Intel tiger-lake.
I tested your implementation.
It works very well and as you mentioned, it improves performance so much! Before:
128bit 4096bytes: 14758 cycles
After:
128bit 4096bytes: 11972 cycles
I will apply your implementation in the v3 patch!
Thank you so much!
Taehee Yoo
PATCH v2 3/3 crypto: tcrypt: add async speed test for aria cipher
In reply to Taehee YooIn order to test for the performance of aria-avx implementation, it needs an async speed test.
So, it adds async speed tests to the tcrypt.
---
v2:
- No changes
crypto/tcrypt.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
crypto/tcrypt.c +13 -0
@@ -2648,6 +2648,13 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb) speed_template_16);break ; +case 519 : +test_acipher_speed ("ecb(aria)" ,ENCRYPT , sec,NULL ,0 , + speed_template_16_24_32); +test_acipher_speed ("ecb(aria)" ,DECRYPT , sec,NULL ,0 , + speed_template_16_24_32); +break ; +case 600 :test_mb_skcipher_speed ("ecb(aes)" ,ENCRYPT , sec,NULL ,0 , speed_template_16_24_32, num_mb);
@@ -2859,6 +2866,12 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)test_mb_skcipher_speed ("ctr(blowfish)" ,DECRYPT , sec,NULL ,0 , speed_template_8_32, num_mb);break ; +case 610 : +test_mb_skcipher_speed ("ecb(aria)" ,ENCRYPT , sec,NULL ,0 , + speed_template_16_32, num_mb); +test_mb_skcipher_speed ("ecb(aria)" ,DECRYPT , sec,NULL ,0 , + speed_template_16_32, num_mb); +break ;case 1000 :test_available ();
Signature · 1 line
2.17.1
Hello,
Quoted 42 lines · Taehee Yoo
The purpose of this patchset is to support the implementation of ARIA-AVX. Many of the ideas in this implementation are from Camellia-avx, especially byte slicing.
Like Camellia, ARIA also uses a 16way strategy.ARIA cipher algorithm is similar to AES.
There are four s-boxes in the ARIA spec and the first and second s-boxes are the same as AES's s-boxes.
Almost functions are based on aria-generic code except for s-box related function.
The aria-avx doesn't implement the key expanding function. it only supportencrypt()anddecrypt().Encryption and Decryption logic is actually the same but it should use separated keys(encryption key and decryption key). En/Decryption steps are like below:
1. Add-Round-Key
2. S-box.
3. Diffusion Layer.There is no special thing in the Add-Round-Key step.
There are some notable things in s-box step.
Like Camellia, it doesn't use a lookup table, instead, it uses aes-ni.To calculate the first s-box, it just uses the aesenclast and then inverts
shift_row. No more process is needed for this job because the first s-box is the same as the AES encryption s-box.To calculate a second s-box(invert of s1), it just uses the aesdeclast and then inverts
shift_row. No more process is needed for this job because the second s-box is the same as the AES decryption s-box.To calculate a third and fourth s-boxes, it uses the aesenclast, then inverts
shift_row, and affine transformation.The aria-generic implementation is based on a 32-bit implementation, not an 8-bit implementation.
The aria-avx Diffusion Layer implementation is based on aria-generic implementation because 8-bit implementation is not fit for parallel implementation but 32-bit is fit for this.
Quoted 8 lines
The first patch in this series is to export functions for aria-avx. The aria-avx uses existing functions in the aria-generic code. The second patch is to implement aria-avx.
The last patch is to add async test for aria.Benchmarks:
The tcrypt is used.
cpu: i3-12100
This CPU also supports Galois Field New Instructions (GFNI) which are even better suited for accelerating ciphers that use same building blocks as AES. For example, I've recently implemented camellia using GFNI for libgcrypt [1].
I quickly hacked GFNI to your implementation and it gives nice extra bit of performance (~55% faster on Intel tiger-lake). Here's GFNI version of 'aria_sbox_8way', that I used:
/////////////////////////////////////////////////////////
#define aria_sbox_8way(x0, x1, x2, x3, \
x4, x5, x6, x7, \
t0, t1, t2, t3, \
t4, t5, t6, t7) \
vpbroadcastq .Ltf_s2_bitmatrix, t0; \
vpbroadcastq .Ltf_inv_bitmatrix, t1; \
vpbroadcastq .Ltf_id_bitmatrix, t2; \
vpbroadcastq .Ltf_aff_bitmatrix, t3; \
vpbroadcastq .Ltf_x2_bitmatrix, t4; \
vgf2p8affineinvqb $(tf_s2_const), t0, x1, x1; \
vgf2p8affineinvqb $(tf_s2_const), t0, x5, x5; \
vgf2p8affineqb $(tf_inv_const), t1, x2, x2; \
vgf2p8affineqb $(tf_inv_const), t1, x6, x6; \
vgf2p8affineinvqb $0, t2, x2, x2; \
vgf2p8affineinvqb $0, t2, x6, x6; \
vgf2p8affineinvqb $(tf_aff_const), t3, x0, x0; \
vgf2p8affineinvqb $(tf_aff_const), t3, x4, x4; \
vgf2p8affineqb $(tf_x2_const), t4, x3, x3; \
vgf2p8affineqb $(tf_x2_const), t4, x7, x7; \
vgf2p8affineinvqb $0, t2, x3, x3; \
vgf2p8affineinvqb $0, t2, x7, x7;#define BV8(a0,a1,a2,a3,a4,a5,a6,a7) \
( (((a0) & 1) << 0) | \
(((a1) & 1) << 1) | \
(((a2) & 1) << 2) | \
(((a3) & 1) << 3) | \
(((a4) & 1) << 4) | \
(((a5) & 1) << 5) | \
(((a6) & 1) << 6) | \
(((a7) & 1) << 7) )#define BM8X8(l0,l1,l2,l3,l4,l5,l6,l7) \
( ((l7) << (0 * 8)) | \
((l6) << (1 * 8)) | \
((l5) << (2 * 8)) | \
((l4) << (3 * 8)) | \
((l3) << (4 * 8)) | \
((l2) << (5 * 8)) | \
((l1) << (6 * 8)) | \
((l0) << (7 * 8)) )/* AES affine: */
#define tf_aff_const BV8(1, 1, 0, 0, 0, 1, 1, 0)
.Ltf_aff_bitmatrix:
.quad BM8X8(BV8(1, 0, 0, 0, 1, 1, 1, 1),
BV8(1, 1, 0, 0, 0, 1, 1, 1),
BV8(1, 1, 1, 0, 0, 0, 1, 1),
BV8(1, 1, 1, 1, 0, 0, 0, 1),
BV8(1, 1, 1, 1, 1, 0, 0, 0),
BV8(0, 1, 1, 1, 1, 1, 0, 0),
BV8(0, 0, 1, 1, 1, 1, 1, 0),
BV8(0, 0, 0, 1, 1, 1, 1, 1))/* AES inverse affine: */
#define tf_inv_const BV8(1, 0, 1, 0, 0, 0, 0, 0)
.Ltf_inv_bitmatrix:
.quad BM8X8(BV8(0, 0, 1, 0, 0, 1, 0, 1),
BV8(1, 0, 0, 1, 0, 0, 1, 0),
BV8(0, 1, 0, 0, 1, 0, 0, 1),
BV8(1, 0, 1, 0, 0, 1, 0, 0),
BV8(0, 1, 0, 1, 0, 0, 1, 0),
BV8(0, 0, 1, 0, 1, 0, 0, 1),
BV8(1, 0, 0, 1, 0, 1, 0, 0),
BV8(0, 1, 0, 0, 1, 0, 1, 0))/* S2: */
#define tf_s2_const BV8(0, 1, 0, 0, 0, 1, 1, 1)
.Ltf_s2_bitmatrix:
.quad BM8X8(BV8(0, 1, 0, 1, 0, 1, 1, 1),
BV8(0, 0, 1, 1, 1, 1, 1, 1),
BV8(1, 1, 1, 0, 1, 1, 0, 1),
BV8(1, 1, 0, 0, 0, 0, 1, 1),
BV8(0, 1, 0, 0, 0, 0, 1, 1),
BV8(1, 1, 0, 0, 1, 1, 1, 0),
BV8(0, 1, 1, 0, 0, 0, 1, 1),
BV8(1, 1, 1, 1, 0, 1, 1, 0))/* X2: */
#define tf_x2_const BV8(0, 0, 1, 1, 0, 1, 0, 0)
.Ltf_x2_bitmatrix:
.quad BM8X8(BV8(0, 0, 0, 1, 1, 0, 0, 0),
BV8(0, 0, 1, 0, 0, 1, 1, 0),
BV8(0, 0, 0, 0, 1, 0, 1, 0),
BV8(1, 1, 1, 0, 0, 0, 1, 1),
BV8(1, 1, 1, 0, 1, 1, 0, 0),
BV8(0, 1, 1, 0, 1, 0, 1, 1),
BV8(1, 0, 1, 1, 1, 1, 0, 1),
BV8(1, 0, 0, 1, 0, 0, 1, 1))/* Identity matrix: */
.Ltf_id_bitmatrix:
.quad BM8X8(BV8(1, 0, 0, 0, 0, 0, 0, 0),
BV8(0, 1, 0, 0, 0, 0, 0, 0),
BV8(0, 0, 1, 0, 0, 0, 0, 0),
BV8(0, 0, 0, 1, 0, 0, 0, 0),
BV8(0, 0, 0, 0, 1, 0, 0, 0),
BV8(0, 0, 0, 0, 0, 1, 0, 0),
BV8(0, 0, 0, 0, 0, 0, 1, 0),
BV8(0, 0, 0, 0, 0, 0, 0, 1))/////////////////////////////////////////////////////////
GFNI also allows easy use of 256-bit vector registers so there is way to get additional 2x speed increase (but requires doubling number of parallel processed blocks).
-Jussi
Hi Jussi,
Thank you so much for this great work!
Quoted 165 lines · Jussi Kivilinna
Hello,
Quoted 43 lines · Taehee Yoo
The purpose of this patchset is to support the implementation of ARIA-AVX.
Many of the ideas in this implementation are from Camellia-avx, especially byte slicing.
Like Camellia, ARIA also uses a 16way strategy.ARIA cipher algorithm is similar to AES.
There are four s-boxes in the ARIA spec and the first and second s-boxes are the same as AES's s-boxes.
Almost functions are based on aria-generic code except for s-box related function.
The aria-avx doesn't implement the key expanding function. it only supportencrypt()anddecrypt().Encryption and Decryption logic is actually the same but it should use separated keys(encryption key and decryption key). En/Decryption steps are like below:
1. Add-Round-Key
2. S-box.
3. Diffusion Layer.There is no special thing in the Add-Round-Key step.
There are some notable things in s-box step.
Like Camellia, it doesn't use a lookup table, instead, it uses aes-ni.To calculate the first s-box, it just uses the aesenclast and then inverts
shift_row. No more process is needed for this job because the first s-box is the same as the AES encryption s-box.To calculate a second s-box(invert of s1), it just uses the aesdeclast and then inverts
shift_row. No more process is needed for this job because the second s-box is the same as the AES decryption s-box.To calculate a third and fourth s-boxes, it uses the aesenclast, then inverts
shift_row, and affine transformation.The aria-generic implementation is based on a 32-bit implementation, not an 8-bit implementation.
The aria-avx Diffusion Layer implementation is based on aria-generic implementation because 8-bit implementation is not fit for parallel implementation but 32-bit is fit for this.Quoted 8 lines
The first patch in this series is to export functions for aria-avx. The aria-avx uses existing functions in the aria-generic code. The second patch is to implement aria-avx.
The last patch is to add async test for aria.Benchmarks:
The tcrypt is used.
cpu: i3-12100This CPU also supports Galois Field New Instructions (GFNI) which are even better suited for accelerating ciphers that use same building blocks as AES. For example, I've recently implemented camellia using GFNI for libgcrypt [1].
I quickly hacked GFNI to your implementation and it gives nice extra bit of performance (~55% faster on Intel tiger-lake). Here's GFNI version of '
aria_sbox_8way', that I used://///////////////////////////////////////////////////////
#define aria_sbox_8way(x0, x1, x2, x3, \ x4, x5, x6, x7, \ t0, t1, t2, t3, \ t4, t5, t6, t7) \ vpbroadcastq .Ltf_s2_bitmatrix, t0; \ vpbroadcastq .Ltf_inv_bitmatrix, t1; \ vpbroadcastq .Ltf_id_bitmatrix, t2; \ vpbroadcastq .Ltf_aff_bitmatrix, t3; \ vpbroadcastq .Ltf_x2_bitmatrix, t4; \ vgf2p8affineinvqb $(tf_s2_const), t0, x1, x1; \ vgf2p8affineinvqb $(tf_s2_const), t0, x5, x5; \ vgf2p8affineqb $(tf_inv_const), t1, x2, x2; \ vgf2p8affineqb $(tf_inv_const), t1, x6, x6; \ vgf2p8affineinvqb $0, t2, x2, x2; \ vgf2p8affineinvqb $0, t2, x6, x6; \ vgf2p8affineinvqb $(tf_aff_const), t3, x0, x0; \ vgf2p8affineinvqb $(tf_aff_const), t3, x4, x4; \ vgf2p8affineqb $(tf_x2_const), t4, x3, x3; \ vgf2p8affineqb $(tf_x2_const), t4, x7, x7; \ vgf2p8affineinvqb $0, t2, x3, x3; \ vgf2p8affineinvqb $0, t2, x7, x7;#define BV8(a0,a1,a2,a3,a4,a5,a6,a7) \ ( (((a0) & 1) << 0) | \ (((a1) & 1) << 1) | \ (((a2) & 1) << 2) | \ (((a3) & 1) << 3) | \ (((a4) & 1) << 4) | \ (((a5) & 1) << 5) | \ (((a6) & 1) << 6) | \ (((a7) & 1) << 7) )#define BM8X8(l0,l1,l2,l3,l4,l5,l6,l7) \ ( ((l7) << (0 * 8)) | \ ((l6) << (1 * 8)) | \ ((l5) << (2 * 8)) | \ ((l4) << (3 * 8)) | \ ((l3) << (4 * 8)) | \ ((l2) << (5 * 8)) | \ ((l1) << (6 * 8)) | \ ((l0) << (7 * 8)) )/* AES affine: */
#definetf_aff_constBV8(1, 1, 0, 0, 0, 1, 1, 0)
.Ltf_aff_bitmatrix:.quad BM8X8(BV8(1, 0, 0, 0, 1, 1, 1, 1), BV8(1, 1, 0, 0, 0, 1, 1, 1), BV8(1, 1, 1, 0, 0, 0, 1, 1), BV8(1, 1, 1, 1, 0, 0, 0, 1), BV8(1, 1, 1, 1, 1, 0, 0, 0), BV8(0, 1, 1, 1, 1, 1, 0, 0), BV8(0, 0, 1, 1, 1, 1, 1, 0), BV8(0, 0, 0, 1, 1, 1, 1, 1))/* AES inverse affine: */
#definetf_inv_constBV8(1, 0, 1, 0, 0, 0, 0, 0)
.Ltf_inv_bitmatrix:.quad BM8X8(BV8(0, 0, 1, 0, 0, 1, 0, 1), BV8(1, 0, 0, 1, 0, 0, 1, 0), BV8(0, 1, 0, 0, 1, 0, 0, 1), BV8(1, 0, 1, 0, 0, 1, 0, 0), BV8(0, 1, 0, 1, 0, 0, 1, 0), BV8(0, 0, 1, 0, 1, 0, 0, 1), BV8(1, 0, 0, 1, 0, 1, 0, 0), BV8(0, 1, 0, 0, 1, 0, 1, 0))/* S2: */
#definetf_s2_constBV8(0, 1, 0, 0, 0, 1, 1, 1)
.Ltf_s2_bitmatrix:.quad BM8X8(BV8(0, 1, 0, 1, 0, 1, 1, 1), BV8(0, 0, 1, 1, 1, 1, 1, 1), BV8(1, 1, 1, 0, 1, 1, 0, 1), BV8(1, 1, 0, 0, 0, 0, 1, 1), BV8(0, 1, 0, 0, 0, 0, 1, 1), BV8(1, 1, 0, 0, 1, 1, 1, 0), BV8(0, 1, 1, 0, 0, 0, 1, 1), BV8(1, 1, 1, 1, 0, 1, 1, 0))/* X2: */
#definetf_x2_constBV8(0, 0, 1, 1, 0, 1, 0, 0)
.Ltf_x2_bitmatrix:.quad BM8X8(BV8(0, 0, 0, 1, 1, 0, 0, 0), BV8(0, 0, 1, 0, 0, 1, 1, 0), BV8(0, 0, 0, 0, 1, 0, 1, 0), BV8(1, 1, 1, 0, 0, 0, 1, 1), BV8(1, 1, 1, 0, 1, 1, 0, 0), BV8(0, 1, 1, 0, 1, 0, 1, 1), BV8(1, 0, 1, 1, 1, 1, 0, 1), BV8(1, 0, 0, 1, 0, 0, 1, 1))/* Identity matrix: */
.Ltf_id_bitmatrix:.quad BM8X8(BV8(1, 0, 0, 0, 0, 0, 0, 0), BV8(0, 1, 0, 0, 0, 0, 0, 0), BV8(0, 0, 1, 0, 0, 0, 0, 0), BV8(0, 0, 0, 1, 0, 0, 0, 0), BV8(0, 0, 0, 0, 1, 0, 0, 0), BV8(0, 0, 0, 0, 0, 1, 0, 0), BV8(0, 0, 0, 0, 0, 0, 1, 0), BV8(0, 0, 0, 0, 0, 0, 0, 1))
Quoted 6 lines
/////////////////////////////////////////////////////////
GFNI also allows easy use of 256-bit vector registers so there is way to get additional 2x speed increase (but requires doubling number of parallel processed blocks).
I checked that my i3-12100 supports GFNI. Then, benchmarked this implementation.
It works very well and it shows amazing performance! Before:
128bit, 4096 bytes: 14758 cycles
After:
128bit, 4096 bytes: 9404 cycles
I think I must add this implementation to the v3 patch.
Like your code[1], I also will add a condition on whether CPU supports GFNI or not. This will be very helpful to the AVX2 and AVX-512 implementation too. I think AVX2 implementation will use 256-bit vector register.
So, as you mentioned, its potential performance increment is also great.
Thank you so much again for your great work!
Taehee Yoo