crypto: aria: implement aria-avx2 and aria-avx512

PATCH Superseded Taehee Yoo 3 participants 7 messages Nov 5, 2022 – Nov 6, 2022

Subsystems (2)

x86 Crypto API

lore

Share & tools

lore

Taehee Yoo

author

2022-11-05 08:20 UTC #0 lore raw

v2 cover crypto: aria: implement aria-avx2 and aria-avx512

This patchset is to implement aria-avx2 and aria-avx512. There are some differences between aria-avx, aria-avx2, and aria-avx512, but they are not core logic(s-box, diffusion layer).

ARIA-AVX2
It supports 32way parallel processing using 256bit registers. Like ARIA-AVX, it supports both AES-NI based s-box layer algorithm and GFNI based s-box layer algorithm.
These algorithms are the same as ARIA-AVX except that AES-NI doesn't support 256bit registers, so it is used twice.

ARIA-AVX512
It supports 64way parallel processing using 512bit registers. It supports only GFNI based s-box layer algorithm.

Benchmarks with i3-12100
commands: modprobe tcrypt mode=610 num_mb=8192

ARIA-AVX512(128bit and 256bit)
testing speed of multibuffer ecb(aria) (ecb-aria-avx512) encryption
tcrypt: 1 operation in 1504 cycles (1024 bytes)
tcrypt: 1 operation in 4595 cycles (4096 bytes)
tcrypt: 1 operation in 1763 cycles (1024 bytes)
tcrypt: 1 operation in 5540 cycles (4096 bytes)
testing speed of multibuffer ecb(aria) (ecb-aria-avx512) decryption
tcrypt: 1 operation in 1502 cycles (1024 bytes)
tcrypt: 1 operation in 4615 cycles (4096 bytes)
tcrypt: 1 operation in 1759 cycles (1024 bytes)
tcrypt: 1 operation in 5554 cycles (4096 bytes)

ARIA-AVX2 with GFNI(128bit and 256bit)
testing speed of multibuffer ecb(aria) (ecb-aria-avx2) encryption
tcrypt: 1 operation in 2003 cycles (1024 bytes)
tcrypt: 1 operation in 5867 cycles (4096 bytes)
tcrypt: 1 operation in 2358 cycles (1024 bytes)
tcrypt: 1 operation in 7295 cycles (4096 bytes)
testing speed of multibuffer ecb(aria) (ecb-aria-avx2) decryption
tcrypt: 1 operation in 2004 cycles (1024 bytes)
tcrypt: 1 operation in 5956 cycles (4096 bytes)
tcrypt: 1 operation in 2409 cycles (1024 bytes)
tcrypt: 1 operation in 7564 cycles (4096 bytes)

ARIA-AVX with GFNI(128bit and 256bit)
testing speed of multibuffer ecb(aria) (ecb-aria-avx) encryption
tcrypt: 1 operation in 2761 cycles (1024 bytes)
tcrypt: 1 operation in 9390 cycles (4096 bytes)
tcrypt: 1 operation in 3401 cycles (1024 bytes)
tcrypt: 1 operation in 11876 cycles (4096 bytes)
testing speed of multibuffer ecb(aria) (ecb-aria-avx) decryption
tcrypt: 1 operation in 2735 cycles (1024 bytes)
tcrypt: 1 operation in 9424 cycles (4096 bytes)
tcrypt: 1 operation in 3369 cycles (1024 bytes)
tcrypt: 1 operation in 11954 cycles (4096 bytes)

v2:
- Add new "add keystream array into struct aria_ctx" patch.
- Use keystream array in the aria_ctx instead of stack memory

Taehee Yoo (3):
crypto: aria: add keystream array into struct aria_ctx
crypto: aria: implement aria-avx2
crypto: aria: implement aria-avx512

 arch/x86/crypto/Kconfig                   |   38 +
 arch/x86/crypto/Makefile                  |    6 +
 arch/x86/crypto/aria-aesni-avx2-asm_64.S  | 1436 +++++++++++++++++++++
 arch/x86/crypto/aria-avx.h                |   41 +-
 arch/x86/crypto/aria-gfni-avx512-asm_64.S | 1023 +++++++++++++++
 arch/x86/crypto/aria_aesni_avx2_glue.c    |  236 ++++
 arch/x86/crypto/aria_aesni_avx_glue.c     |   30 +-
 arch/x86/crypto/aria_gfni_avx512_glue.c   |  232 ++++
 include/crypto/aria.h                     |   24 +
 9 files changed, 3051 insertions(+), 15 deletions(-)
 create mode 100644 arch/x86/crypto/aria-aesni-avx2-asm_64.S
 create mode 100644 arch/x86/crypto/aria-gfni-avx512-asm_64.S
 create mode 100644 arch/x86/crypto/aria_aesni_avx2_glue.c
 create mode 100644 arch/x86/crypto/aria_gfni_avx512_glue.c

Signature · 1 line

2.17.1

Taehee Yoo

author

2022-11-05 08:20 UTC #1 lore raw

PATCH v2 1/3 crypto: aria: add keystream array into struct aria_ctx

avx accelerated aria module used local keystream array. But, keystream array size is too big.
So, it puts the keystream array into struct aria_ctx.

Signed-off-by Taehee Yoo <[email protected]>

---

v2:
- patch introduced

 arch/x86/crypto/aria-avx.h            |  3 ---
 arch/x86/crypto/aria_aesni_avx_glue.c | 24 +++++++++++-------------
 include/crypto/aria.h                 | 11 +++++++++++
 3 files changed, 22 insertions(+), 16 deletions(-)

arch/x86/crypto/aria-avx.h +0 -3

index 01e9a01dc157..afd37af95e58 100644
@@ -4,9 +4,6 @@
 
 #include <linux/types.h>
 
-#define ARIA_AESNI_PARALLEL_BLOCKS 16
-#define ARIA_AESNI_PARALLEL_BLOCK_SIZE  (ARIA_BLOCK_SIZE * 16)
-
 struct aria_avx_ops {
 	void (*aria_encrypt_16way)(const void *ctx, u8 *dst, const u8 *src);
 	void (*aria_decrypt_16way)(const void *ctx, u8 *dst, const u8 *src);

arch/x86/crypto/aria_aesni_avx_glue.c +11 ~~-13~~

index c561ea4fefa5..b122482d0c9d 100644
@@ -86,10 +86,9 @@ static int aria_avx_ctr_encrypt(struct skcipher_request *req)
 		u8 *dst = walk.dst.virt.addr;
 
 		while (nbytes >= ARIA_AESNI_PARALLEL_BLOCK_SIZE) {
-			u8 keystream[ARIA_AESNI_PARALLEL_BLOCK_SIZE];
-
 			kernel_fpu_begin();
-			aria_ops.aria_ctr_crypt_16way(ctx, dst, src, keystream,
+			aria_ops.aria_ctr_crypt_16way(ctx, dst, src,
+						      &ctx->keystream[0],
 						      walk.iv);
 			kernel_fpu_end();
 			dst += ARIA_AESNI_PARALLEL_BLOCK_SIZE;

@@ -98,28 +97,27 @@ static int aria_avx_ctr_encrypt(struct skcipher_request *req)
 		}
 
 		while (nbytes >= ARIA_BLOCK_SIZE) {
-			u8 keystream[ARIA_BLOCK_SIZE];
-
-			memcpy(keystream, walk.iv, ARIA_BLOCK_SIZE);
+			memcpy(&ctx->keystream[0], walk.iv, ARIA_BLOCK_SIZE);
 			crypto_inc(walk.iv, ARIA_BLOCK_SIZE);
 
-			aria_encrypt(ctx, keystream, keystream);
+			aria_encrypt(ctx, &ctx->keystream[0],
+				     &ctx->keystream[0]);
 
-			crypto_xor_cpy(dst, src, keystream, ARIA_BLOCK_SIZE);
+			crypto_xor_cpy(dst, src, &ctx->keystream[0],
+				       ARIA_BLOCK_SIZE);
 			dst += ARIA_BLOCK_SIZE;
 			src += ARIA_BLOCK_SIZE;
 			nbytes -= ARIA_BLOCK_SIZE;
 		}
 
 		if (walk.nbytes == walk.total && nbytes > 0) {
-			u8 keystream[ARIA_BLOCK_SIZE];
-
-			memcpy(keystream, walk.iv, ARIA_BLOCK_SIZE);
+			memcpy(&ctx->keystream[0], walk.iv, ARIA_BLOCK_SIZE);
 			crypto_inc(walk.iv, ARIA_BLOCK_SIZE);
 
-			aria_encrypt(ctx, keystream, keystream);
+			aria_encrypt(ctx, &ctx->keystream[0],
+				     &ctx->keystream[0]);
 
-			crypto_xor_cpy(dst, src, keystream, nbytes);
+			crypto_xor_cpy(dst, src, &ctx->keystream[0], nbytes);
 			dst += nbytes;
 			src += nbytes;
 			nbytes = 0;

include/crypto/aria.h +11 -0

index 254da46cc385..f5c7a87378cd 100644
@@ -31,11 +31,22 @@
 #define ARIA_MAX_RD_KEYS	17
 #define ARIA_RD_KEY_WORDS	(ARIA_BLOCK_SIZE / sizeof(u32))
 
+#define ARIA_AESNI_PARALLEL_BLOCKS 16
+#define ARIA_AESNI_PARALLEL_BLOCK_SIZE  (ARIA_BLOCK_SIZE * 16)
+#if defined(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) ||	\
+	defined(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64_MODULE)
+#define ARIA_KEYSTREAM_SIZE ARIA_AESNI_PARALLEL_BLOCK_SIZE
+#endif
+
 struct aria_ctx {
 	u32 enc_key[ARIA_MAX_RD_KEYS][ARIA_RD_KEY_WORDS];
 	u32 dec_key[ARIA_MAX_RD_KEYS][ARIA_RD_KEY_WORDS];
 	int rounds;
 	int key_length;
+#if defined(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) ||	\
+	defined(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64_MODULE)
+	u8 keystream[ARIA_KEYSTREAM_SIZE];
+#endif
 };
 
 static const u32 s1[256] = {

Signature · 1 line

2.17.1

Taehee Yoo

author

2022-11-05 08:20 UTC #2 lore raw

PATCH v2 2/3 crypto: aria: implement aria-avx2

In reply to Taehee Yoo

aria-avx2 implementation uses AVX2, AES-NI, and GFNI.
It supports 32way parallel processing.
So, byteslicing code is changed to support 32way parallel. And it exports some aria-avx functions such as encrypt() and decrypt().

There are two main logics, s-box layer and diffusion layer. These codes are the same as aria-avx implementation.
But some instruction are exchanged because they don't support 256bit registers.
Also, AES-NI doesn't support 256bit register.
So, aesenclast and aesdeclast are used twice like below:

	vextracti128 $1, ymm0, xmm6;
	vaesenclast xmm7, xmm0, xmm0;
	vaesenclast xmm7, xmm6, xmm6;
	vinserti128 $1, xmm6, ymm0, ymm0;

Benchmark with modprobe tcrypt mode=610 num_mb=8192, i3-12100:

Signed-off-by Taehee Yoo <[email protected]>

---

v2:
- Use keystream array in the aria_ctx instead of stack memory

 arch/x86/crypto/Kconfig                  |   19 +
 arch/x86/crypto/Makefile                 |    3 +
 arch/x86/crypto/aria-aesni-avx2-asm_64.S | 1436 ++++++++++++++++++++++
 arch/x86/crypto/aria-avx.h               |   35 +
 arch/x86/crypto/aria_aesni_avx2_glue.c   |  236 ++++
 arch/x86/crypto/aria_aesni_avx_glue.c    |    6 +
 include/crypto/aria.h                    |    9 +-
 7 files changed, 1743 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/crypto/aria-aesni-avx2-asm_64.S
 create mode 100644 arch/x86/crypto/aria_aesni_avx2_glue.c

arch/x86/crypto/Kconfig +19 -0

index 71c4c473d34b..3837ba8b78c5 100644
@@ -304,6 +304,25 @@ config CRYPTO_ARIA_AESNI_AVX_X86_64
 
 	  Processes 16 blocks in parallel.
 
+config CRYPTO_ARIA_AESNI_AVX2_X86_64
+	tristate "Ciphers: ARIA with modes: ECB, CTR (AES-NI/AVX2/GFNI)"
+	depends on X86 && 64BIT
+	select CRYPTO_SKCIPHER
+	select CRYPTO_SIMD
+	select CRYPTO_ALGAPI
+	select CRYPTO_ARIA
+	select CRYPTO_ARIA_AESNI_AVX_X86_64
+	help
+	  Length-preserving cipher: ARIA cipher algorithms
+	  (RFC 5794) with ECB and CTR modes
+
+	  Architecture: x86_64 using:
+	  - AES-NI (AES New Instructions)
+	  - AVX2 (Advanced Vector Extensions)
+	  - GFNI (Galois Field New Instructions)
+
+	  Processes 32 blocks in parallel.
+
 config CRYPTO_CHACHA20_X86_64
 	tristate "Ciphers: ChaCha20, XChaCha20, XChaCha12 (SSSE3/AVX2/AVX-512VL)"
 	depends on X86 && 64BIT

arch/x86/crypto/Makefile +3 -0

index 3b1d701a4f6c..5a349c7a8127 100644
@@ -103,6 +103,9 @@ sm4-aesni-avx2-x86_64-y := sm4-aesni-avx2-asm_64.o sm4_aesni_avx2_glue.o
 obj-$(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) += aria-aesni-avx-x86_64.o
 aria-aesni-avx-x86_64-y := aria-aesni-avx-asm_64.o aria_aesni_avx_glue.o
 
+obj-$(CONFIG_CRYPTO_ARIA_AESNI_AVX2_X86_64) += aria-aesni-avx2-x86_64.o
+aria-aesni-avx2-x86_64-y := aria-aesni-avx2-asm_64.o aria_aesni_avx2_glue.o
+
 quiet_cmd_perlasm = PERLASM $@
       cmd_perlasm = $(PERL) $< > $@
 $(obj)/%.S: $(src)/%.pl FORCE

arch/x86/crypto/aria-aesni-avx2-asm_64.S new file +1436 -0

1,436 lines not on this page · raw message

arch/x86/crypto/aria-avx.h +35 -0

index afd37af95e58..5dd7f18255eb 100644
@@ -4,10 +4,45 @@
 
 #include <linux/types.h>
 
+asmlinkage void aria_aesni_avx_encrypt_16way(const void *ctx, u8 *dst,
+					     const u8 *src);
+asmlinkage void aria_aesni_avx_decrypt_16way(const void *ctx, u8 *dst,
+					     const u8 *src);
+asmlinkage void aria_aesni_avx_ctr_crypt_16way(const void *ctx, u8 *dst,
+					       const u8 *src,
+					       u8 *keystream, u8 *iv);
+asmlinkage void aria_aesni_avx_gfni_encrypt_16way(const void *ctx, u8 *dst,
+						  const u8 *src);
+asmlinkage void aria_aesni_avx_gfni_decrypt_16way(const void *ctx, u8 *dst,
+						  const u8 *src);
+asmlinkage void aria_aesni_avx_gfni_ctr_crypt_16way(const void *ctx, u8 *dst,
+						    const u8 *src,
+						    u8 *keystream, u8 *iv);
+
+asmlinkage void aria_aesni_avx2_encrypt_32way(const void *ctx, u8 *dst,
+					      const u8 *src);
+asmlinkage void aria_aesni_avx2_decrypt_32way(const void *ctx, u8 *dst,
+					      const u8 *src);
+asmlinkage void aria_aesni_avx2_ctr_crypt_32way(const void *ctx, u8 *dst,
+						const u8 *src,
+						u8 *keystream, u8 *iv);
+asmlinkage void aria_aesni_avx2_gfni_encrypt_32way(const void *ctx, u8 *dst,
+						   const u8 *src);
+asmlinkage void aria_aesni_avx2_gfni_decrypt_32way(const void *ctx, u8 *dst,
+						   const u8 *src);
+asmlinkage void aria_aesni_avx2_gfni_ctr_crypt_32way(const void *ctx, u8 *dst,
+						     const u8 *src,
+						     u8 *keystream, u8 *iv);
+
 struct aria_avx_ops {
 	void (*aria_encrypt_16way)(const void *ctx, u8 *dst, const u8 *src);
 	void (*aria_decrypt_16way)(const void *ctx, u8 *dst, const u8 *src);
 	void (*aria_ctr_crypt_16way)(const void *ctx, u8 *dst, const u8 *src,
 				     u8 *keystream, u8 *iv);
+	void (*aria_encrypt_32way)(const void *ctx, u8 *dst, const u8 *src);
+	void (*aria_decrypt_32way)(const void *ctx, u8 *dst, const u8 *src);
+	void (*aria_ctr_crypt_32way)(const void *ctx, u8 *dst, const u8 *src,
+				     u8 *keystream, u8 *iv);
+
 };
 #endif

arch/x86/crypto/aria_aesni_avx2_glue.c new file +236 -0

236 lines not on this page · raw message

arch/x86/crypto/aria_aesni_avx_glue.c +6 -0

index b122482d0c9d..9999b7d2deac 100644
@@ -18,18 +18,24 @@
 
 asmlinkage void aria_aesni_avx_encrypt_16way(const void *ctx, u8 *dst,
 					     const u8 *src);
+EXPORT_SYMBOL_GPL(aria_aesni_avx_encrypt_16way);
 asmlinkage void aria_aesni_avx_decrypt_16way(const void *ctx, u8 *dst,
 					     const u8 *src);
+EXPORT_SYMBOL_GPL(aria_aesni_avx_decrypt_16way);
 asmlinkage void aria_aesni_avx_ctr_crypt_16way(const void *ctx, u8 *dst,
 					       const u8 *src,
 					       u8 *keystream, u8 *iv);
+EXPORT_SYMBOL_GPL(aria_aesni_avx_ctr_crypt_16way);
 asmlinkage void aria_aesni_avx_gfni_encrypt_16way(const void *ctx, u8 *dst,
 						  const u8 *src);
+EXPORT_SYMBOL_GPL(aria_aesni_avx_gfni_encrypt_16way);
 asmlinkage void aria_aesni_avx_gfni_decrypt_16way(const void *ctx, u8 *dst,
 						  const u8 *src);
+EXPORT_SYMBOL_GPL(aria_aesni_avx_gfni_decrypt_16way);
 asmlinkage void aria_aesni_avx_gfni_ctr_crypt_16way(const void *ctx, u8 *dst,
 						    const u8 *src,
 						    u8 *keystream, u8 *iv);
+EXPORT_SYMBOL_GPL(aria_aesni_avx_gfni_ctr_crypt_16way);
 
 static struct aria_avx_ops aria_ops;

include/crypto/aria.h +8 -1

index f5c7a87378cd..bc5dc71d6a98 100644
@@ -33,7 +33,14 @@
 
 #define ARIA_AESNI_PARALLEL_BLOCKS 16
 #define ARIA_AESNI_PARALLEL_BLOCK_SIZE  (ARIA_BLOCK_SIZE * 16)
-#if defined(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) ||	\
+
+#define ARIA_AESNI_AVX2_PARALLEL_BLOCKS 32
+#define ARIA_AESNI_AVX2_PARALLEL_BLOCK_SIZE  (ARIA_BLOCK_SIZE * 32)
+
+#if defined(CONFIG_CRYPTO_ARIA_AESNI_AVX2_X86_64) ||   \
+	defined(CONFIG_CRYPTO_ARIA_AESNI_AVX2_X86_64_MODULE)
+#define ARIA_KEYSTREAM_SIZE ARIA_AESNI_AVX2_PARALLEL_BLOCK_SIZE
+#elif defined(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) ||  \
 	defined(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64_MODULE)
 #define ARIA_KEYSTREAM_SIZE ARIA_AESNI_PARALLEL_BLOCK_SIZE
 #endif

Signature · 1 line

2.17.1

Taehee Yoo

author

2022-11-05 08:21 UTC #3 lore raw

PATCH v2 3/3 crypto: aria: implement aria-avx512

In reply to Taehee Yoo

aria-avx512 implementation uses AVX512 and GFNI.
It supports 64way parallel processing.
So, byteslicing code is changed to support 64way parallel. And it exports some aria-avx2 functions such as encrypt() and decrypt().

AVX and AVX2 have 16 registers.
They should use memory to store/load state because of lack of registers. But AVX512 supports 32 registers.
So, it doesn't require store/load in the s-box layer.
It means that it can reduce overhead of store/load in the s-box layer. Also code become much simpler.

Benchmark with modprobe tcrypt mode=610 num_mb=8192, i3-12100:

Signed-off-by Taehee Yoo <[email protected]>

---

v2:
- Use keystream array in the aria_ctx instead of stack memory

 arch/x86/crypto/Kconfig                   |   19 +
 arch/x86/crypto/Makefile                  |    3 +
 arch/x86/crypto/aria-avx.h                |    5 +
 arch/x86/crypto/aria-gfni-avx512-asm_64.S | 1023 +++++++++++++++++++++
 arch/x86/crypto/aria_gfni_avx512_glue.c   |  232 +++++
 include/crypto/aria.h                     |    8 +-
 6 files changed, 1289 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/crypto/aria-gfni-avx512-asm_64.S
 create mode 100644 arch/x86/crypto/aria_gfni_avx512_glue.c

arch/x86/crypto/Kconfig +19 -0

index 3837ba8b78c5..688e848f740d 100644
@@ -323,6 +323,25 @@ config CRYPTO_ARIA_AESNI_AVX2_X86_64
 
 	  Processes 32 blocks in parallel.
 
+config CRYPTO_ARIA_GFNI_AVX512_X86_64
+	tristate "Ciphers: ARIA with modes: ECB, CTR (AVX512/GFNI)"
+	depends on X86 && 64BIT
+	select CRYPTO_SKCIPHER
+	select CRYPTO_SIMD
+	select CRYPTO_ALGAPI
+	select CRYPTO_ARIA
+	select CRYPTO_ARIA_AESNI_AVX_X86_64
+	select CRYPTO_ARIA_AESNI_AVX2_X86_64
+	help
+	  Length-preserving cipher: ARIA cipher algorithms
+	  (RFC 5794) with ECB and CTR modes
+
+	  Architecture: x86_64 using:
+	  - AVX512 (Advanced Vector Extensions)
+	  - GFNI (Galois Field New Instructions)
+
+	  Processes 64 blocks in parallel.
+
 config CRYPTO_CHACHA20_X86_64
 	tristate "Ciphers: ChaCha20, XChaCha20, XChaCha12 (SSSE3/AVX2/AVX-512VL)"
 	depends on X86 && 64BIT

arch/x86/crypto/Makefile +3 -0

index 5a349c7a8127..1844925f1f4a 100644
@@ -106,6 +106,9 @@ aria-aesni-avx-x86_64-y := aria-aesni-avx-asm_64.o aria_aesni_avx_glue.o
 obj-$(CONFIG_CRYPTO_ARIA_AESNI_AVX2_X86_64) += aria-aesni-avx2-x86_64.o
 aria-aesni-avx2-x86_64-y := aria-aesni-avx2-asm_64.o aria_aesni_avx2_glue.o
 
+obj-$(CONFIG_CRYPTO_ARIA_GFNI_AVX512_X86_64) += aria-gfni-avx512-x86_64.o
+aria-gfni-avx512-x86_64-y := aria-gfni-avx512-asm_64.o aria_gfni_avx512_glue.o
+
 quiet_cmd_perlasm = PERLASM $@
       cmd_perlasm = $(PERL) $< > $@
 $(obj)/%.S: $(src)/%.pl FORCE

arch/x86/crypto/aria-avx.h +5 -0

index 5dd7f18255eb..b62c255ce850 100644
@@ -43,6 +43,11 @@ struct aria_avx_ops {
 	void (*aria_decrypt_32way)(const void *ctx, u8 *dst, const u8 *src);
 	void (*aria_ctr_crypt_32way)(const void *ctx, u8 *dst, const u8 *src,
 				     u8 *keystream, u8 *iv);
+	void (*aria_encrypt_64way)(const void *ctx, u8 *dst, const u8 *src);
+	void (*aria_decrypt_64way)(const void *ctx, u8 *dst, const u8 *src);
+	void (*aria_ctr_crypt_64way)(const void *ctx, u8 *dst, const u8 *src,
+				     u8 *keystream, u8 *iv);
+
 
 };
 #endif

arch/x86/crypto/aria-gfni-avx512-asm_64.S new file +1023 -0

1,023 lines not on this page · raw message

arch/x86/crypto/aria_gfni_avx512_glue.c new file +232 -0

232 lines not on this page · raw message

include/crypto/aria.h +7 -1

index bc5dc71d6a98..50164cdf996b 100644
@@ -37,7 +37,13 @@
 #define ARIA_AESNI_AVX2_PARALLEL_BLOCKS 32
 #define ARIA_AESNI_AVX2_PARALLEL_BLOCK_SIZE  (ARIA_BLOCK_SIZE * 32)
 
-#if defined(CONFIG_CRYPTO_ARIA_AESNI_AVX2_X86_64) ||   \
+#define ARIA_GFNI_AVX512_PARALLEL_BLOCKS 64
+#define ARIA_GFNI_AVX512_PARALLEL_BLOCK_SIZE  (ARIA_BLOCK_SIZE * 64)
+
+#if defined(CRYPTO_ARIA_GFNI_AVX512_X86_64) || \
+	defined(CRYPTO_ARIA_GFNI_AVX512_X86_64_MODULE)
+#define ARIA_KEYSTREAM_SIZE ARIA_GFNI_AVX512_PARALLEL_BLOCK_SIZE
+#elif defined(CONFIG_CRYPTO_ARIA_AESNI_AVX2_X86_64) ||   \
 	defined(CONFIG_CRYPTO_ARIA_AESNI_AVX2_X86_64_MODULE)
 #define ARIA_KEYSTREAM_SIZE ARIA_AESNI_AVX2_PARALLEL_BLOCK_SIZE
 #elif defined(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) ||  \

Signature · 1 line

2.17.1

Elliott, Robert (Servers)

2022-11-05 16:21 UTC #4 lore raw

RE: [PATCH v2 3/3] crypto: aria: implement aria-avx512

Quoted 3 lines

+++ b/arch/x86/crypto/aria-gfni-avx512-asm_64.S

...

Review

Quoted 4 lines

(quoted diff) +4 -0

+/* struct aria_ctx: */
+#define enc_key 0
+#define dec_key 272
+#define rounds 544

I've noticed other modules with assembly code also has hardcoded assumptions about the offsets in the context structure. Sometimes they don't use macros, so they're less apparent. Example from aesni:

SYM_FUNC_START(aesni_dec)
FRAME_BEGIN
#ifndef __x86_64__
pushl KEYP
pushl KLEN
movl (FRAME_OFFSET+12)(%esp), KEYP # ctx
movl (FRAME_OFFSET+16)(%esp), OUTP # dst
movl (FRAME_OFFSET+20)(%esp), INP # src
#endif
mov 480(KEYP), KLEN # key length
add $240, KEYP
movups (INP), STATE # input
call _aesni_dec1

To ensure the C code and assembly code stay in agreement, it might be prudent to add a compile-time check in one of the C functions for each hardcoded offset value, like this for aesni:

arch/x86/crypto/aesni-intel_glue.c +4 -0

index 5db4814583ea..e4bcba765c77 100644
@@ -288,6 +288,10 @@ static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx,
        struct crypto_aes_ctx *ctx = aes_ctx(raw_ctx);
        int err;
 
+       BUILD_BUG_ON(offsetof(struct crypto_aes_ctx, key_enc) != 0);
+       BUILD_BUG_ON(offsetof(struct crypto_aes_ctx, key_dec) != 240);
+       BUILD_BUG_ON(offsetof(struct crypto_aes_ctx, key_length) != 480);
+
        if (key_len != AES_KEYSIZE_128 && key_len != AES_KEYSIZE_192 &&
            key_len != AES_KEYSIZE_256)
                return -EINVAL;

The sha functions and sm3 have such a check for the first field (but only that one) in their context structures:

    static int sha1_update(struct shash_desc *desc, const u8 *data,
    ...
        /*
         * Make sure struct sha1_state begins directly with the SHA1
         * 160-bit internal state, as this is what the asm functions expect.
         */
        BUILD_BUG_ON(offsetof(struct sha1_state, state) != 0);

Dave Hansen

2022-11-05 17:31 UTC #5 lore raw

Re: [PATCH v2 3/3] crypto: aria: implement aria-avx512

arch/x86/crypto/aesni-intel_glue.c static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx, · line 293

Quoted 9 lines · Robert (Servers)

arch/x86/crypto/aesni-intel_glue.c +3 -0

@@ -288,6 +288,10 @@ static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx,
        struct crypto_aes_ctx *ctx = aes_ctx(raw_ctx);
        int err;
 
+       BUILD_BUG_ON(offsetof(struct crypto_aes_ctx, key_enc) != 0);
+       BUILD_BUG_ON(offsetof(struct crypto_aes_ctx, key_dec) != 240);
+       BUILD_BUG_ON(offsetof(struct crypto_aes_ctx, key_length) != 480);

We have a nice fancy way of doing these. See things like CPU_ENTRY_AREA_entry_stack or TSS_sp0. It's all put together from arch/x86/kernel/asm-offsets.c and gets plopped in include/generated/asm-offsets.h.

This is vastly preferred to hard-coded magic number offsets, even if they do have a BUILD_BUG_ON() somewhere.

Taehee Yoo

author

2022-11-06 07:07 UTC #6 lore raw

Re: [PATCH v2 3/3] crypto: aria: implement aria-avx512

Hi Elliott and Dave,
Thanks a lot for the reviews!

Quoted 16 lines · Dave Hansen

arch/x86/crypto/aesni-intel_glue.c static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx, · line 293
Quoted 9 lines · Robert (Servers)
arch/x86/crypto/aesni-intel_glue.c +3 -0
@@ -288,6 +288,10 @@ static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx,
         struct crypto_aes_ctx *ctx = aes_ctx(raw_ctx);
         int err;
 
+       BUILD_BUG_ON(offsetof(struct crypto_aes_ctx, key_enc) != 0);
+       BUILD_BUG_ON(offsetof(struct crypto_aes_ctx, key_dec) != 240);
+       BUILD_BUG_ON(offsetof(struct crypto_aes_ctx, key_length) != 480);
We have a nice fancy way of doing these. See things like CPU_ENTRY_AREA_entry_stack or TSS_sp0. It's all put together from arch/x86/kernel/asm-offsets.c and gets plopped in include/generated/asm-offsets.h.

Quoted 2 lines

This is vastly preferred to hard-coded magic number offsets, even if they do have a BUILD_BUG_ON() somewhere.

I will define ARIA_CTX_xxx with asm-offsets.c.
Then, the assembly code can use the correct offset of enc_key, dec_key, and the rounds in the struct aria_ctx. Due to we can sure the offsets are correct, BUILD_BUG_ON() will become unnecessary.

I will send the v3 patch.

Thanks a lot!
Taehee Yoo