crypto: x86 - remove glue helper module
Subsystems (2)
v2 cover crypto: x86 - remove glue helper module
NOTE: this is a follow-up to '[RFC PATCH 00/10] crypto: x86 - remove XTS and CTR glue helper code' [0].
After applying my performance fixes for AES-NI in XTS mode, the only remaining users of the x86 glue helper module are the niche algorithms camellia, cast5/6, serpent and twofish.
It is not clear from the history why all these different versions of these algorithms in XTS and CTR modes were added in the first place: the only in-kernel references that seem to exist are to cbc(serpent), cbc(camellia) and cbc(twofish) in the IPsec stack. The XTS spec only mentions AES, and CTR modes don't seem to be widely used either.
Since the glue helper code relies heavily on indirect calls for small chunks of in/output, it needs some work to recover from the performance hit caused by the retpoline changes. However, it makes sense to only expend the effort for algorithms that are being used in the first place, and this does not seem to be the case for XTS and CTR.
CTR mode can simply be removed: it is not used in the kernel, and it is highly unlikely that it is being relied upon via algif_skcipher. And even if it was, the generic CTR mode driver can still provide the CTR transforms if necessary. While at it, accelerated implementations of DES and Blowfish in CTR mode are removed as well, for the same reasons.
XTS mode may actually be in use by dm-crypt users *, so we cannot simply drop this code entirely. However, as it turns out, the XTS template wrapped around the ECB mode skciphers perform roughly on par, and so there is no need to retain all the complicated XTS helper logic. Users of dm-crypt that are relying on xts(camellia) or xts(serpent) in the field should not be impacted by these changes at all.
ECB and CBC are retained, but the glue helper indirection is replaced with a set of preprocessor macros which can be used to instantiate the same logic flow, but without relying on indirect calls into the glue helper module. Please refer to patch #14 for more background.
* Milan points out that Serpent, Camellia and Twofish in XTS mode are used by TrueCrypt/Veracrypt, which means that dm-crypt should retain support for these algorithms as well.
[0] https://lore.kernel.org/linux-crypto/[email protected]/
Changes since v1:
- avoid unnecessary memcpy()s in the new CBC decryption helper macro
- avoid unnecessary memcpy() in camellia_decrypt_cbc_2way()
- add Eric's ack
Changes since RFC:
- add Eric's ack to the initial XTS and CTR patches
- add patches to convert ECB and CBC modes
- add patches to remove DES and Blowfish in CTR mode
Cc Megha Dey <[email protected]>
Cc Eric Biggers <[email protected]>
Cc Herbert Xu <[email protected]>
Cc Milan Broz <[email protected]>
Cc Mike Snitzer <[email protected]>
Ard Biesheuvel (21):
crypto: x86/camellia - switch to XTS template
crypto: x86/cast6 - switch to XTS template
crypto: x86/serpent- switch to XTS template
crypto: x86/twofish - switch to XTS template
crypto: x86/glue-helper - drop XTS helper routines
crypto: x86/camellia - drop CTR mode implementation
crypto: x86/serpent - drop CTR mode implementation
crypto: x86/cast5 - drop CTR mode implementation
crypto: x86/cast6 - drop CTR mode implementation
crypto: x86/twofish - drop CTR mode implementation
crypto: x86/glue-helper - drop CTR helper routines
crypto: x86/des - drop CTR mode implementation
crypto: x86/blowfish - drop CTR mode implementation
crypto: x86 - add some helper macros for ECB and CBC modes
crypto: x86/camellia - drop dependency on glue helper
crypto: x86/serpent - drop dependency on glue helper
crypto: x86/cast5 - drop dependency on glue helper
crypto: x86/cast6 - drop dependency on glue helper
crypto: x86/twofish - drop dependency on glue helper
crypto: x86 - remove glue helper module
crypto: x86 - use local headers for x86 specific shared declarations
arch/x86/crypto/Makefile | 2 -
arch/x86/crypto/blowfish_glue.c | 107 -----
arch/x86/crypto/camellia-aesni-avx-asm_64.S | 298 --------------
arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 351 ----------------
arch/x86/{include/asm => }/crypto/camellia.h | 24 --
arch/x86/crypto/camellia_aesni_avx2_glue.c | 198 ++-------
arch/x86/crypto/camellia_aesni_avx_glue.c | 216 +---------
arch/x86/crypto/camellia_glue.c | 145 ++-----
arch/x86/crypto/cast5_avx_glue.c | 287 +------------
arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 84 ----
arch/x86/crypto/cast6_avx_glue.c | 207 +---------
arch/x86/crypto/des3_ede_glue.c | 104 -----
arch/x86/crypto/ecb_cbc_helpers.h | 76 ++++
arch/x86/crypto/glue_helper-asm-avx.S | 104 -----
arch/x86/crypto/glue_helper-asm-avx2.S | 136 -------
arch/x86/crypto/glue_helper.c | 381 ------------------
arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 68 ----
arch/x86/crypto/serpent-avx.h | 21 +
arch/x86/crypto/serpent-avx2-asm_64.S | 87 ----
.../{include/asm => }/crypto/serpent-sse2.h | 0
arch/x86/crypto/serpent_avx2_glue.c | 185 +--------
arch/x86/crypto/serpent_avx_glue.c | 215 +---------
arch/x86/crypto/serpent_sse2_glue.c | 150 ++-----
arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 80 ----
arch/x86/{include/asm => }/crypto/twofish.h | 4 -
arch/x86/crypto/twofish_avx_glue.c | 211 +---------
arch/x86/crypto/twofish_glue_3way.c | 160 ++------
arch/x86/include/asm/crypto/glue_helper.h | 118 ------
arch/x86/include/asm/crypto/serpent-avx.h | 42 --
crypto/Kconfig | 30 +-
crypto/skcipher.c | 6 -
include/crypto/internal/skcipher.h | 1 -
32 files changed, 326 insertions(+), 3772 deletions(-)
rename arch/x86/{include/asm => }/crypto/camellia.h (69%)
create mode 100644 arch/x86/crypto/ecb_cbc_helpers.h
delete mode 100644 arch/x86/crypto/glue_helper.c
create mode 100644 arch/x86/crypto/serpent-avx.h
rename arch/x86/{include/asm => }/crypto/serpent-sse2.h (100%)
rename arch/x86/{include/asm => }/crypto/twofish.h (80%)
delete mode 100644 arch/x86/include/asm/crypto/glue_helper.h
delete mode 100644 arch/x86/include/asm/crypto/serpent-avx.hbase-commit: 16e733ce76cf5e48e22ffc0d2ed4720534753310
Signature · 1 line
2.17.1
PATCH v2 1/21 crypto: x86/camellia - switch to XTS template
Now that the XTS template can wrap accelerated ECB modes, it can be used to implement Camellia in XTS mode as well, which turns out to be at least as fast, and sometimes even faster.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 181 ----------------- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 207 -------------------- arch/x86/crypto/camellia_aesni_avx2_glue.c | 70 ------- arch/x86/crypto/camellia_aesni_avx_glue.c | 101 +--------- arch/x86/include/asm/crypto/camellia.h | 18 -- crypto/Kconfig | 2 +- 6 files changed, 2 insertions(+), 577 deletions(-)
arch/x86/crypto/camellia-aesni-avx-asm_64.S +0 -181
@@ -17,7 +17,6 @@ #include <linux/linkage.h> #include <asm/frame.h> -#include <asm/nospec-branch.h> #define CAMELLIA_TABLE_BYTE_LEN 272
@@ -593,10 +592,6 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) .Lbswap128_mask: .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 -/* For XTS mode IV generation */ -.Lxts_gf128mul_and_shl1_mask: - .byte 0x87, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0 - /* * pre-SubByte transform *
@@ -1111,179 +1106,3 @@ SYM_FUNC_START(camellia_ctr_16way) FRAME_END ret; SYM_FUNC_END(camellia_ctr_16way) - -#define gf128mul_x_ble(iv, mask, tmp) \ - vpsrad $31, iv, tmp; \ - vpaddq iv, iv, iv; \ - vpshufd $0x13, tmp, tmp; \ - vpand mask, tmp, tmp; \ - vpxor tmp, iv, iv; - -.align 8 -SYM_FUNC_START_LOCAL(camellia_xts_crypt_16way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst (16 blocks) - * %rdx: src (16 blocks) - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - * %r8: index for input whitening key - * %r9: pointer to __camellia_enc_blk16 or __camellia_dec_blk16 - */ - FRAME_BEGIN - - subq $(16 * 16), %rsp; - movq %rsp, %rax; - - vmovdqa .Lxts_gf128mul_and_shl1_mask, %xmm14; - - /* load IV */ - vmovdqu (%rcx), %xmm0; - vpxor 0 * 16(%rdx), %xmm0, %xmm15; - vmovdqu %xmm15, 15 * 16(%rax); - vmovdqu %xmm0, 0 * 16(%rsi); - - /* construct IVs */ - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 1 * 16(%rdx), %xmm0, %xmm15; - vmovdqu %xmm15, 14 * 16(%rax); - vmovdqu %xmm0, 1 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 2 * 16(%rdx), %xmm0, %xmm13; - vmovdqu %xmm0, 2 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 3 * 16(%rdx), %xmm0, %xmm12; - vmovdqu %xmm0, 3 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 4 * 16(%rdx), %xmm0, %xmm11; - vmovdqu %xmm0, 4 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 5 * 16(%rdx), %xmm0, %xmm10; - vmovdqu %xmm0, 5 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 6 * 16(%rdx), %xmm0, %xmm9; - vmovdqu %xmm0, 6 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 7 * 16(%rdx), %xmm0, %xmm8; - vmovdqu %xmm0, 7 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 8 * 16(%rdx), %xmm0, %xmm7; - vmovdqu %xmm0, 8 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 9 * 16(%rdx), %xmm0, %xmm6; - vmovdqu %xmm0, 9 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 10 * 16(%rdx), %xmm0, %xmm5; - vmovdqu %xmm0, 10 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 11 * 16(%rdx), %xmm0, %xmm4; - vmovdqu %xmm0, 11 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 12 * 16(%rdx), %xmm0, %xmm3; - vmovdqu %xmm0, 12 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 13 * 16(%rdx), %xmm0, %xmm2; - vmovdqu %xmm0, 13 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 14 * 16(%rdx), %xmm0, %xmm1; - vmovdqu %xmm0, 14 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 15 * 16(%rdx), %xmm0, %xmm15; - vmovdqu %xmm15, 0 * 16(%rax); - vmovdqu %xmm0, 15 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vmovdqu %xmm0, (%rcx); - - /* inpack16_pre: */ - vmovq (key_table)(CTX, %r8, 8), %xmm15; - vpshufb .Lpack_bswap, %xmm15, %xmm15; - vpxor 0 * 16(%rax), %xmm15, %xmm0; - vpxor %xmm1, %xmm15, %xmm1; - vpxor %xmm2, %xmm15, %xmm2; - vpxor %xmm3, %xmm15, %xmm3; - vpxor %xmm4, %xmm15, %xmm4; - vpxor %xmm5, %xmm15, %xmm5; - vpxor %xmm6, %xmm15, %xmm6; - vpxor %xmm7, %xmm15, %xmm7; - vpxor %xmm8, %xmm15, %xmm8; - vpxor %xmm9, %xmm15, %xmm9; - vpxor %xmm10, %xmm15, %xmm10; - vpxor %xmm11, %xmm15, %xmm11; - vpxor %xmm12, %xmm15, %xmm12; - vpxor %xmm13, %xmm15, %xmm13; - vpxor 14 * 16(%rax), %xmm15, %xmm14; - vpxor 15 * 16(%rax), %xmm15, %xmm15; - - CALL_NOSPEC r9; - - addq $(16 * 16), %rsp; - - vpxor 0 * 16(%rsi), %xmm7, %xmm7; - vpxor 1 * 16(%rsi), %xmm6, %xmm6; - vpxor 2 * 16(%rsi), %xmm5, %xmm5; - vpxor 3 * 16(%rsi), %xmm4, %xmm4; - vpxor 4 * 16(%rsi), %xmm3, %xmm3; - vpxor 5 * 16(%rsi), %xmm2, %xmm2; - vpxor 6 * 16(%rsi), %xmm1, %xmm1; - vpxor 7 * 16(%rsi), %xmm0, %xmm0; - vpxor 8 * 16(%rsi), %xmm15, %xmm15; - vpxor 9 * 16(%rsi), %xmm14, %xmm14; - vpxor 10 * 16(%rsi), %xmm13, %xmm13; - vpxor 11 * 16(%rsi), %xmm12, %xmm12; - vpxor 12 * 16(%rsi), %xmm11, %xmm11; - vpxor 13 * 16(%rsi), %xmm10, %xmm10; - vpxor 14 * 16(%rsi), %xmm9, %xmm9; - vpxor 15 * 16(%rsi), %xmm8, %xmm8; - write_output(%xmm7, %xmm6, %xmm5, %xmm4, %xmm3, %xmm2, %xmm1, %xmm0, - %xmm15, %xmm14, %xmm13, %xmm12, %xmm11, %xmm10, %xmm9, - %xmm8, %rsi); - - FRAME_END - ret; -SYM_FUNC_END(camellia_xts_crypt_16way) - -SYM_FUNC_START(camellia_xts_enc_16way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst (16 blocks) - * %rdx: src (16 blocks) - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - xorl %r8d, %r8d; /* input whitening key, 0 for enc */ - - leaq __camellia_enc_blk16, %r9; - - jmp camellia_xts_crypt_16way; -SYM_FUNC_END(camellia_xts_enc_16way) - -SYM_FUNC_START(camellia_xts_dec_16way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst (16 blocks) - * %rdx: src (16 blocks) - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - - cmpl $16, key_length(CTX); - movl $32, %r8d; - movl $24, %eax; - cmovel %eax, %r8d; /* input whitening key, last for dec */ - - leaq __camellia_dec_blk16, %r9; - - jmp camellia_xts_crypt_16way; -SYM_FUNC_END(camellia_xts_dec_16way)
arch/x86/crypto/camellia-aesni-avx2-asm_64.S +0 -207
222 lines not on this page · raw message
arch/x86/crypto/camellia_aesni_avx2_glue.c +0 -70
106 lines not on this page · raw message
arch/x86/crypto/camellia_aesni_avx_glue.c +1 -100
137 lines not on this page · raw message
arch/x86/include/asm/crypto/camellia.h +0 -18
@@ -19,18 +19,10 @@ struct camellia_ctx {u32 key_length; }; -struct camellia_xts_ctx { -struct camellia_ctx tweak_ctx; -struct camellia_ctx crypt_ctx; -}; -extern int __camellia_setkey (struct camellia_ctx *cctx,const unsigned char *key,unsigned int key_len); -extern int xts_camellia_setkey (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keylen); -/* regular block cipher functions */ asmlinkage void __camellia_enc_blk(const void *ctx, u8 *dst, const u8 *src,bool xor);
@@ -49,11 +41,6 @@ asmlinkage void camellia_cbc_dec_16way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void camellia_ctr_16way(const void *ctx, u8 *dst, const u8 *src,le128 *iv); -asmlinkage void camellia_xts_enc_16way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv); -asmlinkage void camellia_xts_dec_16way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv); -static inline void camellia_enc_blk (const void *ctx,u8 *dst,const u8 *src) {__camellia_enc_blk (ctx, dst, src, false);
@@ -83,9 +70,4 @@ extern void camellia_crypt_ctr(const void *ctx, u8 *dst, const u8 *src,extern void camellia_crypt_ctr_2way (const void *ctx,u8 *dst,const u8 *src,le128 *iv); -extern void camellia_xts_enc (const void *ctx,u8 *dst,const u8 *src, -le128 *iv); -extern void camellia_xts_dec (const void *ctx,u8 *dst,const u8 *src, -le128 *iv); -#endif /* ASM_X86_CAMELLIA_H */
crypto/Kconfig +1 -1
@@ -1305,7 +1305,7 @@ config CRYPTO_CAMELLIA_AESNI_AVX_X86_64 select CRYPTO_CAMELLIA_X86_64 select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SIMD - select CRYPTO_XTS + imply CRYPTO_XTS help Camellia cipher algorithm module (x86_64/AES-NI/AVX).
Signature · 1 line
2.17.1
PATCH v2 2/21 crypto: x86/cast6 - switch to XTS template
In reply to Ard BiesheuvelNow that the XTS template can wrap accelerated ECB modes, it can be used to implement CAST6 in XTS mode as well, which turns out to be at least as fast, and sometimes even faster
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 56 ----------- arch/x86/crypto/cast6_avx_glue.c | 98 -------------------- crypto/Kconfig | 2 +- 3 files changed, 1 insertion(+), 155 deletions(-)
arch/x86/crypto/cast6-avx-x86_64-asm_64.S +0 -56
@@ -212,8 +212,6 @@ .section .rodata.cst16, "aM", @progbits, 16 .align 16 -.Lxts_gf128mul_and_shl1_mask: - .byte 0x87, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0 .Lbswap_mask: .byte 3, 2, 1, 0, 7, 6, 5, 4, 11, 10, 9, 8, 15, 14, 13, 12 .Lbswap128_mask:
@@ -440,57 +438,3 @@ SYM_FUNC_START(cast6_ctr_8way) FRAME_END ret; SYM_FUNC_END(cast6_ctr_8way) - -SYM_FUNC_START(cast6_xts_enc_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - pushq %r15; - - movq %rdi, CTX - movq %rsi, %r11; - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2, - RX, RKR, RKM, .Lxts_gf128mul_and_shl1_mask); - - call __cast6_enc_blk8; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%r11, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - popq %r15; - FRAME_END - ret; -SYM_FUNC_END(cast6_xts_enc_8way) - -SYM_FUNC_START(cast6_xts_dec_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - pushq %r15; - - movq %rdi, CTX - movq %rsi, %r11; - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2, - RX, RKR, RKM, .Lxts_gf128mul_and_shl1_mask); - - call __cast6_dec_blk8; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%r11, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - popq %r15; - FRAME_END - ret; -SYM_FUNC_END(cast6_xts_dec_8way)
arch/x86/crypto/cast6_avx_glue.c +0 -98
140 lines not on this page · raw message
crypto/Kconfig +1 -1
@@ -1394,7 +1394,7 @@ config CRYPTO_CAST6_AVX_X86_64 select CRYPTO_CAST_COMMON select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SIMD - select CRYPTO_XTS + imply CRYPTO_XTS help The CAST6 encryption algorithm (synonymous with CAST-256) is described in RFC2612.
Signature · 1 line
2.17.1
PATCH v2 3/21 crypto: x86/serpent- switch to XTS template
In reply to Ard BiesheuvelNow that the XTS template can wrap accelerated ECB modes, it can be used to implement Serpent in XTS mode as well, which turns out to be at least as fast, and sometimes even faster
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 48 ---------- arch/x86/crypto/serpent-avx2-asm_64.S | 62 ------------ arch/x86/crypto/serpent_avx2_glue.c | 72 -------------- arch/x86/crypto/serpent_avx_glue.c | 101 -------------------- arch/x86/include/asm/crypto/serpent-avx.h | 21 ---- crypto/Kconfig | 2 +- 6 files changed, 1 insertion(+), 305 deletions(-)
arch/x86/crypto/serpent-avx-x86_64-asm_64.S +0 -48
@@ -18,10 +18,6 @@ .align 16 .Lbswap128_mask: .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 -.section .rodata.cst16.xts_gf128mul_and_shl1_mask, "aM", @progbits, 16 -.align 16 -.Lxts_gf128mul_and_shl1_mask: - .byte 0x87, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0 .text
@@ -735,47 +731,3 @@ SYM_FUNC_START(serpent_ctr_8way_avx) FRAME_END ret; SYM_FUNC_END(serpent_ctr_8way_avx) - -SYM_FUNC_START(serpent_xts_enc_8way_avx) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2, - RK0, RK1, RK2, .Lxts_gf128mul_and_shl1_mask); - - call __serpent_enc_blk8_avx; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - FRAME_END - ret; -SYM_FUNC_END(serpent_xts_enc_8way_avx) - -SYM_FUNC_START(serpent_xts_dec_8way_avx) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2, - RK0, RK1, RK2, .Lxts_gf128mul_and_shl1_mask); - - call __serpent_dec_blk8_avx; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%rsi, RC1, RD1, RB1, RE1, RC2, RD2, RB2, RE2); - - FRAME_END - ret; -SYM_FUNC_END(serpent_xts_dec_8way_avx)
arch/x86/crypto/serpent-avx2-asm_64.S +0 -62
71 lines not on this page · raw message
arch/x86/crypto/serpent_avx2_glue.c +0 -72
108 lines not on this page · raw message
arch/x86/crypto/serpent_avx_glue.c +0 -101
149 lines not on this page · raw message
arch/x86/include/asm/crypto/serpent-avx.h +0 -21
@@ -10,11 +10,6 @@ struct crypto_skcipher;#define SERPENT_PARALLEL_BLOCKS 8 -struct serpent_xts_ctx { -struct serpent_ctx tweak_ctx; -struct serpent_ctx crypt_ctx; -}; - asmlinkage void serpent_ecb_enc_8way_avx(const void *ctx, u8 *dst,const u8 *src); asmlinkage void serpent_ecb_dec_8way_avx(const void *ctx, u8 *dst,
@@ -22,21 +17,5 @@ asmlinkage void serpent_ecb_dec_8way_avx(const void *ctx, u8 *dst, asmlinkage void serpent_cbc_dec_8way_avx(const void *ctx, u8 *dst,const u8 *src); -asmlinkage void serpent_ctr_8way_avx(const void *ctx, u8 *dst, const u8 *src, -le128 *iv); - -asmlinkage void serpent_xts_enc_8way_avx(const void *ctx, u8 *dst, -const u8 *src,le128 *iv); -asmlinkage void serpent_xts_dec_8way_avx(const void *ctx, u8 *dst, -const u8 *src,le128 *iv); - -extern void __serpent_crypt_ctr (const void *ctx,u8 *dst,const u8 *src, -le128 *iv); - -extern void serpent_xts_enc (const void *ctx,u8 *dst,const u8 *src,le128 *iv); -extern void serpent_xts_dec (const void *ctx,u8 *dst,const u8 *src,le128 *iv); - -extern int xts_serpent_setkey (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keylen);#endif
crypto/Kconfig +1 -1
@@ -1576,7 +1576,7 @@ config CRYPTO_SERPENT_AVX_X86_64 select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SERPENT select CRYPTO_SIMD - select CRYPTO_XTS + imply CRYPTO_XTS help Serpent cipher algorithm, by Anderson, Biham & Knudsen.
Signature · 1 line
2.17.1
PATCH v2 4/21 crypto: x86/twofish - switch to XTS template
In reply to Ard BiesheuvelNow that the XTS template can wrap accelerated ECB modes, it can be used to implement Twofish in XTS mode as well, which turns out to be at least as fast, and sometimes even faster
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 53 ----------- arch/x86/crypto/twofish_avx_glue.c | 98 -------------------- crypto/Kconfig | 1 + 3 files changed, 1 insertion(+), 151 deletions(-)
arch/x86/crypto/twofish-avx-x86_64-asm_64.S +0 -53
@@ -19,11 +19,6 @@ .Lbswap128_mask: .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 -.section .rodata.cst16.xts_gf128mul_and_shl1_mask, "aM", @progbits, 16 -.align 16 -.Lxts_gf128mul_and_shl1_mask: - .byte 0x87, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0 - .text /* structure of crypto context */
@@ -406,51 +401,3 @@ SYM_FUNC_START(twofish_ctr_8way) FRAME_END ret; SYM_FUNC_END(twofish_ctr_8way) - -SYM_FUNC_START(twofish_xts_enc_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - - movq %rsi, %r11; - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2, - RX0, RX1, RY0, .Lxts_gf128mul_and_shl1_mask); - - call __twofish_enc_blk8; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%r11, RC1, RD1, RA1, RB1, RC2, RD2, RA2, RB2); - - FRAME_END - ret; -SYM_FUNC_END(twofish_xts_enc_8way) - -SYM_FUNC_START(twofish_xts_dec_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - - movq %rsi, %r11; - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RC1, RD1, RA1, RB1, RC2, RD2, RA2, RB2, - RX0, RX1, RY0, .Lxts_gf128mul_and_shl1_mask); - - call __twofish_dec_blk8; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%r11, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - FRAME_END - ret; -SYM_FUNC_END(twofish_xts_dec_8way)
arch/x86/crypto/twofish_avx_glue.c +0 -98
140 lines not on this page · raw message
crypto/Kconfig +1 -0
@@ -1731,6 +1731,7 @@ config CRYPTO_TWOFISH_AVX_X86_64 select CRYPTO_TWOFISH_COMMON select CRYPTO_TWOFISH_X86_64 select CRYPTO_TWOFISH_X86_64_3WAY + imply CRYPTO_XTS help Twofish cipher algorithm (x86_64/AVX).
Signature · 1 line
2.17.1
PATCH v2 5/21 crypto: x86/glue-helper - drop XTS helper routines
In reply to Ard BiesheuvelThe glue helper's XTS routines are no longer used, so drop them.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/glue_helper-asm-avx.S | 59 -------- arch/x86/crypto/glue_helper-asm-avx2.S | 78 ---------- arch/x86/crypto/glue_helper.c | 154 -------------------- arch/x86/include/asm/crypto/glue_helper.h | 12 -- 4 files changed, 303 deletions(-)
arch/x86/crypto/glue_helper-asm-avx.S +0 -59
@@ -79,62 +79,3 @@ vpxor (6*16)(src), x6, x6; \ vpxor (7*16)(src), x7, x7; \ store_8way(dst, x0, x1, x2, x3, x4, x5, x6, x7); - -#define gf128mul_x_ble(iv, mask, tmp) \ - vpsrad $31, iv, tmp; \ - vpaddq iv, iv, iv; \ - vpshufd $0x13, tmp, tmp; \ - vpand mask, tmp, tmp; \ - vpxor tmp, iv, iv; - -#define load_xts_8way(iv, src, dst, x0, x1, x2, x3, x4, x5, x6, x7, tiv, t0, \ - t1, xts_gf128mul_and_shl1_mask) \ - vmovdqa xts_gf128mul_and_shl1_mask, t0; \ - \ - /* load IV */ \ - vmovdqu (iv), tiv; \ - vpxor (0*16)(src), tiv, x0; \ - vmovdqu tiv, (0*16)(dst); \ - \ - /* construct and store IVs, also xor with source */ \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (1*16)(src), tiv, x1; \ - vmovdqu tiv, (1*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (2*16)(src), tiv, x2; \ - vmovdqu tiv, (2*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (3*16)(src), tiv, x3; \ - vmovdqu tiv, (3*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (4*16)(src), tiv, x4; \ - vmovdqu tiv, (4*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (5*16)(src), tiv, x5; \ - vmovdqu tiv, (5*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (6*16)(src), tiv, x6; \ - vmovdqu tiv, (6*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (7*16)(src), tiv, x7; \ - vmovdqu tiv, (7*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vmovdqu tiv, (iv); - -#define store_xts_8way(dst, x0, x1, x2, x3, x4, x5, x6, x7) \ - vpxor (0*16)(dst), x0, x0; \ - vpxor (1*16)(dst), x1, x1; \ - vpxor (2*16)(dst), x2, x2; \ - vpxor (3*16)(dst), x3, x3; \ - vpxor (4*16)(dst), x4, x4; \ - vpxor (5*16)(dst), x5, x5; \ - vpxor (6*16)(dst), x6, x6; \ - vpxor (7*16)(dst), x7, x7; \ - store_8way(dst, x0, x1, x2, x3, x4, x5, x6, x7);
arch/x86/crypto/glue_helper-asm-avx2.S +0 -78
81 lines not on this page · raw message
arch/x86/crypto/glue_helper.c +0 -154
166 lines not on this page · raw message
arch/x86/include/asm/crypto/glue_helper.h +0 -12
@@ -15,8 +15,6 @@ typedef void (*common_glue_func_t)(const void *ctx, u8 *dst, const u8 *src);typedef void (*common_glue_cbc_func_t )(const void *ctx,u8 *dst,const u8 *src);typedef void (*common_glue_ctr_func_t )(const void *ctx,u8 *dst,const u8 *src,le128 *iv); -typedef void (*common_glue_xts_func_t )(const void *ctx,u8 *dst,const u8 *src, -le128 *iv);struct common_glue_func_entry {unsigned int num_blocks;/* number of blocks that @fn will process */
@@ -24,7 +22,6 @@ struct common_glue_func_entry {common_glue_func_t ecb;common_glue_cbc_func_t cbc;common_glue_ctr_func_t ctr; -common_glue_xts_func_t xts; } fn_u; };
@@ -106,13 +103,4 @@ extern int glue_cbc_decrypt_req_128bit(const struct common_glue_ctx *gctx,extern int glue_ctr_req_128bit (const struct common_glue_ctx *gctx,struct skcipher_request *req); -extern int glue_xts_req_128bit (const struct common_glue_ctx *gctx, -struct skcipher_request *req, -common_glue_func_t tweak_fn,void *tweak_ctx, -void *crypt_ctx,bool decrypt); - -extern void glue_xts_crypt_128bit_one (const void *ctx,u8 *dst, -const u8 *src,le128 *iv, -common_glue_func_t fn); -#endif /* _CRYPTO_GLUE_HELPER_H */
Signature · 1 line
2.17.1
PATCH v2 6/21 crypto: x86/camellia - drop CTR mode implementation
In reply to Ard BiesheuvelCamellia in CTR mode is never used by the kernel directly, and is highly unlikely to be relied upon by dm-crypt or algif_skcipher. So let's drop the accelerated CTR mode implementation, and instead, rely on the CTR template and the bare cipher.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 117 ---------------- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 144 -------------------- arch/x86/crypto/camellia_aesni_avx2_glue.c | 41 ------ arch/x86/crypto/camellia_aesni_avx_glue.c | 40 ------ arch/x86/crypto/camellia_glue.c | 68 --------- arch/x86/include/asm/crypto/camellia.h | 6 - crypto/Kconfig | 1 + 7 files changed, 1 insertion(+), 416 deletions(-)
arch/x86/crypto/camellia-aesni-avx-asm_64.S +0 -117
@@ -588,10 +588,6 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) .long 0x80808080 .long 0x80808080 -/* For CTR-mode IV byteswap */ -.Lbswap128_mask: - .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 - /* * pre-SubByte transform *
@@ -993,116 +989,3 @@ SYM_FUNC_START(camellia_cbc_dec_16way) FRAME_END ret; SYM_FUNC_END(camellia_cbc_dec_16way) - -#define inc_le128(x, minus_one, tmp) \ - vpcmpeqq minus_one, x, tmp; \ - vpsubq minus_one, x, x; \ - vpslldq $8, tmp, tmp; \ - vpsubq tmp, x, x; - -SYM_FUNC_START(camellia_ctr_16way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst (16 blocks) - * %rdx: src (16 blocks) - * %rcx: iv (little endian, 128bit) - */ - FRAME_BEGIN - - subq $(16 * 16), %rsp; - movq %rsp, %rax; - - vmovdqa .Lbswap128_mask, %xmm14; - - /* load IV and byteswap */ - vmovdqu (%rcx), %xmm0; - vpshufb %xmm14, %xmm0, %xmm15; - vmovdqu %xmm15, 15 * 16(%rax); - - vpcmpeqd %xmm15, %xmm15, %xmm15; - vpsrldq $8, %xmm15, %xmm15; /* low: -1, high: 0 */ - - /* construct IVs */ - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm13; - vmovdqu %xmm13, 14 * 16(%rax); - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm13; - vmovdqu %xmm13, 13 * 16(%rax); - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm12; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm11; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm10; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm9; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm8; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm7; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm6; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm5; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm4; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm3; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm2; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm1; - inc_le128(%xmm0, %xmm15, %xmm13); - vmovdqa %xmm0, %xmm13; - vpshufb %xmm14, %xmm0, %xmm0; - inc_le128(%xmm13, %xmm15, %xmm14); - vmovdqu %xmm13, (%rcx); - - /* inpack16_pre: */ - vmovq (key_table)(CTX), %xmm15; - vpshufb .Lpack_bswap, %xmm15, %xmm15; - vpxor %xmm0, %xmm15, %xmm0; - vpxor %xmm1, %xmm15, %xmm1; - vpxor %xmm2, %xmm15, %xmm2; - vpxor %xmm3, %xmm15, %xmm3; - vpxor %xmm4, %xmm15, %xmm4; - vpxor %xmm5, %xmm15, %xmm5; - vpxor %xmm6, %xmm15, %xmm6; - vpxor %xmm7, %xmm15, %xmm7; - vpxor %xmm8, %xmm15, %xmm8; - vpxor %xmm9, %xmm15, %xmm9; - vpxor %xmm10, %xmm15, %xmm10; - vpxor %xmm11, %xmm15, %xmm11; - vpxor %xmm12, %xmm15, %xmm12; - vpxor 13 * 16(%rax), %xmm15, %xmm13; - vpxor 14 * 16(%rax), %xmm15, %xmm14; - vpxor 15 * 16(%rax), %xmm15, %xmm15; - - call __camellia_enc_blk16; - - addq $(16 * 16), %rsp; - - vpxor 0 * 16(%rdx), %xmm7, %xmm7; - vpxor 1 * 16(%rdx), %xmm6, %xmm6; - vpxor 2 * 16(%rdx), %xmm5, %xmm5; - vpxor 3 * 16(%rdx), %xmm4, %xmm4; - vpxor 4 * 16(%rdx), %xmm3, %xmm3; - vpxor 5 * 16(%rdx), %xmm2, %xmm2; - vpxor 6 * 16(%rdx), %xmm1, %xmm1; - vpxor 7 * 16(%rdx), %xmm0, %xmm0; - vpxor 8 * 16(%rdx), %xmm15, %xmm15; - vpxor 9 * 16(%rdx), %xmm14, %xmm14; - vpxor 10 * 16(%rdx), %xmm13, %xmm13; - vpxor 11 * 16(%rdx), %xmm12, %xmm12; - vpxor 12 * 16(%rdx), %xmm11, %xmm11; - vpxor 13 * 16(%rdx), %xmm10, %xmm10; - vpxor 14 * 16(%rdx), %xmm9, %xmm9; - vpxor 15 * 16(%rdx), %xmm8, %xmm8; - write_output(%xmm7, %xmm6, %xmm5, %xmm4, %xmm3, %xmm2, %xmm1, %xmm0, - %xmm15, %xmm14, %xmm13, %xmm12, %xmm11, %xmm10, %xmm9, - %xmm8, %rsi); - - FRAME_END - ret; -SYM_FUNC_END(camellia_ctr_16way)
arch/x86/crypto/camellia-aesni-avx2-asm_64.S +0 -144
153 lines not on this page · raw message
arch/x86/crypto/camellia_aesni_avx2_glue.c +0 -41
65 lines not on this page · raw message
arch/x86/crypto/camellia_aesni_avx_glue.c +0 -40
@@ -26,10 +26,6 @@ EXPORT_SYMBOL_GPL(camellia_ecb_dec_16way); asmlinkage void camellia_cbc_dec_16way(const void *ctx, u8 *dst, const u8 *src);EXPORT_SYMBOL_GPL (camellia_cbc_dec_16way); -asmlinkage void camellia_ctr_16way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv); -EXPORT_SYMBOL_GPL (camellia_ctr_16way); -static const struct common_glue_ctx camellia_enc = { .num_funcs =3 , .fpu_blocks_limit =CAMELLIA_AESNI_PARALLEL_BLOCKS ,
@@ -46,22 +42,6 @@ static const struct common_glue_ctx camellia_enc = { } } }; -static const struct common_glue_ctx camellia_ctr = { - .num_funcs =3 , - .fpu_blocks_limit =CAMELLIA_AESNI_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =CAMELLIA_AESNI_PARALLEL_BLOCKS , - .fn_u = { .ctr = camellia_ctr_16way } - }, { - .num_blocks =2 , - .fn_u = { .ctr = camellia_crypt_ctr_2way } - }, { - .num_blocks =1 , - .fn_u = { .ctr = camellia_crypt_ctr } - } } -}; -static const struct common_glue_ctx camellia_dec = { .num_funcs =3 , .fpu_blocks_limit =CAMELLIA_AESNI_PARALLEL_BLOCKS ,
@@ -120,11 +100,6 @@ static int cbc_decrypt(struct skcipher_request *req)return glue_cbc_decrypt_req_128bit (&camellia_dec_cbc, req); } -static int ctr_crypt (struct skcipher_request *req) -{ -return glue_ctr_req_128bit (&camellia_ctr, req); -} -static struct skcipher_alg camellia_algs[] = { { .base.cra_name ="__ecb(camellia)" ,
@@ -153,21 +128,6 @@ static struct skcipher_alg camellia_algs[] = { .setkey = camellia_setkey, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, - }, { - .base.cra_name = "__ctr(camellia)", - .base.cra_driver_name = "__ctr-camellia-aesni", - .base.cra_priority = 400, - .base.cra_flags = CRYPTO_ALG_INTERNAL, - .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct camellia_ctx), - .base.cra_module = THIS_MODULE, - .min_keysize = CAMELLIA_MIN_KEY_SIZE, - .max_keysize = CAMELLIA_MAX_KEY_SIZE, - .ivsize = CAMELLIA_BLOCK_SIZE, - .chunksize = CAMELLIA_BLOCK_SIZE, - .setkey = camellia_setkey, - .encrypt = ctr_crypt, - .decrypt = ctr_crypt, } };
arch/x86/crypto/camellia_glue.c +0 -68
92 lines not on this page · raw message
arch/x86/include/asm/crypto/camellia.h +0 -6
@@ -38,8 +38,6 @@ asmlinkage void camellia_ecb_enc_16way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void camellia_ecb_dec_16way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void camellia_cbc_dec_16way(const void *ctx, u8 *dst, const u8 *src); -asmlinkage void camellia_ctr_16way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv);static inline void camellia_enc_blk (const void *ctx,u8 *dst,const u8 *src) {
@@ -65,9 +63,5 @@ static inline void camellia_enc_blk_xor_2way(const void *ctx, u8 *dst,/* glue helpers */ extern void camellia_decrypt_cbc_2way (const void *ctx,u8 *dst,const u8 *src); -extern void camellia_crypt_ctr (const void *ctx,u8 *dst,const u8 *src, -le128 *iv); -extern void camellia_crypt_ctr_2way (const void *ctx,u8 *dst,const u8 *src, -le128 *iv);#endif /* ASM_X86_CAMELLIA_H */
crypto/Kconfig +1 -0
@@ -1286,6 +1286,7 @@ config CRYPTO_CAMELLIA_X86_64 depends on CRYPTO select CRYPTO_SKCIPHER select CRYPTO_GLUE_HELPER_X86 + imply CRYPTO_CTR help Camellia cipher algorithm module (x86_64).
Signature · 1 line
2.17.1
PATCH v2 7/21 crypto: x86/serpent - drop CTR mode implementation
In reply to Ard BiesheuvelSerpent in CTR mode is never used by the kernel directly, and is highly unlikely to be relied upon by dm-crypt or algif_skcipher. So let's drop the accelerated CTR mode implementation, and instead, rely on the CTR template and the bare cipher.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 20 ------ arch/x86/crypto/serpent-avx2-asm_64.S | 25 -------- arch/x86/crypto/serpent_avx2_glue.c | 38 ----------- arch/x86/crypto/serpent_avx_glue.c | 51 --------------- arch/x86/crypto/serpent_sse2_glue.c | 67 -------------------- crypto/Kconfig | 3 + 6 files changed, 3 insertions(+), 201 deletions(-)
arch/x86/crypto/serpent-avx-x86_64-asm_64.S +0 -20
@@ -711,23 +711,3 @@ SYM_FUNC_START(serpent_cbc_dec_8way_avx) FRAME_END ret; SYM_FUNC_END(serpent_cbc_dec_8way_avx) - -SYM_FUNC_START(serpent_ctr_8way_avx) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (little endian, 128bit) - */ - FRAME_BEGIN - - load_ctr_8way(%rcx, .Lbswap128_mask, RA1, RB1, RC1, RD1, RA2, RB2, RC2, - RD2, RK0, RK1, RK2); - - call __serpent_enc_blk8_avx; - - store_ctr_8way(%rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - FRAME_END - ret; -SYM_FUNC_END(serpent_ctr_8way_avx)
arch/x86/crypto/serpent-avx2-asm_64.S +0 -25
@@ -724,28 +724,3 @@ SYM_FUNC_START(serpent_cbc_dec_16way) FRAME_END ret; SYM_FUNC_END(serpent_cbc_dec_16way) - -SYM_FUNC_START(serpent_ctr_16way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst (16 blocks) - * %rdx: src (16 blocks) - * %rcx: iv (little endian, 128bit) - */ - FRAME_BEGIN - - vzeroupper; - - load_ctr_16way(%rcx, .Lbswap128_mask, RA1, RB1, RC1, RD1, RA2, RB2, RC2, - RD2, RK0, RK0x, RK1, RK1x, RK2, RK2x, RK3, RK3x, RNOT, - tp); - - call __serpent_enc_blk16; - - store_ctr_16way(%rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - vzeroupper; - - FRAME_END - ret; -SYM_FUNC_END(serpent_ctr_16way)
arch/x86/crypto/serpent_avx2_glue.c +0 -38
@@ -22,8 +22,6 @@ asmlinkage void serpent_ecb_enc_16way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void serpent_ecb_dec_16way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void serpent_cbc_dec_16way(const void *ctx, u8 *dst, const u8 *src); -asmlinkage void serpent_ctr_16way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv);static int serpent_setkey_skcipher (struct crypto_skcipher *tfm,const u8 *key,unsigned int keylen) {
@@ -46,22 +44,6 @@ static const struct common_glue_ctx serpent_enc = { } } }; -static const struct common_glue_ctx serpent_ctr = { - .num_funcs =3 , - .fpu_blocks_limit =8 , - - .funcs = { { - .num_blocks =16 , - .fn_u = { .ctr = serpent_ctr_16way } - }, { - .num_blocks =8 , - .fn_u = { .ctr = serpent_ctr_8way_avx } - }, { - .num_blocks =1 , - .fn_u = { .ctr = __serpent_crypt_ctr } - } } -}; -static const struct common_glue_ctx serpent_dec = { .num_funcs =3 , .fpu_blocks_limit =8 ,
@@ -114,11 +96,6 @@ static int cbc_decrypt(struct skcipher_request *req)return glue_cbc_decrypt_req_128bit (&serpent_dec_cbc, req); } -static int ctr_crypt (struct skcipher_request *req) -{ -return glue_ctr_req_128bit (&serpent_ctr, req); -} -static struct skcipher_alg serpent_algs[] = { { .base.cra_name ="__ecb(serpent)" ,
@@ -147,21 +124,6 @@ static struct skcipher_alg serpent_algs[] = { .setkey = serpent_setkey_skcipher, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, - }, { - .base.cra_name = "__ctr(serpent)", - .base.cra_driver_name = "__ctr-serpent-avx2", - .base.cra_priority = 600, - .base.cra_flags = CRYPTO_ALG_INTERNAL, - .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct serpent_ctx), - .base.cra_module = THIS_MODULE, - .min_keysize = SERPENT_MIN_KEY_SIZE, - .max_keysize = SERPENT_MAX_KEY_SIZE, - .ivsize = SERPENT_BLOCK_SIZE, - .chunksize = SERPENT_BLOCK_SIZE, - .setkey = serpent_setkey_skcipher, - .encrypt = ctr_crypt, - .decrypt = ctr_crypt, }, };
arch/x86/crypto/serpent_avx_glue.c +0 -51
75 lines not on this page · raw message
arch/x86/crypto/serpent_sse2_glue.c +0 -67
97 lines not on this page · raw message
crypto/Kconfig +3 -0
@@ -1539,6 +1539,7 @@ config CRYPTO_SERPENT_SSE2_X86_64 select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SERPENT select CRYPTO_SIMD + imply CRYPTO_CTR help Serpent cipher algorithm, by Anderson, Biham & Knudsen.
@@ -1558,6 +1559,7 @@ config CRYPTO_SERPENT_SSE2_586 select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SERPENT select CRYPTO_SIMD + imply CRYPTO_CTR help Serpent cipher algorithm, by Anderson, Biham & Knudsen.
@@ -1578,6 +1580,7 @@ config CRYPTO_SERPENT_AVX_X86_64 select CRYPTO_SERPENT select CRYPTO_SIMD imply CRYPTO_XTS + imply CRYPTO_CTR help Serpent cipher algorithm, by Anderson, Biham & Knudsen.
Signature · 1 line
2.17.1
PATCH v2 8/21 crypto: x86/cast5 - drop CTR mode implementation
In reply to Ard BiesheuvelCAST5 in CTR mode is never used by the kernel directly, and is highly unlikely to be relied upon by dm-crypt or algif_skcipher. So let's drop the accelerated CTR mode implementation, and instead, rely on the CTR template and the bare cipher.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/cast5_avx_glue.c | 103 -------------------- crypto/Kconfig | 1 + 2 files changed, 1 insertion(+), 103 deletions(-)
arch/x86/crypto/cast5_avx_glue.c +0 -103
@@ -23,8 +23,6 @@ asmlinkage void cast5_ecb_dec_16way(struct cast5_ctx *ctx, u8 *dst, const u8 *src); asmlinkage void cast5_cbc_dec_16way(struct cast5_ctx *ctx, u8 *dst,const u8 *src); -asmlinkage void cast5_ctr_16way(struct cast5_ctx *ctx, u8 *dst, const u8 *src, -__be64 *iv);static int cast5_setkey_skcipher (struct crypto_skcipher *tfm,const u8 *key,unsigned int keylen)
@@ -214,92 +212,6 @@ static int cbc_decrypt(struct skcipher_request *req)return err; } -static void ctr_crypt_final (struct skcipher_walk *walk,struct cast5_ctx *ctx) -{ -u8 *ctrblk = walk->iv; -u8 keystream[CAST5_BLOCK_SIZE ]; -u8 *src = walk->src.virt.addr; -u8 *dst = walk->dst.virt.addr; -unsigned int nbytes = walk->nbytes; - -__cast5_encrypt (ctx, keystream, ctrblk); -crypto_xor_cpy (dst, keystream, src, nbytes); - -crypto_inc (ctrblk,CAST5_BLOCK_SIZE ); -} - -static unsigned int __ctr_crypt (struct skcipher_walk *walk, -struct cast5_ctx *ctx) -{ -const unsigned int bsize =CAST5_BLOCK_SIZE ; -unsigned int nbytes = walk->nbytes; -u64 *src = (u64 *)walk->src.virt.addr; -u64 *dst = (u64 *)walk->dst.virt.addr; - -/* Process multi-block batch */ -if (nbytes >= bsize *CAST5_PARALLEL_BLOCKS ) { -do { -cast5_ctr_16way (ctx, (u8 *)dst, (u8 *)src, - (__be64 *)walk->iv); - - src +=CAST5_PARALLEL_BLOCKS ; - dst +=CAST5_PARALLEL_BLOCKS ; - nbytes -= bsize *CAST5_PARALLEL_BLOCKS ; - }while (nbytes >= bsize *CAST5_PARALLEL_BLOCKS ); - -if (nbytes < bsize) - goto done; - } - -/* Handle leftovers */ -do { -u64 ctrblk; - -if (dst != src) - *dst = *src; - - ctrblk = *(u64 *)walk->iv; -be64_add_cpu ((__be64 *)walk->iv,1 ); - -__cast5_encrypt (ctx, (u8 *)&ctrblk, (u8 *)&ctrblk); - *dst ^= ctrblk; - - src +=1 ; - dst +=1 ; - nbytes -= bsize; - }while (nbytes >= bsize); - -done: -return nbytes; -} - -static int ctr_crypt (struct skcipher_request *req) -{ -struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct cast5_ctx *ctx =crypto_skcipher_ctx (tfm); -bool fpu_enabled = false; -struct skcipher_walk walk; -unsigned int nbytes; -int err; - - err =skcipher_walk_virt (&walk, req, false); - -while ((nbytes = walk.nbytes) >=CAST5_BLOCK_SIZE ) { - fpu_enabled =cast5_fpu_begin (fpu_enabled, &walk, nbytes); - nbytes =__ctr_crypt (&walk, ctx); - err =skcipher_walk_done (&walk, nbytes); - } - -cast5_fpu_end (fpu_enabled); - -if (walk.nbytes) { -ctr_crypt_final (&walk, ctx); - err =skcipher_walk_done (&walk,0 ); - } - -return err; -} -static struct skcipher_alg cast5_algs[] = { { .base.cra_name ="__ecb(cast5)" ,
@@ -328,21 +240,6 @@ static struct skcipher_alg cast5_algs[] = { .setkey = cast5_setkey_skcipher, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, - }, { - .base.cra_name = "__ctr(cast5)", - .base.cra_driver_name = "__ctr-cast5-avx", - .base.cra_priority = 200, - .base.cra_flags = CRYPTO_ALG_INTERNAL, - .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct cast5_ctx), - .base.cra_module = THIS_MODULE, - .min_keysize = CAST5_MIN_KEY_SIZE, - .max_keysize = CAST5_MAX_KEY_SIZE, - .ivsize = CAST5_BLOCK_SIZE, - .chunksize = CAST5_BLOCK_SIZE, - .setkey = cast5_setkey_skcipher, - .encrypt = ctr_crypt, - .decrypt = ctr_crypt, } };
crypto/Kconfig +1 -0
@@ -1372,6 +1372,7 @@ config CRYPTO_CAST5_AVX_X86_64 select CRYPTO_CAST5 select CRYPTO_CAST_COMMON select CRYPTO_SIMD + imply CRYPTO_CTR help The CAST5 encryption algorithm (synonymous with CAST-128) is described in RFC2144.
Signature · 1 line
2.17.1
PATCH v2 9/21 crypto: x86/cast6 - drop CTR mode implementation
In reply to Ard BiesheuvelCAST6 in CTR mode is never used by the kernel directly, and is highly unlikely to be relied upon by dm-crypt or algif_skcipher. So let's drop the accelerated CTR mode implementation, and instead, rely on the CTR template and the bare cipher.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 28 ------------ arch/x86/crypto/cast6_avx_glue.c | 48 -------------------- crypto/Kconfig | 1 + 3 files changed, 1 insertion(+), 76 deletions(-)
arch/x86/crypto/cast6-avx-x86_64-asm_64.S +0 -28
@@ -410,31 +410,3 @@ SYM_FUNC_START(cast6_cbc_dec_8way) FRAME_END ret; SYM_FUNC_END(cast6_cbc_dec_8way) - -SYM_FUNC_START(cast6_ctr_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (little endian, 128bit) - */ - FRAME_BEGIN - pushq %r12; - pushq %r15 - - movq %rdi, CTX; - movq %rsi, %r11; - movq %rdx, %r12; - - load_ctr_8way(%rcx, .Lbswap128_mask, RA1, RB1, RC1, RD1, RA2, RB2, RC2, - RD2, RX, RKR, RKM); - - call __cast6_enc_blk8; - - store_ctr_8way(%r12, %r11, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - popq %r15; - popq %r12; - FRAME_END - ret; -SYM_FUNC_END(cast6_ctr_8way)
arch/x86/crypto/cast6_avx_glue.c +0 -48
78 lines not on this page · raw message
crypto/Kconfig +1 -0
@@ -1397,6 +1397,7 @@ config CRYPTO_CAST6_AVX_X86_64 select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SIMD imply CRYPTO_XTS + imply CRYPTO_CTR help The CAST6 encryption algorithm (synonymous with CAST-256) is described in RFC2612.
Signature · 1 line
2.17.1
PATCH v2 10/21 crypto: x86/twofish - drop CTR mode implementation
In reply to Ard BiesheuvelTwofish in CTR mode is never used by the kernel directly, and is highly unlikely to be relied upon by dm-crypt or algif_skcipher. So let's drop the accelerated CTR mode implementation, and instead, rely on the CTR template and the bare cipher.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 27 ------- arch/x86/crypto/twofish_avx_glue.c | 38 ---------- arch/x86/crypto/twofish_glue_3way.c | 78 -------------------- arch/x86/include/asm/crypto/twofish.h | 4 - crypto/Kconfig | 2 + 5 files changed, 2 insertions(+), 147 deletions(-)
arch/x86/crypto/twofish-avx-x86_64-asm_64.S +0 -27
@@ -374,30 +374,3 @@ SYM_FUNC_START(twofish_cbc_dec_8way) FRAME_END ret; SYM_FUNC_END(twofish_cbc_dec_8way) - -SYM_FUNC_START(twofish_ctr_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (little endian, 128bit) - */ - FRAME_BEGIN - - pushq %r12; - - movq %rsi, %r11; - movq %rdx, %r12; - - load_ctr_8way(%rcx, .Lbswap128_mask, RA1, RB1, RC1, RD1, RA2, RB2, RC2, - RD2, RX0, RX1, RY0); - - call __twofish_enc_blk8; - - store_ctr_8way(%r12, %r11, RC1, RD1, RA1, RB1, RC2, RD2, RA2, RB2); - - popq %r12; - - FRAME_END - ret; -SYM_FUNC_END(twofish_ctr_8way)
arch/x86/crypto/twofish_avx_glue.c +0 -38
@@ -25,8 +25,6 @@ asmlinkage void twofish_ecb_enc_8way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void twofish_ecb_dec_8way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void twofish_cbc_dec_8way(const void *ctx, u8 *dst, const u8 *src); -asmlinkage void twofish_ctr_8way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv);static int twofish_setkey_skcipher (struct crypto_skcipher *tfm,const u8 *key,unsigned int keylen)
@@ -55,22 +53,6 @@ static const struct common_glue_ctx twofish_enc = { } } }; -static const struct common_glue_ctx twofish_ctr = { - .num_funcs =3 , - .fpu_blocks_limit =TWOFISH_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =TWOFISH_PARALLEL_BLOCKS , - .fn_u = { .ctr = twofish_ctr_8way } - }, { - .num_blocks =3 , - .fn_u = { .ctr = twofish_enc_blk_ctr_3way } - }, { - .num_blocks =1 , - .fn_u = { .ctr = twofish_enc_blk_ctr } - } } -}; -static const struct common_glue_ctx twofish_dec = { .num_funcs =3 , .fpu_blocks_limit =TWOFISH_PARALLEL_BLOCKS ,
@@ -123,11 +105,6 @@ static int cbc_decrypt(struct skcipher_request *req)return glue_cbc_decrypt_req_128bit (&twofish_dec_cbc, req); } -static int ctr_crypt (struct skcipher_request *req) -{ -return glue_ctr_req_128bit (&twofish_ctr, req); -} -static struct skcipher_alg twofish_algs[] = { { .base.cra_name ="__ecb(twofish)" ,
@@ -156,21 +133,6 @@ static struct skcipher_alg twofish_algs[] = { .setkey = twofish_setkey_skcipher, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, - }, { - .base.cra_name = "__ctr(twofish)", - .base.cra_driver_name = "__ctr-twofish-avx", - .base.cra_priority = 400, - .base.cra_flags = CRYPTO_ALG_INTERNAL, - .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct twofish_ctx), - .base.cra_module = THIS_MODULE, - .min_keysize = TF_MIN_KEY_SIZE, - .max_keysize = TF_MAX_KEY_SIZE, - .ivsize = TF_BLOCK_SIZE, - .chunksize = TF_BLOCK_SIZE, - .setkey = twofish_setkey_skcipher, - .encrypt = ctr_crypt, - .decrypt = ctr_crypt, }, };
arch/x86/crypto/twofish_glue_3way.c +0 -78
108 lines not on this page · raw message
arch/x86/include/asm/crypto/twofish.h +0 -4
@@ -17,9 +17,5 @@ asmlinkage void twofish_dec_blk_3way(const void *ctx, u8 *dst, const u8 *src);/* helpers from twofish_x86_64-3way module */ extern void twofish_dec_blk_cbc_3way (const void *ctx,u8 *dst,const u8 *src); -extern void twofish_enc_blk_ctr (const void *ctx,u8 *dst,const u8 *src, -le128 *iv); -extern void twofish_enc_blk_ctr_3way (const void *ctx,u8 *dst,const u8 *src, -le128 *iv);#endif /* ASM_X86_TWOFISH_H */
crypto/Kconfig +2 -0
@@ -1680,6 +1680,7 @@ config CRYPTO_TWOFISH_586 depends on (X86 || UML_X86) && !64BIT select CRYPTO_ALGAPI select CRYPTO_TWOFISH_COMMON + imply CRYPTO_CTR help Twofish cipher algorithm.
@@ -1696,6 +1697,7 @@ config CRYPTO_TWOFISH_X86_64 depends on (X86 || UML_X86) && 64BIT select CRYPTO_ALGAPI select CRYPTO_TWOFISH_COMMON + imply CRYPTO_CTR help Twofish cipher algorithm (x86_64).
Signature · 1 line
2.17.1
PATCH v2 11/21 crypto: x86/glue-helper - drop CTR helper routines
In reply to Ard BiesheuvelThe glue helper's CTR routines are no longer used, so drop them.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/glue_helper-asm-avx.S | 45 ------------ arch/x86/crypto/glue_helper-asm-avx2.S | 58 ---------------- arch/x86/crypto/glue_helper.c | 72 -------------------- arch/x86/include/asm/crypto/glue_helper.h | 32 --------- 4 files changed, 207 deletions(-)
arch/x86/crypto/glue_helper-asm-avx.S +0 -45
@@ -34,48 +34,3 @@ vpxor (5*16)(src), x6, x6; \ vpxor (6*16)(src), x7, x7; \ store_8way(dst, x0, x1, x2, x3, x4, x5, x6, x7); - -#define inc_le128(x, minus_one, tmp) \ - vpcmpeqq minus_one, x, tmp; \ - vpsubq minus_one, x, x; \ - vpslldq $8, tmp, tmp; \ - vpsubq tmp, x, x; - -#define load_ctr_8way(iv, bswap, x0, x1, x2, x3, x4, x5, x6, x7, t0, t1, t2) \ - vpcmpeqd t0, t0, t0; \ - vpsrldq $8, t0, t0; /* low: -1, high: 0 */ \ - vmovdqa bswap, t1; \ - \ - /* load IV and byteswap */ \ - vmovdqu (iv), x7; \ - vpshufb t1, x7, x0; \ - \ - /* construct IVs */ \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x1; \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x2; \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x3; \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x4; \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x5; \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x6; \ - inc_le128(x7, t0, t2); \ - vmovdqa x7, t2; \ - vpshufb t1, x7, x7; \ - inc_le128(t2, t0, t1); \ - vmovdqu t2, (iv); - -#define store_ctr_8way(src, dst, x0, x1, x2, x3, x4, x5, x6, x7) \ - vpxor (0*16)(src), x0, x0; \ - vpxor (1*16)(src), x1, x1; \ - vpxor (2*16)(src), x2, x2; \ - vpxor (3*16)(src), x3, x3; \ - vpxor (4*16)(src), x4, x4; \ - vpxor (5*16)(src), x5, x5; \ - vpxor (6*16)(src), x6, x6; \ - vpxor (7*16)(src), x7, x7; \ - store_8way(dst, x0, x1, x2, x3, x4, x5, x6, x7);
arch/x86/crypto/glue_helper-asm-avx2.S +0 -58
61 lines not on this page · raw message
arch/x86/crypto/glue_helper.c +0 -72
82 lines not on this page · raw message
arch/x86/include/asm/crypto/glue_helper.h +0 -32
@@ -9,19 +9,15 @@#include <crypto/internal/skcipher.h> #include <linux/kernel.h> #include <asm/fpu/api.h> -#include <crypto/b128ops.h> typedef void (*common_glue_func_t )(const void *ctx,u8 *dst,const u8 *src);typedef void (*common_glue_cbc_func_t )(const void *ctx,u8 *dst,const u8 *src); -typedef void (*common_glue_ctr_func_t )(const void *ctx,u8 *dst,const u8 *src, -le128 *iv);struct common_glue_func_entry {unsigned int num_blocks;/* number of blocks that @fn will process */ union {common_glue_func_t ecb;common_glue_cbc_func_t cbc; -common_glue_ctr_func_t ctr; } fn_u; };
@@ -66,31 +62,6 @@ static inline void glue_fpu_end(bool fpu_enabled)kernel_fpu_end (); } -static inline void le128_to_be128 (be128 *dst,const le128 *src) -{ - dst->a =cpu_to_be64 (le64_to_cpu (src->a)); - dst->b =cpu_to_be64 (le64_to_cpu (src->b)); -} - -static inline void be128_to_le128 (le128 *dst,const be128 *src) -{ - dst->a =cpu_to_le64 (be64_to_cpu (src->a)); - dst->b =cpu_to_le64 (be64_to_cpu (src->b)); -} - -static inline void le128_inc (le128 *i) -{ -u64 a =le64_to_cpu (i->a); -u64 b =le64_to_cpu (i->b); - - b++; -if (!b) - a++; - - i->a =cpu_to_le64 (a); - i->b =cpu_to_le64 (b); -} -extern int glue_ecb_req_128bit (const struct common_glue_ctx *gctx,struct skcipher_request *req);
@@ -100,7 +71,4 @@ extern int glue_cbc_encrypt_req_128bit(const common_glue_func_t fn,extern int glue_cbc_decrypt_req_128bit (const struct common_glue_ctx *gctx,struct skcipher_request *req); -extern int glue_ctr_req_128bit (const struct common_glue_ctx *gctx, -struct skcipher_request *req); -#endif /* _CRYPTO_GLUE_HELPER_H */
Signature · 1 line
2.17.1
PATCH v2 12/21 crypto: x86/des - drop CTR mode implementation
In reply to Ard BiesheuvelDES or Triple DES in counter mode is never used in the kernel, so there is no point in keeping an accelerated implementation around.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/des3_ede_glue.c | 104 -------------------- crypto/Kconfig | 1 + 2 files changed, 1 insertion(+), 104 deletions(-)
arch/x86/crypto/des3_ede_glue.c +0 -104
@@ -6,8 +6,6 @@* * CBC & ECB parts based on code (crypto/cbc.c,ecb.c) by: * Copyright (c) 2006 Herbert Xu <[email protected]> -* CTR part based on code (crypto/ctr.c) by: -* (C) Copyright IBM Corp. 2007 - Joy Latten <[email protected]> */ #include <crypto/algapi.h>
@@ -253,94 +251,6 @@ static int cbc_decrypt(struct skcipher_request *req)return err; } -static void ctr_crypt_final (struct des3_ede_x86_ctx *ctx, -struct skcipher_walk *walk) -{ -u8 *ctrblk = walk->iv; -u8 keystream[DES3_EDE_BLOCK_SIZE ]; -u8 *src = walk->src.virt.addr; -u8 *dst = walk->dst.virt.addr; -unsigned int nbytes = walk->nbytes; - -des3_ede_enc_blk (ctx, keystream, ctrblk); -crypto_xor_cpy (dst, keystream, src, nbytes); - -crypto_inc (ctrblk,DES3_EDE_BLOCK_SIZE ); -} - -static unsigned int __ctr_crypt (struct des3_ede_x86_ctx *ctx, -struct skcipher_walk *walk) -{ -unsigned int bsize =DES3_EDE_BLOCK_SIZE ; -unsigned int nbytes = walk->nbytes; -__be64 *src = (__be64 *)walk->src.virt.addr; -__be64 *dst = (__be64 *)walk->dst.virt.addr; -u64 ctrblk =be64_to_cpu (*(__be64 *)walk->iv); -__be64 ctrblocks[3 ]; - -/* Process four block batch */ -if (nbytes >= bsize *3 ) { -do { -/* create ctrblks for parallel encrypt */ - ctrblocks[0 ] =cpu_to_be64 (ctrblk++); - ctrblocks[1 ] =cpu_to_be64 (ctrblk++); - ctrblocks[2 ] =cpu_to_be64 (ctrblk++); - -des3_ede_enc_blk_3way (ctx, (u8 *)ctrblocks, - (u8 *)ctrblocks); - - dst[0 ] = src[0 ] ^ ctrblocks[0 ]; - dst[1 ] = src[1 ] ^ ctrblocks[1 ]; - dst[2 ] = src[2 ] ^ ctrblocks[2 ]; - - src +=3 ; - dst +=3 ; - }while ((nbytes -= bsize *3 ) >= bsize *3 ); - -if (nbytes < bsize) - goto done; - } - -/* Handle leftovers */ -do { - ctrblocks[0 ] =cpu_to_be64 (ctrblk++); - -des3_ede_enc_blk (ctx, (u8 *)ctrblocks, (u8 *)ctrblocks); - - dst[0 ] = src[0 ] ^ ctrblocks[0 ]; - - src +=1 ; - dst +=1 ; - }while ((nbytes -= bsize) >= bsize); - -done: - *(__be64 *)walk->iv =cpu_to_be64 (ctrblk); -return nbytes; -} - -static int ctr_crypt (struct skcipher_request *req) -{ -struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct des3_ede_x86_ctx *ctx =crypto_skcipher_ctx (tfm); -struct skcipher_walk walk; -unsigned int nbytes; -int err; - - err =skcipher_walk_virt (&walk, req, false); - -while ((nbytes = walk.nbytes) >=DES3_EDE_BLOCK_SIZE ) { - nbytes =__ctr_crypt (ctx, &walk); - err =skcipher_walk_done (&walk, nbytes); - } - -if (nbytes) { -ctr_crypt_final (ctx, &walk); - err =skcipher_walk_done (&walk,0 ); - } - -return err; -} -static int des3_ede_x86_setkey (struct crypto_tfm *tfm,const u8 *key,unsigned int keylen) {
@@ -428,20 +338,6 @@ static struct skcipher_alg des3_ede_skciphers[] = { .setkey = des3_ede_x86_setkey_skcipher, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, - }, { - .base.cra_name = "ctr(des3_ede)", - .base.cra_driver_name = "ctr-des3_ede-asm", - .base.cra_priority = 300, - .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct des3_ede_x86_ctx), - .base.cra_module = THIS_MODULE, - .min_keysize = DES3_EDE_KEY_SIZE, - .max_keysize = DES3_EDE_KEY_SIZE, - .ivsize = DES3_EDE_BLOCK_SIZE, - .chunksize = DES3_EDE_BLOCK_SIZE, - .setkey = des3_ede_x86_setkey_skcipher, - .encrypt = ctr_crypt, - .decrypt = ctr_crypt, } };
crypto/Kconfig +1 -0
@@ -1427,6 +1427,7 @@ config CRYPTO_DES3_EDE_X86_64 depends on X86 && 64BIT select CRYPTO_SKCIPHER select CRYPTO_LIB_DES + imply CRYPTO_CTR help Triple DES EDE (FIPS 46-3) algorithm.
Signature · 1 line
2.17.1
PATCH v2 13/21 crypto: x86/blowfish - drop CTR mode implementation
In reply to Ard BiesheuvelBlowfish in counter mode is never used in the kernel, so there is no point in keeping an accelerated implementation around.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/blowfish_glue.c | 107 -------------------- crypto/Kconfig | 1 + 2 files changed, 1 insertion(+), 107 deletions(-)
arch/x86/crypto/blowfish_glue.c +0 -107
@@ -6,8 +6,6 @@* * CBC & ECB parts based on code (crypto/cbc.c,ecb.c) by: * Copyright (c) 2006 Herbert Xu <[email protected]> -* CTR part based on code (crypto/ctr.c) by: -* (C) Copyright IBM Corp. 2007 - Joy Latten <[email protected]> */ #include <crypto/algapi.h>
@@ -247,97 +245,6 @@ static int cbc_decrypt(struct skcipher_request *req)return err; } -static void ctr_crypt_final (struct bf_ctx *ctx,struct skcipher_walk *walk) -{ -u8 *ctrblk = walk->iv; -u8 keystream[BF_BLOCK_SIZE ]; -u8 *src = walk->src.virt.addr; -u8 *dst = walk->dst.virt.addr; -unsigned int nbytes = walk->nbytes; - -blowfish_enc_blk (ctx, keystream, ctrblk); -crypto_xor_cpy (dst, keystream, src, nbytes); - -crypto_inc (ctrblk,BF_BLOCK_SIZE ); -} - -static unsigned int __ctr_crypt (struct bf_ctx *ctx,struct skcipher_walk *walk) -{ -unsigned int bsize =BF_BLOCK_SIZE ; -unsigned int nbytes = walk->nbytes; -u64 *src = (u64 *)walk->src.virt.addr; -u64 *dst = (u64 *)walk->dst.virt.addr; -u64 ctrblk =be64_to_cpu (*(__be64 *)walk->iv); -__be64 ctrblocks[4 ]; - -/* Process four block batch */ -if (nbytes >= bsize *4 ) { -do { -if (dst != src) { - dst[0 ] = src[0 ]; - dst[1 ] = src[1 ]; - dst[2 ] = src[2 ]; - dst[3 ] = src[3 ]; - } - -/* create ctrblks for parallel encrypt */ - ctrblocks[0 ] =cpu_to_be64 (ctrblk++); - ctrblocks[1 ] =cpu_to_be64 (ctrblk++); - ctrblocks[2 ] =cpu_to_be64 (ctrblk++); - ctrblocks[3 ] =cpu_to_be64 (ctrblk++); - -blowfish_enc_blk_xor_4way (ctx, (u8 *)dst, - (u8 *)ctrblocks); - - src +=4 ; - dst +=4 ; - }while ((nbytes -= bsize *4 ) >= bsize *4 ); - -if (nbytes < bsize) - goto done; - } - -/* Handle leftovers */ -do { -if (dst != src) - *dst = *src; - - ctrblocks[0 ] =cpu_to_be64 (ctrblk++); - -blowfish_enc_blk_xor (ctx, (u8 *)dst, (u8 *)ctrblocks); - - src +=1 ; - dst +=1 ; - }while ((nbytes -= bsize) >= bsize); - -done: - *(__be64 *)walk->iv =cpu_to_be64 (ctrblk); -return nbytes; -} - -static int ctr_crypt (struct skcipher_request *req) -{ -struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct bf_ctx *ctx =crypto_skcipher_ctx (tfm); -struct skcipher_walk walk; -unsigned int nbytes; -int err; - - err =skcipher_walk_virt (&walk, req, false); - -while ((nbytes = walk.nbytes) >=BF_BLOCK_SIZE ) { - nbytes =__ctr_crypt (ctx, &walk); - err =skcipher_walk_done (&walk, nbytes); - } - -if (nbytes) { -ctr_crypt_final (ctx, &walk); - err =skcipher_walk_done (&walk,0 ); - } - -return err; -} -static struct crypto_alg bf_cipher_alg = { .cra_name ="blowfish" , .cra_driver_name ="blowfish-asm" ,
@@ -384,20 +291,6 @@ static struct skcipher_alg bf_skcipher_algs[] = { .setkey = blowfish_setkey_skcipher, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, - }, { - .base.cra_name = "ctr(blowfish)", - .base.cra_driver_name = "ctr-blowfish-asm", - .base.cra_priority = 300, - .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct bf_ctx), - .base.cra_module = THIS_MODULE, - .min_keysize = BF_MIN_KEY_SIZE, - .max_keysize = BF_MAX_KEY_SIZE, - .ivsize = BF_BLOCK_SIZE, - .chunksize = BF_BLOCK_SIZE, - .setkey = blowfish_setkey_skcipher, - .encrypt = ctr_crypt, - .decrypt = ctr_crypt, }, };
crypto/Kconfig +1 -0
@@ -1255,6 +1255,7 @@ config CRYPTO_BLOWFISH_X86_64 depends on X86 && 64BIT select CRYPTO_SKCIPHER select CRYPTO_BLOWFISH_COMMON + imply CRYPTO_CTR help Blowfish cipher algorithm (x86_64), by Bruce Schneier.
Signature · 1 line
2.17.1
PATCH v2 14/21 crypto: x86 - add some helper macros for ECB and CBC modes
In reply to Ard BiesheuvelThe x86 glue helper module is starting to show its age:
- It relies heavily on function pointers to invoke asm helper functions that operate on fixed input sizes that are relatively small. This means the performance is severely impacted by retpolines.
- It goes to great lengths to amortize the cost of kernel_fpu_begin()/end() over as much work as possible, which is no longer necessary now that FPU save/restore is done lazily, and doing so may cause unbounded scheduling blackouts due to the fact that enabling the FPU in kernel mode disables preemption.
- The CBC mode decryption helper makes backward strides through the input, in order to avoid a single block size memcpy() between chunks. Consuming the input in this manner is highly likely to defeat any hardware prefetchers, so it is better to go through the data linearly, and perform the extra memcpy() where needed (which is turned into direct loads and stores by the compiler anyway). Note that benchmarks won't show this effect, given that the memory they use is always cache hot.
- It implements blockwise XOR in terms of le128 pointers, which imply an alignment that is not guaranteed by the API, violating the C standard.
GCC does not seem to be smart enough to elide the indirect calls when the function pointers are passed as arguments to static inline helper routines modeled after the existing ones. So instead, let's create some CPP macros that encapsulate the core of the ECB and CBC processing, so we can wire them up for existing users of the glue helper module, i.e., Camellia, Serpent, Twofish and CAST6.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/ecb_cbc_helpers.h | 76 ++++++++++++++++++++ 1 file changed, 76 insertions(+)
arch/x86/crypto/ecb_cbc_helpers.h new file +76 -0
@@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _CRYPTO_ECB_CBC_HELPER_H +#define _CRYPTO_ECB_CBC_HELPER_H + +#include <crypto/internal/skcipher.h> +#include <asm/fpu/api.h> + +/* +* Mode helpers to instantiate parameterized skcipher ECB/CBC modes without +* having to rely on indirect calls and retpolines. +*/ + +#define ECB_WALK_START (req, bsize, fpu_blocks) do { \ + void *ctx = crypto_skcipher_ctx(crypto_skcipher_reqtfm(req)); \ + const int __bsize = (bsize); \ + struct skcipher_walk walk; \ + int err = skcipher_walk_virt(&walk, (req), false); \ + while (walk.nbytes > 0) { \ + unsigned int nbytes = walk.nbytes; \ + bool do_fpu = (fpu_blocks) != -1 && \ + nbytes >= (fpu_blocks) * __bsize; \ + const u8 *src = walk.src.virt.addr; \ + u8 *dst = walk.dst.virt.addr; \ + u8 __maybe_unused buf[(bsize)]; \ + if (do_fpu) kernel_fpu_begin() + +#define CBC_WALK_START (req, bsize, fpu_blocks) \ + ECB_WALK_START(req, bsize, fpu_blocks) + +#define ECB_WALK_ADVANCE (blocks) do { \ + dst += (blocks) * __bsize; \ + src += (blocks) * __bsize; \ + nbytes -= (blocks) * __bsize; \ +} while (0) + +#define ECB_BLOCK (blocks, func) do { \ + while (nbytes >= (blocks) * __bsize) { \ + (func)(ctx, dst, src); \ + ECB_WALK_ADVANCE(blocks); \ + } \ +} while (0) + +#define CBC_ENC_BLOCK (func) do { \ + const u8 *__iv = walk.iv; \ + while (nbytes >= __bsize) { \ + crypto_xor_cpy(dst, src, __iv, __bsize); \ + (func)(ctx, dst, dst); \ + __iv = dst; \ + ECB_WALK_ADVANCE(1); \ + } \ + memcpy(walk.iv, __iv, __bsize); \ +} while (0) + +#define CBC_DEC_BLOCK (blocks, func) do { \ + while (nbytes >= (blocks) * __bsize) { \ + const u8 *__iv = src + ((blocks) - 1) * __bsize; \ + if (dst == src) \ + __iv = memcpy(buf, __iv, __bsize); \ + (func)(ctx, dst, src); \ + crypto_xor(dst, walk.iv, __bsize); \ + memcpy(walk.iv, __iv, __bsize); \ + ECB_WALK_ADVANCE(blocks); \ + } \ +} while (0) + +#define ECB_WALK_END () \ + if (do_fpu) kernel_fpu_end(); \ + err = skcipher_walk_done(&walk, nbytes); \ + } \ + return err; \ +} while (0) + +#define CBC_WALK_END () ECB_WALK_END() + +#endif
Signature · 1 line
2.17.1
PATCH v2 15/21 crypto: x86/camellia - drop dependency on glue helper
In reply to Ard BiesheuvelReplace the glue helper dependency with implementations of ECB and CBC based on the new CPP macros, which avoid the need for indirect calls.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/camellia_aesni_avx2_glue.c | 85 ++++++-------------- arch/x86/crypto/camellia_aesni_avx_glue.c | 73 +++++------------ arch/x86/crypto/camellia_glue.c | 75 ++++++----------- crypto/Kconfig | 2 - 4 files changed, 67 insertions(+), 168 deletions(-)
arch/x86/crypto/camellia_aesni_avx2_glue.c +23 -62
@@ -6,7 +6,6 @@*/ #include <asm/crypto/camellia.h> -#include <asm/crypto/glue_helper.h> #include <crypto/algapi.h> #include <crypto/internal/simd.h> #include <linux/crypto.h>
@@ -14,6 +13,8 @@#include <linux/module.h> #include <linux/types.h> +#include "ecb_cbc_helpers.h" +#define CAMELLIA_AESNI_PARALLEL_BLOCKS 16#define CAMELLIA_AESNI_AVX2_PARALLEL_BLOCKS 32
@@ -23,63 +24,6 @@ asmlinkage void camellia_ecb_dec_32way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void camellia_cbc_dec_32way(const void *ctx, u8 *dst, const u8 *src); -static const struct common_glue_ctx camellia_enc = { - .num_funcs =4 , - .fpu_blocks_limit =CAMELLIA_AESNI_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =CAMELLIA_AESNI_AVX2_PARALLEL_BLOCKS , - .fn_u = { .ecb = camellia_ecb_enc_32way } - }, { - .num_blocks =CAMELLIA_AESNI_PARALLEL_BLOCKS , - .fn_u = { .ecb = camellia_ecb_enc_16way } - }, { - .num_blocks =2 , - .fn_u = { .ecb = camellia_enc_blk_2way } - }, { - .num_blocks =1 , - .fn_u = { .ecb = camellia_enc_blk } - } } -}; - -static const struct common_glue_ctx camellia_dec = { - .num_funcs =4 , - .fpu_blocks_limit =CAMELLIA_AESNI_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =CAMELLIA_AESNI_AVX2_PARALLEL_BLOCKS , - .fn_u = { .ecb = camellia_ecb_dec_32way } - }, { - .num_blocks =CAMELLIA_AESNI_PARALLEL_BLOCKS , - .fn_u = { .ecb = camellia_ecb_dec_16way } - }, { - .num_blocks =2 , - .fn_u = { .ecb = camellia_dec_blk_2way } - }, { - .num_blocks =1 , - .fn_u = { .ecb = camellia_dec_blk } - } } -}; - -static const struct common_glue_ctx camellia_dec_cbc = { - .num_funcs =4 , - .fpu_blocks_limit =CAMELLIA_AESNI_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =CAMELLIA_AESNI_AVX2_PARALLEL_BLOCKS , - .fn_u = { .cbc = camellia_cbc_dec_32way } - }, { - .num_blocks =CAMELLIA_AESNI_PARALLEL_BLOCKS , - .fn_u = { .cbc = camellia_cbc_dec_16way } - }, { - .num_blocks =2 , - .fn_u = { .cbc = camellia_decrypt_cbc_2way } - }, { - .num_blocks =1 , - .fn_u = { .cbc = camellia_dec_blk } - } } -}; -static int camellia_setkey (struct crypto_skcipher *tfm,const u8 *key,unsigned int keylen) {
@@ -88,22 +32,39 @@ static int camellia_setkey(struct crypto_skcipher *tfm, const u8 *key,static int ecb_encrypt (struct skcipher_request *req) { -return glue_ecb_req_128bit (&camellia_enc, req); +ECB_WALK_START (req,CAMELLIA_BLOCK_SIZE ,CAMELLIA_AESNI_PARALLEL_BLOCKS ); +ECB_BLOCK (CAMELLIA_AESNI_AVX2_PARALLEL_BLOCKS , camellia_ecb_enc_32way); +ECB_BLOCK (CAMELLIA_AESNI_PARALLEL_BLOCKS , camellia_ecb_enc_16way); +ECB_BLOCK (2 , camellia_enc_blk_2way); +ECB_BLOCK (1 , camellia_enc_blk); +ECB_WALK_END (); }static int ecb_decrypt (struct skcipher_request *req) { -return glue_ecb_req_128bit (&camellia_dec, req); +ECB_WALK_START (req,CAMELLIA_BLOCK_SIZE ,CAMELLIA_AESNI_PARALLEL_BLOCKS ); +ECB_BLOCK (CAMELLIA_AESNI_AVX2_PARALLEL_BLOCKS , camellia_ecb_dec_32way); +ECB_BLOCK (CAMELLIA_AESNI_PARALLEL_BLOCKS , camellia_ecb_dec_16way); +ECB_BLOCK (2 , camellia_dec_blk_2way); +ECB_BLOCK (1 , camellia_dec_blk); +ECB_WALK_END (); }static int cbc_encrypt (struct skcipher_request *req) { -return glue_cbc_encrypt_req_128bit (camellia_enc_blk, req); +CBC_WALK_START (req,CAMELLIA_BLOCK_SIZE ,-1 ); +CBC_ENC_BLOCK (camellia_enc_blk); +CBC_WALK_END (); }static int cbc_decrypt (struct skcipher_request *req) { -return glue_cbc_decrypt_req_128bit (&camellia_dec_cbc, req); +CBC_WALK_START (req,CAMELLIA_BLOCK_SIZE ,CAMELLIA_AESNI_PARALLEL_BLOCKS ); +CBC_DEC_BLOCK (CAMELLIA_AESNI_AVX2_PARALLEL_BLOCKS , camellia_cbc_dec_32way); +CBC_DEC_BLOCK (CAMELLIA_AESNI_PARALLEL_BLOCKS , camellia_cbc_dec_16way); +CBC_DEC_BLOCK (2 , camellia_decrypt_cbc_2way); +CBC_DEC_BLOCK (1 , camellia_dec_blk); +CBC_WALK_END (); }static struct skcipher_alg camellia_algs[] = {
arch/x86/crypto/camellia_aesni_avx_glue.c +20 -53
109 lines not on this page · raw message
arch/x86/crypto/camellia_glue.c +24 -51
106 lines not on this page · raw message
crypto/Kconfig +0 -2
@@ -1286,7 +1286,6 @@ config CRYPTO_CAMELLIA_X86_64 depends on X86 && 64BIT depends on CRYPTO select CRYPTO_SKCIPHER - select CRYPTO_GLUE_HELPER_X86 imply CRYPTO_CTR help Camellia cipher algorithm module (x86_64).
@@ -1305,7 +1304,6 @@ config CRYPTO_CAMELLIA_AESNI_AVX_X86_64 depends on CRYPTO select CRYPTO_SKCIPHER select CRYPTO_CAMELLIA_X86_64 - select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SIMD imply CRYPTO_XTS help
Signature · 1 line
2.17.1
PATCH v2 16/21 crypto: x86/serpent - drop dependency on glue helper
In reply to Ard BiesheuvelReplace the glue helper dependency with implementations of ECB and CBC based on the new CPP macros, which avoid the need for indirect calls.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/serpent_avx2_glue.c | 73 +++++------------- arch/x86/crypto/serpent_avx_glue.c | 61 ++++----------- arch/x86/crypto/serpent_sse2_glue.c | 81 ++++++-------------- crypto/Kconfig | 3 - 4 files changed, 61 insertions(+), 157 deletions(-)
arch/x86/crypto/serpent_avx2_glue.c +20 -53
@@ -12,9 +12,10 @@#include <crypto/algapi.h> #include <crypto/internal/simd.h> #include <crypto/serpent.h> -#include <asm/crypto/glue_helper.h> #include <asm/crypto/serpent-avx.h> +#include "ecb_cbc_helpers.h" +#define SERPENT_AVX2_PARALLEL_BLOCKS 16/* 16-way AVX2 parallel cipher functions */
@@ -28,72 +29,38 @@ static int serpent_setkey_skcipher(struct crypto_skcipher *tfm,return __serpent_setkey (crypto_skcipher_ctx (tfm), key, keylen); } -static const struct common_glue_ctx serpent_enc = { - .num_funcs =3 , - .fpu_blocks_limit =8 , - - .funcs = { { - .num_blocks =16 , - .fn_u = { .ecb = serpent_ecb_enc_16way } - }, { - .num_blocks =8 , - .fn_u = { .ecb = serpent_ecb_enc_8way_avx } - }, { - .num_blocks =1 , - .fn_u = { .ecb = __serpent_encrypt } - } } -}; - -static const struct common_glue_ctx serpent_dec = { - .num_funcs =3 , - .fpu_blocks_limit =8 , - - .funcs = { { - .num_blocks =16 , - .fn_u = { .ecb = serpent_ecb_dec_16way } - }, { - .num_blocks =8 , - .fn_u = { .ecb = serpent_ecb_dec_8way_avx } - }, { - .num_blocks =1 , - .fn_u = { .ecb = __serpent_decrypt } - } } -}; - -static const struct common_glue_ctx serpent_dec_cbc = { - .num_funcs =3 , - .fpu_blocks_limit =8 , - - .funcs = { { - .num_blocks =16 , - .fn_u = { .cbc = serpent_cbc_dec_16way } - }, { - .num_blocks =8 , - .fn_u = { .cbc = serpent_cbc_dec_8way_avx } - }, { - .num_blocks =1 , - .fn_u = { .cbc = __serpent_decrypt } - } } -}; -static int ecb_encrypt (struct skcipher_request *req) { -return glue_ecb_req_128bit (&serpent_enc, req); +ECB_WALK_START (req,SERPENT_BLOCK_SIZE ,SERPENT_PARALLEL_BLOCKS ); +ECB_BLOCK (SERPENT_AVX2_PARALLEL_BLOCKS , serpent_ecb_enc_16way); +ECB_BLOCK (SERPENT_PARALLEL_BLOCKS , serpent_ecb_enc_8way_avx); +ECB_BLOCK (1 , __serpent_encrypt); +ECB_WALK_END (); }static int ecb_decrypt (struct skcipher_request *req) { -return glue_ecb_req_128bit (&serpent_dec, req); +ECB_WALK_START (req,SERPENT_BLOCK_SIZE ,SERPENT_PARALLEL_BLOCKS ); +ECB_BLOCK (SERPENT_AVX2_PARALLEL_BLOCKS , serpent_ecb_dec_16way); +ECB_BLOCK (SERPENT_PARALLEL_BLOCKS , serpent_ecb_dec_8way_avx); +ECB_BLOCK (1 , __serpent_decrypt); +ECB_WALK_END (); }static int cbc_encrypt (struct skcipher_request *req) { -return glue_cbc_encrypt_req_128bit (__serpent_encrypt, req); +CBC_WALK_START (req,SERPENT_BLOCK_SIZE ,-1 ); +CBC_ENC_BLOCK (__serpent_encrypt); +CBC_WALK_END (); }static int cbc_decrypt (struct skcipher_request *req) { -return glue_cbc_decrypt_req_128bit (&serpent_dec_cbc, req); +CBC_WALK_START (req,SERPENT_BLOCK_SIZE ,SERPENT_PARALLEL_BLOCKS ); +CBC_DEC_BLOCK (SERPENT_AVX2_PARALLEL_BLOCKS , serpent_cbc_dec_16way); +CBC_DEC_BLOCK (SERPENT_PARALLEL_BLOCKS , serpent_cbc_dec_8way_avx); +CBC_DEC_BLOCK (1 , __serpent_decrypt); +CBC_WALK_END (); }static struct skcipher_alg serpent_algs[] = {
arch/x86/crypto/serpent_avx_glue.c +17 -44
89 lines not on this page · raw message
arch/x86/crypto/serpent_sse2_glue.c +24 -57
111 lines not on this page · raw message
crypto/Kconfig +0 -3
@@ -1538,7 +1538,6 @@ config CRYPTO_SERPENT_SSE2_X86_64 tristate "Serpent cipher algorithm (x86_64/SSE2)" depends on X86 && 64BIT select CRYPTO_SKCIPHER - select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SERPENT select CRYPTO_SIMD imply CRYPTO_CTR
@@ -1558,7 +1557,6 @@ config CRYPTO_SERPENT_SSE2_586 tristate "Serpent cipher algorithm (i586/SSE2)" depends on X86 && !64BIT select CRYPTO_SKCIPHER - select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SERPENT select CRYPTO_SIMD imply CRYPTO_CTR
@@ -1578,7 +1576,6 @@ config CRYPTO_SERPENT_AVX_X86_64 tristate "Serpent cipher algorithm (x86_64/AVX)" depends on X86 && 64BIT select CRYPTO_SKCIPHER - select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SERPENT select CRYPTO_SIMD imply CRYPTO_XTS
Signature · 1 line
2.17.1
PATCH v2 17/21 crypto: x86/cast5 - drop dependency on glue helper
In reply to Ard BiesheuvelReplace the glue helper dependency with implementations of ECB and CBC based on the new CPP macros, which avoid the need for indirect calls.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/cast5_avx_glue.c | 184 ++------------------ 1 file changed, 17 insertions(+), 167 deletions(-)
arch/x86/crypto/cast5_avx_glue.c +17 -167
@@ -6,7 +6,6 @@* <[email protected]> */ -#include <asm/crypto/glue_helper.h> #include <crypto/algapi.h> #include <crypto/cast5.h> #include <crypto/internal/simd.h>
@@ -15,6 +14,8 @@#include <linux/module.h> #include <linux/types.h> +#include "ecb_cbc_helpers.h" +#define CAST5_PARALLEL_BLOCKS 16 asmlinkage void cast5_ecb_enc_16way(struct cast5_ctx *ctx, u8 *dst,
@@ -30,186 +31,35 @@ static int cast5_setkey_skcipher(struct crypto_skcipher *tfm, const u8 *key,return cast5_setkey (&tfm->base, key, keylen); } -static inline bool cast5_fpu_begin (bool fpu_enabled,struct skcipher_walk *walk, -unsigned int nbytes) -{ -return glue_fpu_begin (CAST5_BLOCK_SIZE ,CAST5_PARALLEL_BLOCKS , - walk, fpu_enabled, nbytes); -} - -static inline void cast5_fpu_end (bool fpu_enabled) -{ -return glue_fpu_end (fpu_enabled); -} - -static int ecb_crypt (struct skcipher_request *req,bool enc) -{ -bool fpu_enabled = false; -struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct cast5_ctx *ctx =crypto_skcipher_ctx (tfm); -struct skcipher_walk walk; -const unsigned int bsize =CAST5_BLOCK_SIZE ; -unsigned int nbytes; -void (*fn)(struct cast5_ctx *ctx,u8 *dst,const u8 *src); -int err; - - err =skcipher_walk_virt (&walk, req, false); - -while ((nbytes = walk.nbytes)) { -u8 *wsrc = walk.src.virt.addr; -u8 *wdst = walk.dst.virt.addr; - - fpu_enabled =cast5_fpu_begin (fpu_enabled, &walk, nbytes); - -/* Process multi-block batch */ -if (nbytes >= bsize *CAST5_PARALLEL_BLOCKS ) { - fn = (enc) ? cast5_ecb_enc_16way : cast5_ecb_dec_16way; -do { -fn (ctx, wdst, wsrc); - - wsrc += bsize *CAST5_PARALLEL_BLOCKS ; - wdst += bsize *CAST5_PARALLEL_BLOCKS ; - nbytes -= bsize *CAST5_PARALLEL_BLOCKS ; - }while (nbytes >= bsize *CAST5_PARALLEL_BLOCKS ); - -if (nbytes < bsize) - goto done; - } - - fn = (enc) ? __cast5_encrypt : __cast5_decrypt; - -/* Handle leftovers */ -do { -fn (ctx, wdst, wsrc); - - wsrc += bsize; - wdst += bsize; - nbytes -= bsize; - }while (nbytes >= bsize); - -done: - err =skcipher_walk_done (&walk, nbytes); - } - -cast5_fpu_end (fpu_enabled); -return err; -} -static int ecb_encrypt (struct skcipher_request *req) { -return ecb_crypt (req, true); +ECB_WALK_START (req,CAST5_BLOCK_SIZE ,CAST5_PARALLEL_BLOCKS ); +ECB_BLOCK (CAST5_PARALLEL_BLOCKS , cast5_ecb_enc_16way); +ECB_BLOCK (1 , __cast5_encrypt); +ECB_WALK_END (); }static int ecb_decrypt (struct skcipher_request *req) { -return ecb_crypt (req, false); +ECB_WALK_START (req,CAST5_BLOCK_SIZE ,CAST5_PARALLEL_BLOCKS ); +ECB_BLOCK (CAST5_PARALLEL_BLOCKS , cast5_ecb_dec_16way); +ECB_BLOCK (1 , __cast5_decrypt); +ECB_WALK_END (); }static int cbc_encrypt (struct skcipher_request *req) { -const unsigned int bsize =CAST5_BLOCK_SIZE ; -struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct cast5_ctx *ctx =crypto_skcipher_ctx (tfm); -struct skcipher_walk walk; -unsigned int nbytes; -int err; - - err =skcipher_walk_virt (&walk, req, false); - -while ((nbytes = walk.nbytes)) { -u64 *src = (u64 *)walk.src.virt.addr; -u64 *dst = (u64 *)walk.dst.virt.addr; -u64 *iv = (u64 *)walk.iv; - -do { - *dst = *src ^ *iv; -__cast5_encrypt (ctx, (u8 *)dst, (u8 *)dst); - iv = dst; - src++; - dst++; - nbytes -= bsize; - }while (nbytes >= bsize); - - *(u64 *)walk.iv = *iv; - err =skcipher_walk_done (&walk, nbytes); - } - -return err; -} - -static unsigned int __cbc_decrypt (struct cast5_ctx *ctx, -struct skcipher_walk *walk) -{ -const unsigned int bsize =CAST5_BLOCK_SIZE ; -unsigned int nbytes = walk->nbytes; -u64 *src = (u64 *)walk->src.virt.addr; -u64 *dst = (u64 *)walk->dst.virt.addr; -u64 last_iv; - -/* Start of the last block. */ - src += nbytes / bsize -1 ; - dst += nbytes / bsize -1 ; - - last_iv = *src; - -/* Process multi-block batch */ -if (nbytes >= bsize *CAST5_PARALLEL_BLOCKS ) { -do { - nbytes -= bsize * (CAST5_PARALLEL_BLOCKS -1 ); - src -=CAST5_PARALLEL_BLOCKS -1 ; - dst -=CAST5_PARALLEL_BLOCKS -1 ; - -cast5_cbc_dec_16way (ctx, (u8 *)dst, (u8 *)src); - - nbytes -= bsize; -if (nbytes < bsize) - goto done; - - *dst ^= *(src -1 ); - src -=1 ; - dst -=1 ; - }while (nbytes >= bsize *CAST5_PARALLEL_BLOCKS ); - } - -/* Handle leftovers */ -for (;;) { -__cast5_decrypt (ctx, (u8 *)dst, (u8 *)src); - - nbytes -= bsize; -if (nbytes < bsize) -break ; - - *dst ^= *(src -1 ); - src -=1 ; - dst -=1 ; - } - -done: - *dst ^= *(u64 *)walk->iv; - *(u64 *)walk->iv = last_iv; - -return nbytes; +CBC_WALK_START (req,CAST5_BLOCK_SIZE ,-1 ); +CBC_ENC_BLOCK (__cast5_encrypt); +CBC_WALK_END (); }static int cbc_decrypt (struct skcipher_request *req) { -struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct cast5_ctx *ctx =crypto_skcipher_ctx (tfm); -bool fpu_enabled = false; -struct skcipher_walk walk; -unsigned int nbytes; -int err; - - err =skcipher_walk_virt (&walk, req, false); - -while ((nbytes = walk.nbytes)) { - fpu_enabled =cast5_fpu_begin (fpu_enabled, &walk, nbytes); - nbytes =__cbc_decrypt (ctx, &walk); - err =skcipher_walk_done (&walk, nbytes); - } - -cast5_fpu_end (fpu_enabled); -return err; +CBC_WALK_START (req,CAST5_BLOCK_SIZE ,CAST5_PARALLEL_BLOCKS ); +CBC_DEC_BLOCK (CAST5_PARALLEL_BLOCKS , cast5_cbc_dec_16way); +CBC_DEC_BLOCK (1 , __cast5_decrypt); +CBC_WALK_END (); }static struct skcipher_alg cast5_algs[] = {
Signature · 1 line
2.17.1
PATCH v2 18/21 crypto: x86/cast6 - drop dependency on glue helper
In reply to Ard BiesheuvelReplace the glue helper dependency with implementations of ECB and CBC based on the new CPP macros, which avoid the need for indirect calls.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/cast6_avx_glue.c | 61 ++++++-------------- crypto/Kconfig | 1 - 2 files changed, 17 insertions(+), 45 deletions(-)
arch/x86/crypto/cast6_avx_glue.c +17 -44
@@ -15,7 +15,8 @@#include <crypto/algapi.h> #include <crypto/cast6.h> #include <crypto/internal/simd.h> -#include <asm/crypto/glue_helper.h> + +#include "ecb_cbc_helpers.h" #define CAST6_PARALLEL_BLOCKS 8
@@ -30,63 +31,35 @@ static int cast6_setkey_skcipher(struct crypto_skcipher *tfm,return cast6_setkey (&tfm->base, key, keylen); } -static const struct common_glue_ctx cast6_enc = { - .num_funcs =2 , - .fpu_blocks_limit =CAST6_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =CAST6_PARALLEL_BLOCKS , - .fn_u = { .ecb = cast6_ecb_enc_8way } - }, { - .num_blocks =1 , - .fn_u = { .ecb = __cast6_encrypt } - } } -}; - -static const struct common_glue_ctx cast6_dec = { - .num_funcs =2 , - .fpu_blocks_limit =CAST6_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =CAST6_PARALLEL_BLOCKS , - .fn_u = { .ecb = cast6_ecb_dec_8way } - }, { - .num_blocks =1 , - .fn_u = { .ecb = __cast6_decrypt } - } } -}; - -static const struct common_glue_ctx cast6_dec_cbc = { - .num_funcs =2 , - .fpu_blocks_limit =CAST6_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =CAST6_PARALLEL_BLOCKS , - .fn_u = { .cbc = cast6_cbc_dec_8way } - }, { - .num_blocks =1 , - .fn_u = { .cbc = __cast6_decrypt } - } } -}; -static int ecb_encrypt (struct skcipher_request *req) { -return glue_ecb_req_128bit (&cast6_enc, req); +ECB_WALK_START (req,CAST6_BLOCK_SIZE ,CAST6_PARALLEL_BLOCKS ); +ECB_BLOCK (CAST6_PARALLEL_BLOCKS , cast6_ecb_enc_8way); +ECB_BLOCK (1 , __cast6_encrypt); +ECB_WALK_END (); }static int ecb_decrypt (struct skcipher_request *req) { -return glue_ecb_req_128bit (&cast6_dec, req); +ECB_WALK_START (req,CAST6_BLOCK_SIZE ,CAST6_PARALLEL_BLOCKS ); +ECB_BLOCK (CAST6_PARALLEL_BLOCKS , cast6_ecb_dec_8way); +ECB_BLOCK (1 , __cast6_decrypt); +ECB_WALK_END (); }static int cbc_encrypt (struct skcipher_request *req) { -return glue_cbc_encrypt_req_128bit (__cast6_encrypt, req); +CBC_WALK_START (req,CAST6_BLOCK_SIZE ,-1 ); +CBC_ENC_BLOCK (__cast6_encrypt); +CBC_WALK_END (); }static int cbc_decrypt (struct skcipher_request *req) { -return glue_cbc_decrypt_req_128bit (&cast6_dec_cbc, req); +CBC_WALK_START (req,CAST6_BLOCK_SIZE ,CAST6_PARALLEL_BLOCKS ); +CBC_DEC_BLOCK (CAST6_PARALLEL_BLOCKS , cast6_cbc_dec_8way); +CBC_DEC_BLOCK (1 , __cast6_decrypt); +CBC_WALK_END (); }static struct skcipher_alg cast6_algs[] = {
crypto/Kconfig +0 -1
@@ -1393,7 +1393,6 @@ config CRYPTO_CAST6_AVX_X86_64 select CRYPTO_SKCIPHER select CRYPTO_CAST6 select CRYPTO_CAST_COMMON - select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SIMD imply CRYPTO_XTS imply CRYPTO_CTR
Signature · 1 line
2.17.1
PATCH v2 19/21 crypto: x86/twofish - drop dependency on glue helper
In reply to Ard BiesheuvelReplace the glue helper dependency with implementations of ECB and CBC based on the new CPP macros, which avoid the need for indirect calls.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/twofish_avx_glue.c | 73 +++++------------- arch/x86/crypto/twofish_glue_3way.c | 80 ++++++-------------- crypto/Kconfig | 2 - 3 files changed, 44 insertions(+), 111 deletions(-)
arch/x86/crypto/twofish_avx_glue.c +20 -53
@@ -15,9 +15,10 @@#include <crypto/algapi.h> #include <crypto/internal/simd.h> #include <crypto/twofish.h> -#include <asm/crypto/glue_helper.h> #include <asm/crypto/twofish.h> +#include "ecb_cbc_helpers.h" +#define TWOFISH_PARALLEL_BLOCKS 8/* 8-way parallel cipher functions */
@@ -37,72 +38,38 @@ static inline void twofish_enc_blk_3way(const void *ctx, u8 *dst, const u8 *src)__twofish_enc_blk_3way (ctx, dst, src, false); } -static const struct common_glue_ctx twofish_enc = { - .num_funcs =3 , - .fpu_blocks_limit =TWOFISH_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =TWOFISH_PARALLEL_BLOCKS , - .fn_u = { .ecb = twofish_ecb_enc_8way } - }, { - .num_blocks =3 , - .fn_u = { .ecb = twofish_enc_blk_3way } - }, { - .num_blocks =1 , - .fn_u = { .ecb = twofish_enc_blk } - } } -}; - -static const struct common_glue_ctx twofish_dec = { - .num_funcs =3 , - .fpu_blocks_limit =TWOFISH_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =TWOFISH_PARALLEL_BLOCKS , - .fn_u = { .ecb = twofish_ecb_dec_8way } - }, { - .num_blocks =3 , - .fn_u = { .ecb = twofish_dec_blk_3way } - }, { - .num_blocks =1 , - .fn_u = { .ecb = twofish_dec_blk } - } } -}; - -static const struct common_glue_ctx twofish_dec_cbc = { - .num_funcs =3 , - .fpu_blocks_limit =TWOFISH_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =TWOFISH_PARALLEL_BLOCKS , - .fn_u = { .cbc = twofish_cbc_dec_8way } - }, { - .num_blocks =3 , - .fn_u = { .cbc = twofish_dec_blk_cbc_3way } - }, { - .num_blocks =1 , - .fn_u = { .cbc = twofish_dec_blk } - } } -}; -static int ecb_encrypt (struct skcipher_request *req) { -return glue_ecb_req_128bit (&twofish_enc, req); +ECB_WALK_START (req,TF_BLOCK_SIZE ,TWOFISH_PARALLEL_BLOCKS ); +ECB_BLOCK (TWOFISH_PARALLEL_BLOCKS , twofish_ecb_enc_8way); +ECB_BLOCK (3 , twofish_enc_blk_3way); +ECB_BLOCK (1 , twofish_enc_blk); +ECB_WALK_END (); }static int ecb_decrypt (struct skcipher_request *req) { -return glue_ecb_req_128bit (&twofish_dec, req); +ECB_WALK_START (req,TF_BLOCK_SIZE ,TWOFISH_PARALLEL_BLOCKS ); +ECB_BLOCK (TWOFISH_PARALLEL_BLOCKS , twofish_ecb_dec_8way); +ECB_BLOCK (3 , twofish_dec_blk_3way); +ECB_BLOCK (1 , twofish_dec_blk); +ECB_WALK_END (); }static int cbc_encrypt (struct skcipher_request *req) { -return glue_cbc_encrypt_req_128bit (twofish_enc_blk, req); +CBC_WALK_START (req,TF_BLOCK_SIZE ,-1 ); +CBC_ENC_BLOCK (twofish_enc_blk); +CBC_WALK_END (); }static int cbc_decrypt (struct skcipher_request *req) { -return glue_cbc_decrypt_req_128bit (&twofish_dec_cbc, req); +CBC_WALK_START (req,TF_BLOCK_SIZE ,TWOFISH_PARALLEL_BLOCKS ); +CBC_DEC_BLOCK (TWOFISH_PARALLEL_BLOCKS , twofish_cbc_dec_8way); +CBC_DEC_BLOCK (3 , twofish_dec_blk_cbc_3way); +CBC_DEC_BLOCK (1 , twofish_dec_blk); +CBC_WALK_END (); }static struct skcipher_alg twofish_algs[] = {
arch/x86/crypto/twofish_glue_3way.c +24 -56
120 lines not on this page · raw message
crypto/Kconfig +0 -2
@@ -1711,7 +1711,6 @@ config CRYPTO_TWOFISH_X86_64_3WAY select CRYPTO_SKCIPHER select CRYPTO_TWOFISH_COMMON select CRYPTO_TWOFISH_X86_64 - select CRYPTO_GLUE_HELPER_X86 help Twofish cipher algorithm (x86_64, 3-way parallel).
@@ -1730,7 +1729,6 @@ config CRYPTO_TWOFISH_AVX_X86_64 tristate "Twofish cipher algorithm (x86_64/AVX)" depends on X86 && 64BIT select CRYPTO_SKCIPHER - select CRYPTO_GLUE_HELPER_X86 select CRYPTO_SIMD select CRYPTO_TWOFISH_COMMON select CRYPTO_TWOFISH_X86_64
Signature · 1 line
2.17.1
PATCH v2 20/21 crypto: x86 - remove glue helper module
In reply to Ard BiesheuvelAll dependencies on the x86 glue helper module have been replaced by local instantiations of the new ECB/CBC preprocessor helper macros, so the glue helper module can be retired.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
--- arch/x86/crypto/Makefile | 2 - arch/x86/crypto/glue_helper.c | 155 -------------------- arch/x86/include/asm/crypto/glue_helper.h | 74 ---------- crypto/Kconfig | 5 - crypto/skcipher.c | 6 - include/crypto/internal/skcipher.h | 1 - 6 files changed, 243 deletions(-)
arch/x86/crypto/Makefile +0 -2
@@ -4,8 +4,6 @@ OBJECT_FILES_NON_STANDARD := y -obj-$(CONFIG_CRYPTO_GLUE_HELPER_X86) += glue_helper.o - obj-$(CONFIG_CRYPTO_TWOFISH_586) += twofish-i586.o twofish-i586-y := twofish-i586-asm_32.o twofish_glue.o obj-$(CONFIG_CRYPTO_TWOFISH_X86_64) += twofish-x86_64.o
arch/x86/crypto/glue_helper.c deleted +0 -155
155 lines not on this page · raw message
arch/x86/include/asm/crypto/glue_helper.h deleted +0 -74
74 lines not on this page · raw message
crypto/Kconfig +0 -5
@@ -210,11 +210,6 @@ config CRYPTO_SIMD tristate select CRYPTO_CRYPTD -config CRYPTO_GLUE_HELPER_X86 - tristate - depends on X86 - select CRYPTO_SKCIPHER - config CRYPTO_ENGINE tristate
crypto/skcipher.c +0 -6
@@ -491,12 +491,6 @@ int skcipher_walk_virt(struct skcipher_walk *walk, }EXPORT_SYMBOL_GPL (skcipher_walk_virt); -void skcipher_walk_atomise (struct skcipher_walk *walk) -{ - walk->flags &= ~SKCIPHER_WALK_SLEEP ; -} -EXPORT_SYMBOL_GPL (skcipher_walk_atomise); -int skcipher_walk_async (struct skcipher_walk *walk,struct skcipher_request *req) {
include/crypto/internal/skcipher.h +0 -1
@@ -133,7 +133,6 @@ int skcipher_walk_done(struct skcipher_walk *walk, int err);int skcipher_walk_virt (struct skcipher_walk *walk,struct skcipher_request *req,bool atomic); -void skcipher_walk_atomise (struct skcipher_walk *walk);int skcipher_walk_async (struct skcipher_walk *walk,struct skcipher_request *req); int skcipher_walk_aead_encrypt(struct skcipher_walk *walk,
Signature · 1 line
2.17.1
PATCH v2 21/21 crypto: x86 - use local headers for x86 specific shared declarations
In reply to Ard BiesheuvelThe Camellia, Serpent and Twofish related header files only contain declarations that are shared between different implementations of the respective algorithms residing under arch/x86/crypto, and none of their contents should be used elsewhere. So move the header files into the same location, and use local #includes instead.
Acked-by Eric Biggers <[email protected]>
Signed-off-by Ard Biesheuvel <[email protected]>
---
arch/x86/{include/asm => }/crypto/camellia.h | 0
arch/x86/crypto/camellia_aesni_avx2_glue.c | 2 +-
arch/x86/crypto/camellia_aesni_avx_glue.c | 2 +-
arch/x86/crypto/camellia_glue.c | 2 +-
arch/x86/{include/asm => }/crypto/serpent-avx.h | 0
arch/x86/{include/asm => }/crypto/serpent-sse2.h | 0
arch/x86/crypto/serpent_avx2_glue.c | 2 +-
arch/x86/crypto/serpent_avx_glue.c | 2 +-
arch/x86/crypto/serpent_sse2_glue.c | 2 +-
arch/x86/{include/asm => }/crypto/twofish.h | 0
arch/x86/crypto/twofish_avx_glue.c | 2 +-
arch/x86/crypto/twofish_glue_3way.c | 2 +-
12 files changed, 8 insertions(+), 8 deletions(-)arch/x86/crypto/camellia.h renamed from arch/x86/include/asm/crypto/camellia.h +0 -0
arch/x86/crypto/camellia_aesni_avx2_glue.c +1 -1
@@ -5,7 +5,6 @@* Copyright © 2013 Jussi Kivilinna <[email protected]> */ -#include <asm/crypto/camellia.h> #include <crypto/algapi.h> #include <crypto/internal/simd.h> #include <linux/crypto.h>
@@ -13,6 +12,7 @@#include <linux/module.h> #include <linux/types.h> +#include "camellia.h" #include "ecb_cbc_helpers.h" #define CAMELLIA_AESNI_PARALLEL_BLOCKS 16
arch/x86/crypto/camellia_aesni_avx_glue.c +1 -1
@@ -5,7 +5,6 @@* Copyright © 2012-2013 Jussi Kivilinna <[email protected]> */ -#include <asm/crypto/camellia.h> #include <crypto/algapi.h> #include <crypto/internal/simd.h> #include <linux/crypto.h>
@@ -13,6 +12,7 @@#include <linux/module.h> #include <linux/types.h> +#include "camellia.h" #include "ecb_cbc_helpers.h" #define CAMELLIA_AESNI_PARALLEL_BLOCKS 16
arch/x86/crypto/camellia_glue.c +1 -1
@@ -14,8 +14,8 @@#include <linux/module.h> #include <linux/types.h> #include <crypto/algapi.h> -#include <asm/crypto/camellia.h> +#include "camellia.h" #include "ecb_cbc_helpers.h" /* regular block cipher functions */
arch/x86/crypto/serpent-avx.h renamed from arch/x86/include/asm/crypto/serpent-avx.h +0 -0
arch/x86/crypto/serpent-sse2.h renamed from arch/x86/include/asm/crypto/serpent-sse2.h +0 -0
arch/x86/crypto/serpent_avx2_glue.c +1 -1
@@ -12,8 +12,8 @@#include <crypto/algapi.h> #include <crypto/internal/simd.h> #include <crypto/serpent.h> -#include <asm/crypto/serpent-avx.h> +#include "serpent-avx.h" #include "ecb_cbc_helpers.h" #define SERPENT_AVX2_PARALLEL_BLOCKS 16
arch/x86/crypto/serpent_avx_glue.c +1 -1
@@ -15,8 +15,8 @@#include <crypto/algapi.h> #include <crypto/internal/simd.h> #include <crypto/serpent.h> -#include <asm/crypto/serpent-avx.h> +#include "serpent-avx.h" #include "ecb_cbc_helpers.h" /* 8-way parallel cipher functions */
arch/x86/crypto/serpent_sse2_glue.c +1 -1
@@ -20,8 +20,8 @@#include <crypto/b128ops.h> #include <crypto/internal/simd.h> #include <crypto/serpent.h> -#include <asm/crypto/serpent-sse2.h> +#include "serpent-sse2.h" #include "ecb_cbc_helpers.h" static int serpent_setkey_skcipher(struct crypto_skcipher *tfm,
arch/x86/crypto/twofish.h renamed from arch/x86/include/asm/crypto/twofish.h +0 -0
arch/x86/crypto/twofish_avx_glue.c +1 -1
@@ -15,8 +15,8 @@#include <crypto/algapi.h> #include <crypto/internal/simd.h> #include <crypto/twofish.h> -#include <asm/crypto/twofish.h> +#include "twofish.h" #include "ecb_cbc_helpers.h" #define TWOFISH_PARALLEL_BLOCKS 8
arch/x86/crypto/twofish_glue_3way.c +1 -1
@@ -5,7 +5,6 @@* Copyright (c) 2011 Jussi Kivilinna <[email protected]> */ -#include <asm/crypto/twofish.h> #include <crypto/algapi.h> #include <crypto/twofish.h> #include <linux/crypto.h>
@@ -13,6 +12,7 @@#include <linux/module.h> #include <linux/types.h> +#include "twofish.h" #include "ecb_cbc_helpers.h" EXPORT_SYMBOL_GPL (__twofish_enc_blk_3way);
Signature · 1 line
2.17.1
Herbert Xu
maintainerQuoted 115 lines · Ard Biesheuvel
NOTE: this is a follow-up to '[RFC PATCH 00/10] crypto: x86 - remove XTS and CTR glue helper code' [0].
After applying my performance fixes for AES-NI in XTS mode, the only remaining users of the x86 glue helper module are the niche algorithms camellia, cast5/6, serpent and twofish.
It is not clear from the history why all these different versions of these algorithms in XTS and CTR modes were added in the first place: the only in-kernel references that seem to exist are to cbc(serpent), cbc(camellia) and cbc(twofish) in the IPsec stack. The XTS spec only mentions AES, and CTR modes don't seem to be widely used either.
Since the glue helper code relies heavily on indirect calls for small chunks of in/output, it needs some work to recover from the performance hit caused by the retpoline changes. However, it makes sense to only expend the effort for algorithms that are being used in the first place, and this does not seem to be the case for XTS and CTR.
CTR mode can simply be removed: it is not used in the kernel, and it is highly unlikely that it is being relied upon via
algif_skcipher. And even if it was, the generic CTR mode driver can still provide the CTR transforms if necessary. While at it, accelerated implementations of DES and Blowfish in CTR mode are removed as well, for the same reasons.XTS mode may actually be in use by dm-crypt users *, so we cannot simply drop this code entirely. However, as it turns out, the XTS template wrapped around the ECB mode skciphers perform roughly on par, and so there is no need to retain all the complicated XTS helper logic. Users of dm-crypt that are relying on xts(camellia) or xts(serpent) in the field should not be impacted by these changes at all.
ECB and CBC are retained, but the glue helper indirection is replaced with a set of preprocessor macros which can be used to instantiate the same logic flow, but without relying on indirect calls into the glue helper module. Please refer to patch #14 for more background.
* Milan points out that Serpent, Camellia and Twofish in XTS mode are used by TrueCrypt/Veracrypt, which means that dm-crypt should retain support for these algorithms as well.
[0] https://lore.kernel.org/linux-crypto/[email protected]/
Changes since v1:
- avoid unnecessary memcpy()s in the new CBC decryption helper macro
- avoid unnecessarymemcpy()incamellia_decrypt_cbc_2way()
- add Eric's ackChanges since RFC:
- add Eric's ack to the initial XTS and CTR patches
- add patches to convert ECB and CBC modes
- add patches to remove DES and Blowfish in CTR modeCc Megha Dey <[email protected]>Cc Eric Biggers <[email protected]>Cc Herbert Xu <[email protected]>Cc Milan Broz <[email protected]>Cc Mike Snitzer <[email protected]>Ard Biesheuvel (21):
crypto: x86/camellia - switch to XTS template
crypto: x86/cast6 - switch to XTS template
crypto: x86/serpent- switch to XTS template
crypto: x86/twofish - switch to XTS template
crypto: x86/glue-helper - drop XTS helper routines
crypto: x86/camellia - drop CTR mode implementation
crypto: x86/serpent - drop CTR mode implementation
crypto: x86/cast5 - drop CTR mode implementation
crypto: x86/cast6 - drop CTR mode implementation
crypto: x86/twofish - drop CTR mode implementation
crypto: x86/glue-helper - drop CTR helper routines
crypto: x86/des - drop CTR mode implementation
crypto: x86/blowfish - drop CTR mode implementation
crypto: x86 - add some helper macros for ECB and CBC modes
crypto: x86/camellia - drop dependency on glue helper
crypto: x86/serpent - drop dependency on glue helper
crypto: x86/cast5 - drop dependency on glue helper
crypto: x86/cast6 - drop dependency on glue helper
crypto: x86/twofish - drop dependency on glue helper
crypto: x86 - remove glue helper module
crypto: x86 - use local headers for x86 specific shared declarationsarch/x86/crypto/Makefile | 2 - arch/x86/crypto/blowfish_glue.c | 107 ----- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 298 -------------- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 351 ---------------- arch/x86/{include/asm => }/crypto/camellia.h | 24 -- arch/x86/crypto/camellia_aesni_avx2_glue.c | 198 ++------- arch/x86/crypto/camellia_aesni_avx_glue.c | 216 +--------- arch/x86/crypto/camellia_glue.c | 145 ++----- arch/x86/crypto/cast5_avx_glue.c | 287 +------------ arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 84 ---- arch/x86/crypto/cast6_avx_glue.c | 207 +--------- arch/x86/crypto/des3_ede_glue.c | 104 ----- arch/x86/crypto/ecb_cbc_helpers.h | 76 ++++ arch/x86/crypto/glue_helper-asm-avx.S | 104 ----- arch/x86/crypto/glue_helper-asm-avx2.S | 136 ------- arch/x86/crypto/glue_helper.c | 381 ------------------ arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 68 ---- arch/x86/crypto/serpent-avx.h | 21 + arch/x86/crypto/serpent-avx2-asm_64.S | 87 ---- .../{include/asm => }/crypto/serpent-sse2.h | 0 arch/x86/crypto/serpent_avx2_glue.c | 185 +-------- arch/x86/crypto/serpent_avx_glue.c | 215 +--------- arch/x86/crypto/serpent_sse2_glue.c | 150 ++----- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 80 ---- arch/x86/{include/asm => }/crypto/twofish.h | 4 - arch/x86/crypto/twofish_avx_glue.c | 211 +--------- arch/x86/crypto/twofish_glue_3way.c | 160 ++------ arch/x86/include/asm/crypto/glue_helper.h | 118 ------ arch/x86/include/asm/crypto/serpent-avx.h | 42 -- crypto/Kconfig | 30 +- crypto/skcipher.c | 6 - include/crypto/internal/skcipher.h | 1 - 32 files changed, 326 insertions(+), 3772 deletions(-)
Quoted 8 lines
rename arch/x86/{include/asm => }/crypto/camellia.h (69%)
create mode 100644 arch/x86/crypto/ecb_cbc_helpers.h
delete mode 100644 arch/x86/crypto/glue_helper.c
create mode 100644 arch/x86/crypto/serpent-avx.h
rename arch/x86/{include/asm => }/crypto/serpent-sse2.h (100%)
rename arch/x86/{include/asm => }/crypto/twofish.h (80%)
delete mode 100644 arch/x86/include/asm/crypto/glue_helper.h
delete mode 100644 arch/x86/include/asm/crypto/serpent-avx.hAll applied. Thanks.
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt