crypto: x86 - remove XTS and CTR glue helper code
Subsystems (3)
cover crypto: x86 - remove XTS and CTR glue helper code
After applying my performance fixes for AES-NI in XTS mode, the only remaining users of the x86 glue helper module are the niche algorithms camellia, cast6, serpent and twofish.
It is not clear from the history why all these different versions of these algorithms in XTS and CTR modes were added in the first place: the only in-kernel references that seem to exist are to cbc(serpent), cbc(camellia) and cbc(twofish) in the IPsec stack. The XTS spec only mentions AES, and CTR modes don't seem to be widely used either.
Since the glue helper code relies heavily on indirect calls for small chunks of in/output, it needs some work to recover from the performance hit caused by the retpoline changes. However, it makes sense to only expend the effort for algorithms that are being used in the first place, and this does not seem to be the case for XTS and CTR.
CTR mode can simply be removed: it is not used in the kernel, and it is highly unlikely that it is being relied upon via algif_skcipher. And even if it was, the generic CTR mode driver can still provide the CTR transforms if necessary.
XTS mode may actually be in use by dm-crypt users, so we cannot simply drop this code entirely. However, as it turns out, the XTS template wrapped around the ECB mode skciphers perform roughly on par *, and so there is no need to retain all the complicated XTS helper logic. In the unlikely case that dm-crypt users are relying on xts(camellia) or xts(serpent) in the field, they should not be impacted by these changes at all.
As a follow-up, it makes sense to rework the ECB and CBC mode implementations to get rid of the indirect calls. Or perhaps we could drop [some of] these algorithms entirely ...
* tcrypt results for various XTS implementations below, captured on a Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
Ard Biesheuvel (10):
crypto: x86/camellia - switch to XTS template
crypto: x86/cast6 - switch to XTS template
crypto: x86/serpent- switch to XTS template
crypto: x86/twofish - switch to XTS template
crypto: x86/glue-helper - drop XTS helper routines
crypto: x86/camellia - drop CTR mode implementation
crypto: x86/cast6 - drop CTR mode implementation
crypto: x86/serpent - drop CTR mode implementation
crypto: x86/twofish - drop CTR mode implementation
crypto: x86/glue-helper - drop CTR helper routines
arch/x86/crypto/camellia-aesni-avx-asm_64.S | 297 ---------------- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 350 ------------------- arch/x86/crypto/camellia_aesni_avx2_glue.c | 111 ------ arch/x86/crypto/camellia_aesni_avx_glue.c | 141 +------- arch/x86/crypto/camellia_glue.c | 68 ---- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 84 ----- arch/x86/crypto/cast6_avx_glue.c | 146 -------- arch/x86/crypto/glue_helper-asm-avx.S | 104 ------ arch/x86/crypto/glue_helper-asm-avx2.S | 136 ------- arch/x86/crypto/glue_helper.c | 226 ------------ arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 68 ---- arch/x86/crypto/serpent-avx2-asm_64.S | 87 ----- arch/x86/crypto/serpent_avx2_glue.c | 110 ------ arch/x86/crypto/serpent_avx_glue.c | 152 -------- arch/x86/crypto/serpent_sse2_glue.c | 67 ---- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 80 ----- arch/x86/crypto/twofish_avx_glue.c | 136 ------- arch/x86/crypto/twofish_glue_3way.c | 72 ---- arch/x86/include/asm/crypto/camellia.h | 24 -- arch/x86/include/asm/crypto/glue_helper.h | 44 --- arch/x86/include/asm/crypto/serpent-avx.h | 21 -- arch/x86/include/asm/crypto/twofish.h | 4 - 22 files changed, 1 insertion(+), 2527 deletions(-)
Signature · 123 lines
2.17.1 testing speed of async xts(camellia) (xts-camellia-aesni-avx2) encryption tcrypt: test 0 (256 bit key, 16 byte blocks): 4295101 operations in 1 seconds (68721616 bytes) tcrypt: test 1 (256 bit key, 64 byte blocks): 2029490 operations in 1 seconds (129887360 bytes) tcrypt: test 2 (256 bit key, 256 byte blocks): 1626076 operations in 1 seconds (416275456 bytes) tcrypt: test 3 (256 bit key, 1024 byte blocks): 732878 operations in 1 seconds (750467072 bytes) tcrypt: test 4 (256 bit key, 1424 byte blocks): 366313 operations in 1 seconds (521629712 bytes) tcrypt: test 5 (256 bit key, 4096 byte blocks): 161737 operations in 1 seconds (662474752 bytes) tcrypt: test 6 (512 bit key, 16 byte blocks): 3876371 operations in 1 seconds (62021936 bytes) tcrypt: test 7 (512 bit key, 64 byte blocks): 1787813 operations in 1 seconds (114420032 bytes) tcrypt: test 8 (512 bit key, 256 byte blocks): 1578834 operations in 1 seconds (404181504 bytes) tcrypt: test 9 (512 bit key, 1024 byte blocks): 766805 operations in 1 seconds (785208320 bytes) tcrypt: test 10 (512 bit key, 1424 byte blocks): 366645 operations in 1 seconds (522102480 bytes) tcrypt: test 11 (512 bit key, 4096 byte blocks): 151122 operations in 1 seconds (618995712 bytes) tcrypt: testing speed of async xts(camellia) (xts(ecb-camellia-aesni-avx2)) encryption tcrypt: test 0 (256 bit key, 16 byte blocks): 3981536 operations in 1 seconds (63704576 bytes) tcrypt: test 1 (256 bit key, 64 byte blocks): 2696005 operations in 1 seconds (172544320 bytes) tcrypt: test 2 (256 bit key, 256 byte blocks): 1048119 operations in 1 seconds (268318464 bytes) tcrypt: test 3 (256 bit key, 1024 byte blocks): 716732 operations in 1 seconds (733933568 bytes) tcrypt: test 4 (256 bit key, 1424 byte blocks): 440474 operations in 1 seconds (627234976 bytes) tcrypt: test 5 (256 bit key, 4096 byte blocks): 178906 operations in 1 seconds (732798976 bytes) tcrypt: test 6 (512 bit key, 16 byte blocks): 3119162 operations in 1 seconds (49906592 bytes) tcrypt: test 7 (512 bit key, 64 byte blocks): 2286596 operations in 1 seconds (146342144 bytes) tcrypt: test 8 (512 bit key, 256 byte blocks): 1408661 operations in 1 seconds (360617216 bytes) tcrypt: test 9 (512 bit key, 1024 byte blocks): 669226 operations in 1 seconds (685287424 bytes) tcrypt: test 10 (512 bit key, 1424 byte blocks): 380543 operations in 1 seconds (541893232 bytes) tcrypt: test 11 (512 bit key, 4096 byte blocks): 144126 operations in 1 seconds (590340096 bytes) testing speed of async xts(camellia) (xts-camellia-aesni) encryption tcrypt: test 0 (256 bit key, 16 byte blocks): 3901755 operations in 1 seconds (62428080 bytes) tcrypt: test 1 (256 bit key, 64 byte blocks): 1719855 operations in 1 seconds (110070720 bytes) tcrypt: test 2 (256 bit key, 256 byte blocks): 1416991 operations in 1 seconds (362749696 bytes) tcrypt: test 3 (256 bit key, 1024 byte blocks): 481186 operations in 1 seconds (492734464 bytes) tcrypt: test 4 (256 bit key, 1424 byte blocks): 298401 operations in 1 seconds (424923024 bytes) tcrypt: test 5 (256 bit key, 4096 byte blocks): 120284 operations in 1 seconds (492683264 bytes) tcrypt: test 6 (512 bit key, 16 byte blocks): 3326174 operations in 1 seconds (53218784 bytes) tcrypt: test 7 (512 bit key, 64 byte blocks): 1428259 operations in 1 seconds (91408576 bytes) tcrypt: test 8 (512 bit key, 256 byte blocks): 1175894 operations in 1 seconds (301028864 bytes) tcrypt: test 9 (512 bit key, 1024 byte blocks): 407066 operations in 1 seconds (416835584 bytes) tcrypt: test 10 (512 bit key, 1424 byte blocks): 242931 operations in 1 seconds (345933744 bytes) tcrypt: test 11 (512 bit key, 4096 byte blocks): 95871 operations in 1 seconds (392687616 bytes) testing speed of async xts(camellia) (xts(ecb-camellia-aesni)) encryption tcrypt: test 0 (256 bit key, 16 byte blocks): 4004035 operations in 1 seconds (64064560 bytes) tcrypt: test 1 (256 bit key, 64 byte blocks): 2757081 operations in 1 seconds (176453184 bytes) tcrypt: test 2 (256 bit key, 256 byte blocks): 1626720 operations in 1 seconds (416440320 bytes) tcrypt: test 3 (256 bit key, 1024 byte blocks): 577725 operations in 1 seconds (591590400 bytes) tcrypt: test 4 (256 bit key, 1424 byte blocks): 393937 operations in 1 seconds (560966288 bytes) tcrypt: test 5 (256 bit key, 4096 byte blocks): 150055 operations in 1 seconds (614625280 bytes) tcrypt: test 6 (512 bit key, 16 byte blocks): 3427619 operations in 1 seconds (54841904 bytes) tcrypt: test 7 (512 bit key, 64 byte blocks): 2335827 operations in 1 seconds (149492928 bytes) tcrypt: test 8 (512 bit key, 256 byte blocks): 1412725 operations in 1 seconds (361657600 bytes) tcrypt: test 9 (512 bit key, 1024 byte blocks): 466635 operations in 1 seconds (477834240 bytes) tcrypt: test 10 (512 bit key, 1424 byte blocks): 314378 operations in 1 seconds (447674272 bytes) tcrypt: test 11 (512 bit key, 4096 byte blocks): 119159 operations in 1 seconds (488075264 bytes) testing speed of async xts(serpent) (xts-serpent-avx2) encryption tcrypt: test 0 (256 bit key, 16 byte blocks): 2665863 operations in 1 seconds (42653808 bytes) tcrypt: test 1 (256 bit key, 64 byte blocks): 1151015 operations in 1 seconds (73664960 bytes) tcrypt: test 2 (256 bit key, 256 byte blocks): 1824753 operations in 1 seconds (467136768 bytes) tcrypt: test 3 (256 bit key, 1024 byte blocks): 674375 operations in 1 seconds (690560000 bytes) tcrypt: test 4 (256 bit key, 1424 byte blocks): 434324 operations in 1 seconds (618477376 bytes) tcrypt: test 5 (256 bit key, 4096 byte blocks): 143875 operations in 1 seconds (589312000 bytes) tcrypt: test 6 (512 bit key, 16 byte blocks): 2676467 operations in 1 seconds (42823472 bytes) tcrypt: test 7 (512 bit key, 64 byte blocks): 1161001 operations in 1 seconds (74304064 bytes) tcrypt: test 8 (512 bit key, 256 byte blocks): 1830401 operations in 1 seconds (468582656 bytes) tcrypt: test 9 (512 bit key, 1024 byte blocks): 675560 operations in 1 seconds (691773440 bytes) tcrypt: test 10 (512 bit key, 1424 byte blocks): 431292 operations in 1 seconds (614159808 bytes) tcrypt: test 11 (512 bit key, 4096 byte blocks): 135674 operations in 1 seconds (555720704 bytes) testing speed of async xts(serpent) (xts(ecb-serpent-avx2)) encryption tcrypt: test 0 (256 bit key, 16 byte blocks): 2327282 operations in 1 seconds (37236512 bytes) tcrypt: test 1 (256 bit key, 64 byte blocks): 1121913 operations in 1 seconds (71802432 bytes) tcrypt: test 2 (256 bit key, 256 byte blocks): 1549949 operations in 1 seconds (396786944 bytes) tcrypt: test 3 (256 bit key, 1024 byte blocks): 597772 operations in 1 seconds (612118528 bytes) tcrypt: test 4 (256 bit key, 1424 byte blocks): 397386 operations in 1 seconds (565877664 bytes) tcrypt: test 5 (256 bit key, 4096 byte blocks): 140785 operations in 1 seconds (576655360 bytes) tcrypt: test 6 (512 bit key, 16 byte blocks): 2335122 operations in 1 seconds (37361952 bytes) tcrypt: test 7 (512 bit key, 64 byte blocks): 1123595 operations in 1 seconds (71910080 bytes) tcrypt: test 8 (512 bit key, 256 byte blocks): 1557279 operations in 1 seconds (398663424 bytes) tcrypt: test 9 (512 bit key, 1024 byte blocks): 595629 operations in 1 seconds (609924096 bytes) tcrypt: test 10 (512 bit key, 1424 byte blocks): 396338 operations in 1 seconds (564385312 bytes) tcrypt: test 11 (512 bit key, 4096 byte blocks): 139501 operations in 1 seconds (571396096 bytes) testing speed of async xts(serpent) (xts-serpent-avx) encryption tcrypt: test 0 (256 bit key, 16 byte blocks): 2718471 operations in 1 seconds (43495536 bytes) tcrypt: test 1 (256 bit key, 64 byte blocks): 1164397 operations in 1 seconds (74521408 bytes) tcrypt: test 2 (256 bit key, 256 byte blocks): 1189326 operations in 1 seconds (304467456 bytes) tcrypt: test 3 (256 bit key, 1024 byte blocks): 375279 operations in 1 seconds (384285696 bytes) tcrypt: test 4 (256 bit key, 1424 byte blocks): 260853 operations in 1 seconds (371454672 bytes) tcrypt: test 5 (256 bit key, 4096 byte blocks): 91367 operations in 1 seconds (374239232 bytes) tcrypt: test 6 (512 bit key, 16 byte blocks): 2679109 operations in 1 seconds (42865744 bytes) tcrypt: test 7 (512 bit key, 64 byte blocks): 1149832 operations in 1 seconds (73589248 bytes) tcrypt: test 8 (512 bit key, 256 byte blocks): 1180177 operations in 1 seconds (302125312 bytes) tcrypt: test 9 (512 bit key, 1024 byte blocks): 363975 operations in 1 seconds (372710400 bytes) tcrypt: test 10 (512 bit key, 1424 byte blocks): 267386 operations in 1 seconds (380757664 bytes) tcrypt: test 11 (512 bit key, 4096 byte blocks): 86933 operations in 1 seconds (356077568 bytes) testing speed of async xts(serpent) (xts(ecb-serpent-avx)) encryption tcrypt: test 0 (256 bit key, 16 byte blocks): 2408371 operations in 1 seconds (38533936 bytes) tcrypt: test 1 (256 bit key, 64 byte blocks): 1141626 operations in 1 seconds (73064064 bytes) tcrypt: test 2 (256 bit key, 256 byte blocks): 1072850 operations in 1 seconds (274649600 bytes) tcrypt: test 3 (256 bit key, 1024 byte blocks): 348694 operations in 1 seconds (357062656 bytes) tcrypt: test 4 (256 bit key, 1424 byte blocks): 250621 operations in 1 seconds (356884304 bytes) tcrypt: test 5 (256 bit key, 4096 byte blocks): 86043 operations in 1 seconds (352432128 bytes) tcrypt: test 6 (512 bit key, 16 byte blocks): 2406501 operations in 1 seconds (38504016 bytes) tcrypt: test 7 (512 bit key, 64 byte blocks): 1146211 operations in 1 seconds (73357504 bytes) tcrypt: test 8 (512 bit key, 256 byte blocks): 1075147 operations in 1 seconds (275237632 bytes) tcrypt: test 9 (512 bit key, 1024 byte blocks): 348007 operations in 1 seconds (356359168 bytes) tcrypt: test 10 (512 bit key, 1424 byte blocks): 250311 operations in 1 seconds (356442864 bytes) tcrypt: test 11 (512 bit key, 4096 byte blocks): 86062 operations in 1 seconds (352509952 bytes
RFC PATCH 1/10 crypto: x86/camellia - switch to XTS template
Now that the XTS template can wrap accelerated ECB modes, it can be used to implement Camellia in XTS mode as well, which turns out to be at least as fast, and sometimes even faster.
--- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 180 ----------------- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 206 -------------------- arch/x86/crypto/camellia_aesni_avx2_glue.c | 70 ------- arch/x86/crypto/camellia_aesni_avx_glue.c | 101 +--------- arch/x86/include/asm/crypto/camellia.h | 18 -- 5 files changed, 1 insertion(+), 574 deletions(-)
arch/x86/crypto/camellia-aesni-avx-asm_64.S +0 -180
@@ -593,10 +593,6 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) .Lbswap128_mask: .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 -/* For XTS mode IV generation */ -.Lxts_gf128mul_and_shl1_mask: - .byte 0x87, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0 - /* * pre-SubByte transform *
@@ -1111,179 +1107,3 @@ SYM_FUNC_START(camellia_ctr_16way) FRAME_END ret; SYM_FUNC_END(camellia_ctr_16way) - -#define gf128mul_x_ble(iv, mask, tmp) \ - vpsrad $31, iv, tmp; \ - vpaddq iv, iv, iv; \ - vpshufd $0x13, tmp, tmp; \ - vpand mask, tmp, tmp; \ - vpxor tmp, iv, iv; - -.align 8 -SYM_FUNC_START_LOCAL(camellia_xts_crypt_16way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst (16 blocks) - * %rdx: src (16 blocks) - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - * %r8: index for input whitening key - * %r9: pointer to __camellia_enc_blk16 or __camellia_dec_blk16 - */ - FRAME_BEGIN - - subq $(16 * 16), %rsp; - movq %rsp, %rax; - - vmovdqa .Lxts_gf128mul_and_shl1_mask, %xmm14; - - /* load IV */ - vmovdqu (%rcx), %xmm0; - vpxor 0 * 16(%rdx), %xmm0, %xmm15; - vmovdqu %xmm15, 15 * 16(%rax); - vmovdqu %xmm0, 0 * 16(%rsi); - - /* construct IVs */ - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 1 * 16(%rdx), %xmm0, %xmm15; - vmovdqu %xmm15, 14 * 16(%rax); - vmovdqu %xmm0, 1 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 2 * 16(%rdx), %xmm0, %xmm13; - vmovdqu %xmm0, 2 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 3 * 16(%rdx), %xmm0, %xmm12; - vmovdqu %xmm0, 3 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 4 * 16(%rdx), %xmm0, %xmm11; - vmovdqu %xmm0, 4 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 5 * 16(%rdx), %xmm0, %xmm10; - vmovdqu %xmm0, 5 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 6 * 16(%rdx), %xmm0, %xmm9; - vmovdqu %xmm0, 6 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 7 * 16(%rdx), %xmm0, %xmm8; - vmovdqu %xmm0, 7 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 8 * 16(%rdx), %xmm0, %xmm7; - vmovdqu %xmm0, 8 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 9 * 16(%rdx), %xmm0, %xmm6; - vmovdqu %xmm0, 9 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 10 * 16(%rdx), %xmm0, %xmm5; - vmovdqu %xmm0, 10 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 11 * 16(%rdx), %xmm0, %xmm4; - vmovdqu %xmm0, 11 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 12 * 16(%rdx), %xmm0, %xmm3; - vmovdqu %xmm0, 12 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 13 * 16(%rdx), %xmm0, %xmm2; - vmovdqu %xmm0, 13 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 14 * 16(%rdx), %xmm0, %xmm1; - vmovdqu %xmm0, 14 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vpxor 15 * 16(%rdx), %xmm0, %xmm15; - vmovdqu %xmm15, 0 * 16(%rax); - vmovdqu %xmm0, 15 * 16(%rsi); - - gf128mul_x_ble(%xmm0, %xmm14, %xmm15); - vmovdqu %xmm0, (%rcx); - - /* inpack16_pre: */ - vmovq (key_table)(CTX, %r8, 8), %xmm15; - vpshufb .Lpack_bswap, %xmm15, %xmm15; - vpxor 0 * 16(%rax), %xmm15, %xmm0; - vpxor %xmm1, %xmm15, %xmm1; - vpxor %xmm2, %xmm15, %xmm2; - vpxor %xmm3, %xmm15, %xmm3; - vpxor %xmm4, %xmm15, %xmm4; - vpxor %xmm5, %xmm15, %xmm5; - vpxor %xmm6, %xmm15, %xmm6; - vpxor %xmm7, %xmm15, %xmm7; - vpxor %xmm8, %xmm15, %xmm8; - vpxor %xmm9, %xmm15, %xmm9; - vpxor %xmm10, %xmm15, %xmm10; - vpxor %xmm11, %xmm15, %xmm11; - vpxor %xmm12, %xmm15, %xmm12; - vpxor %xmm13, %xmm15, %xmm13; - vpxor 14 * 16(%rax), %xmm15, %xmm14; - vpxor 15 * 16(%rax), %xmm15, %xmm15; - - CALL_NOSPEC r9; - - addq $(16 * 16), %rsp; - - vpxor 0 * 16(%rsi), %xmm7, %xmm7; - vpxor 1 * 16(%rsi), %xmm6, %xmm6; - vpxor 2 * 16(%rsi), %xmm5, %xmm5; - vpxor 3 * 16(%rsi), %xmm4, %xmm4; - vpxor 4 * 16(%rsi), %xmm3, %xmm3; - vpxor 5 * 16(%rsi), %xmm2, %xmm2; - vpxor 6 * 16(%rsi), %xmm1, %xmm1; - vpxor 7 * 16(%rsi), %xmm0, %xmm0; - vpxor 8 * 16(%rsi), %xmm15, %xmm15; - vpxor 9 * 16(%rsi), %xmm14, %xmm14; - vpxor 10 * 16(%rsi), %xmm13, %xmm13; - vpxor 11 * 16(%rsi), %xmm12, %xmm12; - vpxor 12 * 16(%rsi), %xmm11, %xmm11; - vpxor 13 * 16(%rsi), %xmm10, %xmm10; - vpxor 14 * 16(%rsi), %xmm9, %xmm9; - vpxor 15 * 16(%rsi), %xmm8, %xmm8; - write_output(%xmm7, %xmm6, %xmm5, %xmm4, %xmm3, %xmm2, %xmm1, %xmm0, - %xmm15, %xmm14, %xmm13, %xmm12, %xmm11, %xmm10, %xmm9, - %xmm8, %rsi); - - FRAME_END - ret; -SYM_FUNC_END(camellia_xts_crypt_16way) - -SYM_FUNC_START(camellia_xts_enc_16way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst (16 blocks) - * %rdx: src (16 blocks) - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - xorl %r8d, %r8d; /* input whitening key, 0 for enc */ - - leaq __camellia_enc_blk16, %r9; - - jmp camellia_xts_crypt_16way; -SYM_FUNC_END(camellia_xts_enc_16way) - -SYM_FUNC_START(camellia_xts_dec_16way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst (16 blocks) - * %rdx: src (16 blocks) - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - - cmpl $16, key_length(CTX); - movl $32, %r8d; - movl $24, %eax; - cmovel %eax, %r8d; /* input whitening key, last for dec */ - - leaq __camellia_dec_blk16, %r9; - - jmp camellia_xts_crypt_16way; -SYM_FUNC_END(camellia_xts_dec_16way)
arch/x86/crypto/camellia-aesni-avx2-asm_64.S +0 -206
215 lines not on this page · raw message
arch/x86/crypto/camellia_aesni_avx2_glue.c +0 -70
106 lines not on this page · raw message
arch/x86/crypto/camellia_aesni_avx_glue.c +1 -100
137 lines not on this page · raw message
arch/x86/include/asm/crypto/camellia.h +0 -18
@@ -19,18 +19,10 @@ struct camellia_ctx {u32 key_length; }; -struct camellia_xts_ctx { -struct camellia_ctx tweak_ctx; -struct camellia_ctx crypt_ctx; -}; -extern int __camellia_setkey (struct camellia_ctx *cctx,const unsigned char *key,unsigned int key_len); -extern int xts_camellia_setkey (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keylen); -/* regular block cipher functions */ asmlinkage void __camellia_enc_blk(const void *ctx, u8 *dst, const u8 *src,bool xor);
@@ -49,11 +41,6 @@ asmlinkage void camellia_cbc_dec_16way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void camellia_ctr_16way(const void *ctx, u8 *dst, const u8 *src,le128 *iv); -asmlinkage void camellia_xts_enc_16way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv); -asmlinkage void camellia_xts_dec_16way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv); -static inline void camellia_enc_blk (const void *ctx,u8 *dst,const u8 *src) {__camellia_enc_blk (ctx, dst, src, false);
@@ -83,9 +70,4 @@ extern void camellia_crypt_ctr(const void *ctx, u8 *dst, const u8 *src,extern void camellia_crypt_ctr_2way (const void *ctx,u8 *dst,const u8 *src,le128 *iv); -extern void camellia_xts_enc (const void *ctx,u8 *dst,const u8 *src, -le128 *iv); -extern void camellia_xts_dec (const void *ctx,u8 *dst,const u8 *src, -le128 *iv); -#endif /* ASM_X86_CAMELLIA_H */
Signature · 1 line
2.17.1
RFC PATCH 2/10 crypto: x86/cast6 - switch to XTS template
In reply to Ard BiesheuvelNow that the XTS template can wrap accelerated ECB modes, it can be used to implement CAST6 in XTS mode as well, which turns out to be at least as fast, and sometimes even faster
--- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 56 ----------- arch/x86/crypto/cast6_avx_glue.c | 98 -------------------- 2 files changed, 154 deletions(-)
arch/x86/crypto/cast6-avx-x86_64-asm_64.S +0 -56
@@ -212,8 +212,6 @@ .section .rodata.cst16, "aM", @progbits, 16 .align 16 -.Lxts_gf128mul_and_shl1_mask: - .byte 0x87, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0 .Lbswap_mask: .byte 3, 2, 1, 0, 7, 6, 5, 4, 11, 10, 9, 8, 15, 14, 13, 12 .Lbswap128_mask:
@@ -440,57 +438,3 @@ SYM_FUNC_START(cast6_ctr_8way) FRAME_END ret; SYM_FUNC_END(cast6_ctr_8way) - -SYM_FUNC_START(cast6_xts_enc_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - pushq %r15; - - movq %rdi, CTX - movq %rsi, %r11; - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2, - RX, RKR, RKM, .Lxts_gf128mul_and_shl1_mask); - - call __cast6_enc_blk8; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%r11, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - popq %r15; - FRAME_END - ret; -SYM_FUNC_END(cast6_xts_enc_8way) - -SYM_FUNC_START(cast6_xts_dec_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - pushq %r15; - - movq %rdi, CTX - movq %rsi, %r11; - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2, - RX, RKR, RKM, .Lxts_gf128mul_and_shl1_mask); - - call __cast6_dec_blk8; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%r11, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - popq %r15; - FRAME_END - ret; -SYM_FUNC_END(cast6_xts_dec_8way)
arch/x86/crypto/cast6_avx_glue.c +0 -98
140 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 3/10 crypto: x86/serpent- switch to XTS template
In reply to Ard BiesheuvelNow that the XTS template can wrap accelerated ECB modes, it can be used to implement Serpent in XTS mode as well, which turns out to be at least as fast, and sometimes even faster
--- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 48 ---------- arch/x86/crypto/serpent-avx2-asm_64.S | 62 ------------ arch/x86/crypto/serpent_avx2_glue.c | 72 -------------- arch/x86/crypto/serpent_avx_glue.c | 101 -------------------- arch/x86/include/asm/crypto/serpent-avx.h | 21 ---- 5 files changed, 304 deletions(-)
arch/x86/crypto/serpent-avx-x86_64-asm_64.S +0 -48
@@ -18,10 +18,6 @@ .align 16 .Lbswap128_mask: .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 -.section .rodata.cst16.xts_gf128mul_and_shl1_mask, "aM", @progbits, 16 -.align 16 -.Lxts_gf128mul_and_shl1_mask: - .byte 0x87, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0 .text
@@ -735,47 +731,3 @@ SYM_FUNC_START(serpent_ctr_8way_avx) FRAME_END ret; SYM_FUNC_END(serpent_ctr_8way_avx) - -SYM_FUNC_START(serpent_xts_enc_8way_avx) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2, - RK0, RK1, RK2, .Lxts_gf128mul_and_shl1_mask); - - call __serpent_enc_blk8_avx; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - FRAME_END - ret; -SYM_FUNC_END(serpent_xts_enc_8way_avx) - -SYM_FUNC_START(serpent_xts_dec_8way_avx) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2, - RK0, RK1, RK2, .Lxts_gf128mul_and_shl1_mask); - - call __serpent_dec_blk8_avx; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%rsi, RC1, RD1, RB1, RE1, RC2, RD2, RB2, RE2); - - FRAME_END - ret; -SYM_FUNC_END(serpent_xts_dec_8way_avx)
arch/x86/crypto/serpent-avx2-asm_64.S +0 -62
71 lines not on this page · raw message
arch/x86/crypto/serpent_avx2_glue.c +0 -72
108 lines not on this page · raw message
arch/x86/crypto/serpent_avx_glue.c +0 -101
149 lines not on this page · raw message
arch/x86/include/asm/crypto/serpent-avx.h +0 -21
@@ -10,11 +10,6 @@ struct crypto_skcipher;#define SERPENT_PARALLEL_BLOCKS 8 -struct serpent_xts_ctx { -struct serpent_ctx tweak_ctx; -struct serpent_ctx crypt_ctx; -}; - asmlinkage void serpent_ecb_enc_8way_avx(const void *ctx, u8 *dst,const u8 *src); asmlinkage void serpent_ecb_dec_8way_avx(const void *ctx, u8 *dst,
@@ -22,21 +17,5 @@ asmlinkage void serpent_ecb_dec_8way_avx(const void *ctx, u8 *dst, asmlinkage void serpent_cbc_dec_8way_avx(const void *ctx, u8 *dst,const u8 *src); -asmlinkage void serpent_ctr_8way_avx(const void *ctx, u8 *dst, const u8 *src, -le128 *iv); - -asmlinkage void serpent_xts_enc_8way_avx(const void *ctx, u8 *dst, -const u8 *src,le128 *iv); -asmlinkage void serpent_xts_dec_8way_avx(const void *ctx, u8 *dst, -const u8 *src,le128 *iv); - -extern void __serpent_crypt_ctr (const void *ctx,u8 *dst,const u8 *src, -le128 *iv); - -extern void serpent_xts_enc (const void *ctx,u8 *dst,const u8 *src,le128 *iv); -extern void serpent_xts_dec (const void *ctx,u8 *dst,const u8 *src,le128 *iv); - -extern int xts_serpent_setkey (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keylen);#endif
Signature · 1 line
2.17.1
RFC PATCH 4/10 crypto: x86/twofish - switch to XTS template
In reply to Ard BiesheuvelNow that the XTS template can wrap accelerated ECB modes, it can be used to implement Twofish in XTS mode as well, which turns out to be at least as fast, and sometimes even faster
--- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 53 ----------- arch/x86/crypto/twofish_avx_glue.c | 98 -------------------- 2 files changed, 151 deletions(-)
arch/x86/crypto/twofish-avx-x86_64-asm_64.S +0 -53
@@ -19,11 +19,6 @@ .Lbswap128_mask: .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 -.section .rodata.cst16.xts_gf128mul_and_shl1_mask, "aM", @progbits, 16 -.align 16 -.Lxts_gf128mul_and_shl1_mask: - .byte 0x87, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0 - .text /* structure of crypto context */
@@ -406,51 +401,3 @@ SYM_FUNC_START(twofish_ctr_8way) FRAME_END ret; SYM_FUNC_END(twofish_ctr_8way) - -SYM_FUNC_START(twofish_xts_enc_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - - movq %rsi, %r11; - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2, - RX0, RX1, RY0, .Lxts_gf128mul_and_shl1_mask); - - call __twofish_enc_blk8; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%r11, RC1, RD1, RA1, RB1, RC2, RD2, RA2, RB2); - - FRAME_END - ret; -SYM_FUNC_END(twofish_xts_enc_8way) - -SYM_FUNC_START(twofish_xts_dec_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (t ⊕ αⁿ ∈ GF(2¹²⁸)) - */ - FRAME_BEGIN - - movq %rsi, %r11; - - /* regs <= src, dst <= IVs, regs <= regs xor IVs */ - load_xts_8way(%rcx, %rdx, %rsi, RC1, RD1, RA1, RB1, RC2, RD2, RA2, RB2, - RX0, RX1, RY0, .Lxts_gf128mul_and_shl1_mask); - - call __twofish_dec_blk8; - - /* dst <= regs xor IVs(in dst) */ - store_xts_8way(%r11, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - FRAME_END - ret; -SYM_FUNC_END(twofish_xts_dec_8way)
arch/x86/crypto/twofish_avx_glue.c +0 -98
140 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 5/10 crypto: x86/glue-helper - drop XTS helper routines
In reply to Ard BiesheuvelThe glue helper's XTS routines are no longer used, so drop them.
--- arch/x86/crypto/glue_helper-asm-avx.S | 59 -------- arch/x86/crypto/glue_helper-asm-avx2.S | 78 ---------- arch/x86/crypto/glue_helper.c | 154 -------------------- arch/x86/include/asm/crypto/glue_helper.h | 12 -- 4 files changed, 303 deletions(-)
arch/x86/crypto/glue_helper-asm-avx.S +0 -59
@@ -79,62 +79,3 @@ vpxor (6*16)(src), x6, x6; \ vpxor (7*16)(src), x7, x7; \ store_8way(dst, x0, x1, x2, x3, x4, x5, x6, x7); - -#define gf128mul_x_ble(iv, mask, tmp) \ - vpsrad $31, iv, tmp; \ - vpaddq iv, iv, iv; \ - vpshufd $0x13, tmp, tmp; \ - vpand mask, tmp, tmp; \ - vpxor tmp, iv, iv; - -#define load_xts_8way(iv, src, dst, x0, x1, x2, x3, x4, x5, x6, x7, tiv, t0, \ - t1, xts_gf128mul_and_shl1_mask) \ - vmovdqa xts_gf128mul_and_shl1_mask, t0; \ - \ - /* load IV */ \ - vmovdqu (iv), tiv; \ - vpxor (0*16)(src), tiv, x0; \ - vmovdqu tiv, (0*16)(dst); \ - \ - /* construct and store IVs, also xor with source */ \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (1*16)(src), tiv, x1; \ - vmovdqu tiv, (1*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (2*16)(src), tiv, x2; \ - vmovdqu tiv, (2*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (3*16)(src), tiv, x3; \ - vmovdqu tiv, (3*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (4*16)(src), tiv, x4; \ - vmovdqu tiv, (4*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (5*16)(src), tiv, x5; \ - vmovdqu tiv, (5*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (6*16)(src), tiv, x6; \ - vmovdqu tiv, (6*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vpxor (7*16)(src), tiv, x7; \ - vmovdqu tiv, (7*16)(dst); \ - \ - gf128mul_x_ble(tiv, t0, t1); \ - vmovdqu tiv, (iv); - -#define store_xts_8way(dst, x0, x1, x2, x3, x4, x5, x6, x7) \ - vpxor (0*16)(dst), x0, x0; \ - vpxor (1*16)(dst), x1, x1; \ - vpxor (2*16)(dst), x2, x2; \ - vpxor (3*16)(dst), x3, x3; \ - vpxor (4*16)(dst), x4, x4; \ - vpxor (5*16)(dst), x5, x5; \ - vpxor (6*16)(dst), x6, x6; \ - vpxor (7*16)(dst), x7, x7; \ - store_8way(dst, x0, x1, x2, x3, x4, x5, x6, x7);
arch/x86/crypto/glue_helper-asm-avx2.S +0 -78
81 lines not on this page · raw message
arch/x86/crypto/glue_helper.c +0 -154
166 lines not on this page · raw message
arch/x86/include/asm/crypto/glue_helper.h +0 -12
@@ -15,8 +15,6 @@ typedef void (*common_glue_func_t)(const void *ctx, u8 *dst, const u8 *src);typedef void (*common_glue_cbc_func_t )(const void *ctx,u8 *dst,const u8 *src);typedef void (*common_glue_ctr_func_t )(const void *ctx,u8 *dst,const u8 *src,le128 *iv); -typedef void (*common_glue_xts_func_t )(const void *ctx,u8 *dst,const u8 *src, -le128 *iv);struct common_glue_func_entry {unsigned int num_blocks;/* number of blocks that @fn will process */
@@ -24,7 +22,6 @@ struct common_glue_func_entry {common_glue_func_t ecb;common_glue_cbc_func_t cbc;common_glue_ctr_func_t ctr; -common_glue_xts_func_t xts; } fn_u; };
@@ -106,13 +103,4 @@ extern int glue_cbc_decrypt_req_128bit(const struct common_glue_ctx *gctx,extern int glue_ctr_req_128bit (const struct common_glue_ctx *gctx,struct skcipher_request *req); -extern int glue_xts_req_128bit (const struct common_glue_ctx *gctx, -struct skcipher_request *req, -common_glue_func_t tweak_fn,void *tweak_ctx, -void *crypt_ctx,bool decrypt); - -extern void glue_xts_crypt_128bit_one (const void *ctx,u8 *dst, -const u8 *src,le128 *iv, -common_glue_func_t fn); -#endif /* _CRYPTO_GLUE_HELPER_H */
Signature · 1 line
2.17.1
kernel test robot
build botRe: [RFC PATCH 05/10] crypto: x86/glue-helper - drop XTS helper routines
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Ard,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on cryptodev/master]
[also build test ERROR on crypto/master v5.10 next-20201223] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Ard-Biesheuvel/crypto-x86-remove-XTS-and-CTR-glue-helper-code/20201224-064421 base: https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git master config: x86_64-randconfig-a016-20201223 (attached as .config) compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project cee1e7d14f4628d6174b33640d502bff3b54ae45)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/ad8857244950d3ad8c01b5bfae95f15e4ba703c4
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Ard-Biesheuvel/crypto-x86-remove-XTS-and-CTR-glue-helper-code/20201224-064421 git checkout ad8857244950d3ad8c01b5bfae95f15e4ba703c4
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
>> arch/x86/crypto/aesni-intel_glue.c:542:2: error: implicit declaration of function 'glue_xts_crypt_128bit_one' [-Werror,-Wimplicit-function-declaration]
glue_xts_crypt_128bit_one(ctx, dst, src, iv, aesni_enc);
^
arch/x86/crypto/aesni-intel_glue.c:547:2: error: implicit declaration of function 'glue_xts_crypt_128bit_one' [-Werror,-Wimplicit-function-declaration]
glue_xts_crypt_128bit_one(ctx, dst, src, iv, aesni_dec);
^
>> arch/x86/crypto/aesni-intel_glue.c:566:14: error: field designator 'xts' does not refer to any field in type 'union (anonymous union at arch/x86/include/asm/crypto/glue_helper.h:21:2)'
.fn_u = { .xts = aesni_xts_enc8 }
^
arch/x86/crypto/aesni-intel_glue.c:569:14: error: field designator 'xts' does not refer to any field in type 'union (anonymous union at arch/x86/include/asm/crypto/glue_helper.h:21:2)'
.fn_u = { .xts = aesni_xts_enc }
^
arch/x86/crypto/aesni-intel_glue.c:579:14: error: field designator 'xts' does not refer to any field in type 'union (anonymous union at arch/x86/include/asm/crypto/glue_helper.h:21:2)'
.fn_u = { .xts = aesni_xts_dec8 }
^
arch/x86/crypto/aesni-intel_glue.c:582:14: error: field designator 'xts' does not refer to any field in type 'union (anonymous union at arch/x86/include/asm/crypto/glue_helper.h:21:2)'
.fn_u = { .xts = aesni_xts_dec }
^
>> arch/x86/crypto/aesni-intel_glue.c:591:9: error: implicit declaration of function 'glue_xts_req_128bit' [-Werror,-Wimplicit-function-declaration]
return glue_xts_req_128bit(&aesni_enc_xts, req, aesni_enc,
^
arch/x86/crypto/aesni-intel_glue.c:591:9: note: did you mean 'glue_ctr_req_128bit'?
arch/x86/include/asm/crypto/glue_helper.h:103:12: note: 'glue_ctr_req_128bit' declared here
extern int glue_ctr_req_128bit(const struct common_glue_ctx *gctx,
^ arch/x86/crypto/aesni-intel_glue.c:602:9: error: implicit declaration of function 'glue_xts_req_128bit' [-Werror,-Wimplicit-function-declaration]
return glue_xts_req_128bit(&aesni_dec_xts, req, aesni_enc,
^
8 errors generated.
vim +/glue_xts_crypt_128bit_one +542 arch/x86/crypto/aesni-intel_glue.c
023af608254add7 Jussi Kivilinna 2012-07-22 538
023af608254add7 Jussi Kivilinna 2012-07-22 539
9c1e8836edbbaf3 Kees Cook 2019-11-26 540 static void aesni_xts_enc(const void *ctx, u8 *dst, const u8 *src, le128 *iv)
32bec973a8435af Jussi Kivilinna 2012-10-18 541 {
9c1e8836edbbaf3 Kees Cook 2019-11-26 @542 glue_xts_crypt_128bit_one(ctx, dst, src, iv, aesni_enc);
32bec973a8435af Jussi Kivilinna 2012-10-18 543 }
32bec973a8435af Jussi Kivilinna 2012-10-18 544
9c1e8836edbbaf3 Kees Cook 2019-11-26 545 static void aesni_xts_dec(const void *ctx, u8 *dst, const u8 *src, le128 *iv)
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 546 {
9c1e8836edbbaf3 Kees Cook 2019-11-26 547 glue_xts_crypt_128bit_one(ctx, dst, src, iv, aesni_dec);
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 548 }
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 549
9c1e8836edbbaf3 Kees Cook 2019-11-26 550 static void aesni_xts_enc8(const void *ctx, u8 *dst, const u8 *src, le128 *iv)
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 551 {
9c1e8836edbbaf3 Kees Cook 2019-11-26 552 aesni_xts_crypt8(ctx, dst, src, true, iv);
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 553 }
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 554
9c1e8836edbbaf3 Kees Cook 2019-11-26 555 static void aesni_xts_dec8(const void *ctx, u8 *dst, const u8 *src, le128 *iv)
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 556 {
9c1e8836edbbaf3 Kees Cook 2019-11-26 557 aesni_xts_crypt8(ctx, dst, src, false, iv);
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 558 }
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 559
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 560 static const struct common_glue_ctx aesni_enc_xts = {
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 561 .num_funcs = 2,
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 562 .fpu_blocks_limit = 1,
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 563
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 564 .funcs = { {
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 565 .num_blocks = 8,
9c1e8836edbbaf3 Kees Cook 2019-11-26 @566 .fn_u = { .xts = aesni_xts_enc8 }
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 567 }, {
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 568 .num_blocks = 1,
9c1e8836edbbaf3 Kees Cook 2019-11-26 @569 .fn_u = { .xts = aesni_xts_enc }
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 570 } }
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 571 };
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 572
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 573 static const struct common_glue_ctx aesni_dec_xts = {
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 574 .num_funcs = 2,
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 575 .fpu_blocks_limit = 1,
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 576
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 577 .funcs = { {
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 578 .num_blocks = 8,
9c1e8836edbbaf3 Kees Cook 2019-11-26 579 .fn_u = { .xts = aesni_xts_dec8 }
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 580 }, {
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 581 .num_blocks = 1,
9c1e8836edbbaf3 Kees Cook 2019-11-26 582 .fn_u = { .xts = aesni_xts_dec }
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 583 } }
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 584 };
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 585
85671860caaca2f Herbert Xu 2016-11-22 586 static int xts_encrypt(struct skcipher_request *req)
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 587 {
85671860caaca2f Herbert Xu 2016-11-22 588 struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
85671860caaca2f Herbert Xu 2016-11-22 589 struct aesni_xts_ctx *ctx = crypto_skcipher_ctx(tfm);
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 590
9c1e8836edbbaf3 Kees Cook 2019-11-26 @591 return glue_xts_req_128bit(&aesni_enc_xts, req, aesni_enc,
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 592 aes_ctx(ctx->raw_tweak_ctx),
8ce5fac2dc1bf64 Ard Biesheuvel 2019-08-16 593 aes_ctx(ctx->raw_crypt_ctx),
8ce5fac2dc1bf64 Ard Biesheuvel 2019-08-16 594 false);
c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 595 }c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 596
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Re: [RFC PATCH 05/10] crypto: x86/glue-helper - drop XTS helper routines
Quoted 10 lines · kernel test robot
Hi Ard,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on cryptodev/master]
[also build test ERROR oncrypto/masterv5.10 next-20201223] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
This is a false positive - the base is a patch that was also posted to the linux-crypto list in the days before.
'[PATCH 2/2] crypto: x86/aes-ni-xts - rewrite and drop indirections via glue helper'
Quoted 117 lines
url: https://github.com/0day-ci/linux/commits/Ard-Biesheuvel/crypto-x86-remove-XTS-and-CTR-glue-helper-code/20201224-064421 base: https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git master config: x86_64-randconfig-a016-20201223 (attached as .config) compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project cee1e7d14f4628d6174b33640d502bff3b54ae45)reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# installx86_64cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/ad8857244950d3ad8c01b5bfae95f15e4ba703c4
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Ard-Biesheuvel/crypto-x86-remove-XTS-and-CTR-glue-helper-code/20201224-064421 git checkoutad8857244950d3ad8c01b5bfae95f15e4ba703c4
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64If you fix the issue, kindly add following tag as appropriate
Reported-by kernel test robot <[email protected]>All errors (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
arch/x86/crypto/aesni-intel_glue.c:542:2: error: implicit declaration of function 'glue_xts_crypt_128bit_one' [-Werror,-Wimplicit-function-declaration]glue_xts_crypt_128bit_one(ctx, dst, src, iv, aesni_enc); ^ arch/x86/crypto/aesni-intel_glue.c:547:2: error: implicit declaration of function 'glue_xts_crypt_128bit_one' [-Werror,-Wimplicit-function-declaration] glue_xts_crypt_128bit_one(ctx, dst, src, iv, aesni_dec); ^Quoted 1 line
Quoted 1 line
arch/x86/crypto/aesni-intel_glue.c:566:14: error: field designator 'xts' does not refer to any field in type 'union (anonymous union atarch/x86/include/asm/crypto/glue_helper.h:21:2)'.fn_u = { .xts = aesni_xts_enc8 } ^ arch/x86/crypto/aesni-intel_glue.c:569:14: error: field designator 'xts' does not refer to any field in type 'union (anonymous union at arch/x86/include/asm/crypto/glue_helper.h:21:2)' .fn_u = { .xts = aesni_xts_enc } ^ arch/x86/crypto/aesni-intel_glue.c:579:14: error: field designator 'xts' does not refer to any field in type 'union (anonymous union at arch/x86/include/asm/crypto/glue_helper.h:21:2)' .fn_u = { .xts = aesni_xts_dec8 } ^ arch/x86/crypto/aesni-intel_glue.c:582:14: error: field designator 'xts' does not refer to any field in type 'union (anonymous union at arch/x86/include/asm/crypto/glue_helper.h:21:2)' .fn_u = { .xts = aesni_xts_dec } ^Quoted 1 line
Quoted 1 line
arch/x86/crypto/aesni-intel_glue.c:591:9: error: implicit declaration of function 'glue_xts_req_128bit' [-Werror,-Wimplicit-function-declaration]return glue_xts_req_128bit(&aesni_enc_xts, req,
aesni_enc,
^
arch/x86/crypto/aesni-intel_glue.c:591:9: note: did you mean 'glue_ctr_req_128bit'?arch/x86/include/asm/crypto/glue_helper.h:103:12: note: 'glue_ctr_req_128bit' declared here extern int glue_ctr_req_128bit(const structcommon_glue_ctx*gctx,^
arch/x86/crypto/aesni-intel_glue.c:602:9: error: implicit declaration of function 'glue_xts_req_128bit' [-Werror,-Wimplicit-function-declaration]
return glue_xts_req_128bit(&aesni_dec_xts, req,aesni_enc,
^
8 errors generated.vim +/glue_xts_crypt_128bit_one +542
arch/x86/crypto/aesni-intel_glue.c023af608254add7 Jussi Kivilinna 2012-07-22 538 023af608254add7 Jussi Kivilinna 2012-07-22 539 9c1e8836edbbaf3 Kees Cook 2019-11-26 540 static void aesni_xts_enc(const void *ctx, u8 *dst, const u8 *src, le128 *iv) 32bec973a8435af Jussi Kivilinna 2012-10-18 541 { 9c1e8836edbbaf3 Kees Cook 2019-11-26 @542 glue_xts_crypt_128bit_one(ctx, dst, src, iv, aesni_enc); 32bec973a8435af Jussi Kivilinna 2012-10-18 543 } 32bec973a8435af Jussi Kivilinna 2012-10-18 544 9c1e8836edbbaf3 Kees Cook 2019-11-26 545 static void aesni_xts_dec(const void *ctx, u8 *dst, const u8 *src, le128 *iv) c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 546 { 9c1e8836edbbaf3 Kees Cook 2019-11-26 547 glue_xts_crypt_128bit_one(ctx, dst, src, iv, aesni_dec); c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 548 } c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 549 9c1e8836edbbaf3 Kees Cook 2019-11-26 550 static void aesni_xts_enc8(const void *ctx, u8 *dst, const u8 *src, le128 *iv) c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 551 { 9c1e8836edbbaf3 Kees Cook 2019-11-26 552 aesni_xts_crypt8(ctx, dst, src, true, iv); c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 553 } c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 554 9c1e8836edbbaf3 Kees Cook 2019-11-26 555 static void aesni_xts_dec8(const void *ctx, u8 *dst, const u8 *src, le128 *iv) c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 556 { 9c1e8836edbbaf3 Kees Cook 2019-11-26 557 aesni_xts_crypt8(ctx, dst, src, false, iv); c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 558 } c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 559 c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 560 static const struct common_glue_ctx aesni_enc_xts = { c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 561 .num_funcs = 2, c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 562 .fpu_blocks_limit = 1, c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 563 c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 564 .funcs = { { c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 565 .num_blocks = 8,
9c1e8836edbbaf3Kees Cook 2019-11-26 @566 .fn_u = { .xts =aesni_xts_enc8}c456a9cd1ac4eaeJussi Kivilinna 2013-04-08 567 }, {c456a9cd1ac4eaeJussi Kivilinna 2013-04-08 568 .num_blocks = 1,9c1e8836edbbaf3Kees Cook 2019-11-26 @569 .fn_u = { .xts =aesni_xts_enc}c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 570 } } c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 571 }; c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 572 c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 573 static const struct common_glue_ctx aesni_dec_xts = { c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 574 .num_funcs = 2, c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 575 .fpu_blocks_limit = 1, c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 576 c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 577 .funcs = { { c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 578 .num_blocks = 8, 9c1e8836edbbaf3 Kees Cook 2019-11-26 579 .fn_u = { .xts = aesni_xts_dec8 } c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 580 }, { c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 581 .num_blocks = 1, 9c1e8836edbbaf3 Kees Cook 2019-11-26 582 .fn_u = { .xts = aesni_xts_dec } c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 583 } } c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 584 }; c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 585 85671860caaca2f Herbert Xu 2016-11-22 586 static int xts_encrypt(struct skcipher_request *req) c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 587 { 85671860caaca2f Herbert Xu 2016-11-22 588 struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 85671860caaca2f Herbert Xu 2016-11-22 589 struct aesni_xts_ctx *ctx = crypto_skcipher_ctx(tfm); c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 590 9c1e8836edbbaf3 Kees Cook 2019-11-26 @591 return glue_xts_req_128bit(&aesni_enc_xts, req, aesni_enc, c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 592 aes_ctx(ctx->raw_tweak_ctx), 8ce5fac2dc1bf64 Ard Biesheuvel 2019-08-16 593 aes_ctx(ctx->raw_crypt_ctx), 8ce5fac2dc1bf64 Ard Biesheuvel 2019-08-16 594 false); c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 595 } c456a9cd1ac4eae Jussi Kivilinna 2013-04-08 596---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RFC PATCH 6/10 crypto: x86/camellia - drop CTR mode implementation
In reply to Ard BiesheuvelCamellia in CTR mode is never used by the kernel directly, and is highly unlikely to be relied upon by dm-crypt or algif_skcipher. So let's drop the accelerated CTR mode implementation, and instead, rely on the CTR template and the bare cipher.
--- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 117 ---------------- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 144 -------------------- arch/x86/crypto/camellia_aesni_avx2_glue.c | 41 ------ arch/x86/crypto/camellia_aesni_avx_glue.c | 40 ------ arch/x86/crypto/camellia_glue.c | 68 --------- arch/x86/include/asm/crypto/camellia.h | 6 - 6 files changed, 416 deletions(-)
arch/x86/crypto/camellia-aesni-avx-asm_64.S +0 -117
@@ -589,10 +589,6 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) .long 0x80808080 .long 0x80808080 -/* For CTR-mode IV byteswap */ -.Lbswap128_mask: - .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 - /* * pre-SubByte transform *
@@ -994,116 +990,3 @@ SYM_FUNC_START(camellia_cbc_dec_16way) FRAME_END ret; SYM_FUNC_END(camellia_cbc_dec_16way) - -#define inc_le128(x, minus_one, tmp) \ - vpcmpeqq minus_one, x, tmp; \ - vpsubq minus_one, x, x; \ - vpslldq $8, tmp, tmp; \ - vpsubq tmp, x, x; - -SYM_FUNC_START(camellia_ctr_16way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst (16 blocks) - * %rdx: src (16 blocks) - * %rcx: iv (little endian, 128bit) - */ - FRAME_BEGIN - - subq $(16 * 16), %rsp; - movq %rsp, %rax; - - vmovdqa .Lbswap128_mask, %xmm14; - - /* load IV and byteswap */ - vmovdqu (%rcx), %xmm0; - vpshufb %xmm14, %xmm0, %xmm15; - vmovdqu %xmm15, 15 * 16(%rax); - - vpcmpeqd %xmm15, %xmm15, %xmm15; - vpsrldq $8, %xmm15, %xmm15; /* low: -1, high: 0 */ - - /* construct IVs */ - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm13; - vmovdqu %xmm13, 14 * 16(%rax); - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm13; - vmovdqu %xmm13, 13 * 16(%rax); - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm12; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm11; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm10; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm9; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm8; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm7; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm6; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm5; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm4; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm3; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm2; - inc_le128(%xmm0, %xmm15, %xmm13); - vpshufb %xmm14, %xmm0, %xmm1; - inc_le128(%xmm0, %xmm15, %xmm13); - vmovdqa %xmm0, %xmm13; - vpshufb %xmm14, %xmm0, %xmm0; - inc_le128(%xmm13, %xmm15, %xmm14); - vmovdqu %xmm13, (%rcx); - - /* inpack16_pre: */ - vmovq (key_table)(CTX), %xmm15; - vpshufb .Lpack_bswap, %xmm15, %xmm15; - vpxor %xmm0, %xmm15, %xmm0; - vpxor %xmm1, %xmm15, %xmm1; - vpxor %xmm2, %xmm15, %xmm2; - vpxor %xmm3, %xmm15, %xmm3; - vpxor %xmm4, %xmm15, %xmm4; - vpxor %xmm5, %xmm15, %xmm5; - vpxor %xmm6, %xmm15, %xmm6; - vpxor %xmm7, %xmm15, %xmm7; - vpxor %xmm8, %xmm15, %xmm8; - vpxor %xmm9, %xmm15, %xmm9; - vpxor %xmm10, %xmm15, %xmm10; - vpxor %xmm11, %xmm15, %xmm11; - vpxor %xmm12, %xmm15, %xmm12; - vpxor 13 * 16(%rax), %xmm15, %xmm13; - vpxor 14 * 16(%rax), %xmm15, %xmm14; - vpxor 15 * 16(%rax), %xmm15, %xmm15; - - call __camellia_enc_blk16; - - addq $(16 * 16), %rsp; - - vpxor 0 * 16(%rdx), %xmm7, %xmm7; - vpxor 1 * 16(%rdx), %xmm6, %xmm6; - vpxor 2 * 16(%rdx), %xmm5, %xmm5; - vpxor 3 * 16(%rdx), %xmm4, %xmm4; - vpxor 4 * 16(%rdx), %xmm3, %xmm3; - vpxor 5 * 16(%rdx), %xmm2, %xmm2; - vpxor 6 * 16(%rdx), %xmm1, %xmm1; - vpxor 7 * 16(%rdx), %xmm0, %xmm0; - vpxor 8 * 16(%rdx), %xmm15, %xmm15; - vpxor 9 * 16(%rdx), %xmm14, %xmm14; - vpxor 10 * 16(%rdx), %xmm13, %xmm13; - vpxor 11 * 16(%rdx), %xmm12, %xmm12; - vpxor 12 * 16(%rdx), %xmm11, %xmm11; - vpxor 13 * 16(%rdx), %xmm10, %xmm10; - vpxor 14 * 16(%rdx), %xmm9, %xmm9; - vpxor 15 * 16(%rdx), %xmm8, %xmm8; - write_output(%xmm7, %xmm6, %xmm5, %xmm4, %xmm3, %xmm2, %xmm1, %xmm0, - %xmm15, %xmm14, %xmm13, %xmm12, %xmm11, %xmm10, %xmm9, - %xmm8, %rsi); - - FRAME_END - ret; -SYM_FUNC_END(camellia_ctr_16way)
arch/x86/crypto/camellia-aesni-avx2-asm_64.S +0 -144
153 lines not on this page · raw message
arch/x86/crypto/camellia_aesni_avx2_glue.c +0 -41
65 lines not on this page · raw message
arch/x86/crypto/camellia_aesni_avx_glue.c +0 -40
@@ -26,10 +26,6 @@ EXPORT_SYMBOL_GPL(camellia_ecb_dec_16way); asmlinkage void camellia_cbc_dec_16way(const void *ctx, u8 *dst, const u8 *src);EXPORT_SYMBOL_GPL (camellia_cbc_dec_16way); -asmlinkage void camellia_ctr_16way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv); -EXPORT_SYMBOL_GPL (camellia_ctr_16way); -static const struct common_glue_ctx camellia_enc = { .num_funcs =3 , .fpu_blocks_limit =CAMELLIA_AESNI_PARALLEL_BLOCKS ,
@@ -46,22 +42,6 @@ static const struct common_glue_ctx camellia_enc = { } } }; -static const struct common_glue_ctx camellia_ctr = { - .num_funcs =3 , - .fpu_blocks_limit =CAMELLIA_AESNI_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =CAMELLIA_AESNI_PARALLEL_BLOCKS , - .fn_u = { .ctr = camellia_ctr_16way } - }, { - .num_blocks =2 , - .fn_u = { .ctr = camellia_crypt_ctr_2way } - }, { - .num_blocks =1 , - .fn_u = { .ctr = camellia_crypt_ctr } - } } -}; -static const struct common_glue_ctx camellia_dec = { .num_funcs =3 , .fpu_blocks_limit =CAMELLIA_AESNI_PARALLEL_BLOCKS ,
@@ -120,11 +100,6 @@ static int cbc_decrypt(struct skcipher_request *req)return glue_cbc_decrypt_req_128bit (&camellia_dec_cbc, req); } -static int ctr_crypt (struct skcipher_request *req) -{ -return glue_ctr_req_128bit (&camellia_ctr, req); -} -static struct skcipher_alg camellia_algs[] = { { .base.cra_name ="__ecb(camellia)" ,
@@ -153,21 +128,6 @@ static struct skcipher_alg camellia_algs[] = { .setkey = camellia_setkey, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, - }, { - .base.cra_name = "__ctr(camellia)", - .base.cra_driver_name = "__ctr-camellia-aesni", - .base.cra_priority = 400, - .base.cra_flags = CRYPTO_ALG_INTERNAL, - .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct camellia_ctx), - .base.cra_module = THIS_MODULE, - .min_keysize = CAMELLIA_MIN_KEY_SIZE, - .max_keysize = CAMELLIA_MAX_KEY_SIZE, - .ivsize = CAMELLIA_BLOCK_SIZE, - .chunksize = CAMELLIA_BLOCK_SIZE, - .setkey = camellia_setkey, - .encrypt = ctr_crypt, - .decrypt = ctr_crypt, } };
arch/x86/crypto/camellia_glue.c +0 -68
92 lines not on this page · raw message
arch/x86/include/asm/crypto/camellia.h +0 -6
@@ -38,8 +38,6 @@ asmlinkage void camellia_ecb_enc_16way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void camellia_ecb_dec_16way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void camellia_cbc_dec_16way(const void *ctx, u8 *dst, const u8 *src); -asmlinkage void camellia_ctr_16way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv);static inline void camellia_enc_blk (const void *ctx,u8 *dst,const u8 *src) {
@@ -65,9 +63,5 @@ static inline void camellia_enc_blk_xor_2way(const void *ctx, u8 *dst,/* glue helpers */ extern void camellia_decrypt_cbc_2way (const void *ctx,u8 *dst,const u8 *src); -extern void camellia_crypt_ctr (const void *ctx,u8 *dst,const u8 *src, -le128 *iv); -extern void camellia_crypt_ctr_2way (const void *ctx,u8 *dst,const u8 *src, -le128 *iv);#endif /* ASM_X86_CAMELLIA_H */
Signature · 1 line
2.17.1
RFC PATCH 7/10 crypto: x86/cast6 - drop CTR mode implementation
In reply to Ard BiesheuvelCAST6 in CTR mode is never used by the kernel directly*, and is highly unlikely to be relied upon by dm-crypt or algif_skcipher. So let's drop the accelerated CTR mode implementation, and instead, rely on the CTR template and the bare cipher.
* it is not used at all, so we might remove the accelerated implementation entirely in a future patch.
--- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 28 ------------ arch/x86/crypto/cast6_avx_glue.c | 48 -------------------- 2 files changed, 76 deletions(-)
arch/x86/crypto/cast6-avx-x86_64-asm_64.S +0 -28
@@ -410,31 +410,3 @@ SYM_FUNC_START(cast6_cbc_dec_8way) FRAME_END ret; SYM_FUNC_END(cast6_cbc_dec_8way) - -SYM_FUNC_START(cast6_ctr_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (little endian, 128bit) - */ - FRAME_BEGIN - pushq %r12; - pushq %r15 - - movq %rdi, CTX; - movq %rsi, %r11; - movq %rdx, %r12; - - load_ctr_8way(%rcx, .Lbswap128_mask, RA1, RB1, RC1, RD1, RA2, RB2, RC2, - RD2, RX, RKR, RKM); - - call __cast6_enc_blk8; - - store_ctr_8way(%r12, %r11, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - popq %r15; - popq %r12; - FRAME_END - ret; -SYM_FUNC_END(cast6_ctr_8way)
arch/x86/crypto/cast6_avx_glue.c +0 -48
78 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 8/10 crypto: x86/serpent - drop CTR mode implementation
In reply to Ard BiesheuvelSerpent in CTR mode is never used by the kernel directly, and is highly unlikely to be relied upon by dm-crypt or algif_skcipher. So let's drop the accelerated CTR mode implementation, and instead, rely on the CTR template and the bare cipher.
--- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 20 ------ arch/x86/crypto/serpent-avx2-asm_64.S | 25 -------- arch/x86/crypto/serpent_avx2_glue.c | 38 ----------- arch/x86/crypto/serpent_avx_glue.c | 51 --------------- arch/x86/crypto/serpent_sse2_glue.c | 67 -------------------- 5 files changed, 201 deletions(-)
arch/x86/crypto/serpent-avx-x86_64-asm_64.S +0 -20
@@ -711,23 +711,3 @@ SYM_FUNC_START(serpent_cbc_dec_8way_avx) FRAME_END ret; SYM_FUNC_END(serpent_cbc_dec_8way_avx) - -SYM_FUNC_START(serpent_ctr_8way_avx) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (little endian, 128bit) - */ - FRAME_BEGIN - - load_ctr_8way(%rcx, .Lbswap128_mask, RA1, RB1, RC1, RD1, RA2, RB2, RC2, - RD2, RK0, RK1, RK2); - - call __serpent_enc_blk8_avx; - - store_ctr_8way(%rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - FRAME_END - ret; -SYM_FUNC_END(serpent_ctr_8way_avx)
arch/x86/crypto/serpent-avx2-asm_64.S +0 -25
@@ -724,28 +724,3 @@ SYM_FUNC_START(serpent_cbc_dec_16way) FRAME_END ret; SYM_FUNC_END(serpent_cbc_dec_16way) - -SYM_FUNC_START(serpent_ctr_16way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst (16 blocks) - * %rdx: src (16 blocks) - * %rcx: iv (little endian, 128bit) - */ - FRAME_BEGIN - - vzeroupper; - - load_ctr_16way(%rcx, .Lbswap128_mask, RA1, RB1, RC1, RD1, RA2, RB2, RC2, - RD2, RK0, RK0x, RK1, RK1x, RK2, RK2x, RK3, RK3x, RNOT, - tp); - - call __serpent_enc_blk16; - - store_ctr_16way(%rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); - - vzeroupper; - - FRAME_END - ret; -SYM_FUNC_END(serpent_ctr_16way)
arch/x86/crypto/serpent_avx2_glue.c +0 -38
@@ -22,8 +22,6 @@ asmlinkage void serpent_ecb_enc_16way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void serpent_ecb_dec_16way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void serpent_cbc_dec_16way(const void *ctx, u8 *dst, const u8 *src); -asmlinkage void serpent_ctr_16way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv);static int serpent_setkey_skcipher (struct crypto_skcipher *tfm,const u8 *key,unsigned int keylen) {
@@ -46,22 +44,6 @@ static const struct common_glue_ctx serpent_enc = { } } }; -static const struct common_glue_ctx serpent_ctr = { - .num_funcs =3 , - .fpu_blocks_limit =8 , - - .funcs = { { - .num_blocks =16 , - .fn_u = { .ctr = serpent_ctr_16way } - }, { - .num_blocks =8 , - .fn_u = { .ctr = serpent_ctr_8way_avx } - }, { - .num_blocks =1 , - .fn_u = { .ctr = __serpent_crypt_ctr } - } } -}; -static const struct common_glue_ctx serpent_dec = { .num_funcs =3 , .fpu_blocks_limit =8 ,
@@ -114,11 +96,6 @@ static int cbc_decrypt(struct skcipher_request *req)return glue_cbc_decrypt_req_128bit (&serpent_dec_cbc, req); } -static int ctr_crypt (struct skcipher_request *req) -{ -return glue_ctr_req_128bit (&serpent_ctr, req); -} -static struct skcipher_alg serpent_algs[] = { { .base.cra_name ="__ecb(serpent)" ,
@@ -147,21 +124,6 @@ static struct skcipher_alg serpent_algs[] = { .setkey = serpent_setkey_skcipher, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, - }, { - .base.cra_name = "__ctr(serpent)", - .base.cra_driver_name = "__ctr-serpent-avx2", - .base.cra_priority = 600, - .base.cra_flags = CRYPTO_ALG_INTERNAL, - .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct serpent_ctx), - .base.cra_module = THIS_MODULE, - .min_keysize = SERPENT_MIN_KEY_SIZE, - .max_keysize = SERPENT_MAX_KEY_SIZE, - .ivsize = SERPENT_BLOCK_SIZE, - .chunksize = SERPENT_BLOCK_SIZE, - .setkey = serpent_setkey_skcipher, - .encrypt = ctr_crypt, - .decrypt = ctr_crypt, }, };
arch/x86/crypto/serpent_avx_glue.c +0 -51
75 lines not on this page · raw message
arch/x86/crypto/serpent_sse2_glue.c +0 -67
97 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 9/10 crypto: x86/twofish - drop CTR mode implementation
In reply to Ard BiesheuvelTwofish in CTR mode is never used by the kernel directly, and is highly unlikely to be relied upon by dm-crypt or algif_skcipher. So let's drop the accelerated CTR mode implementation, and instead, rely on the CTR template and the bare cipher.
--- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 27 -------- arch/x86/crypto/twofish_avx_glue.c | 38 ----------- arch/x86/crypto/twofish_glue_3way.c | 72 -------------------- arch/x86/include/asm/crypto/twofish.h | 4 -- 4 files changed, 141 deletions(-)
arch/x86/crypto/twofish-avx-x86_64-asm_64.S +0 -27
@@ -374,30 +374,3 @@ SYM_FUNC_START(twofish_cbc_dec_8way) FRAME_END ret; SYM_FUNC_END(twofish_cbc_dec_8way) - -SYM_FUNC_START(twofish_ctr_8way) - /* input: - * %rdi: ctx, CTX - * %rsi: dst - * %rdx: src - * %rcx: iv (little endian, 128bit) - */ - FRAME_BEGIN - - pushq %r12; - - movq %rsi, %r11; - movq %rdx, %r12; - - load_ctr_8way(%rcx, .Lbswap128_mask, RA1, RB1, RC1, RD1, RA2, RB2, RC2, - RD2, RX0, RX1, RY0); - - call __twofish_enc_blk8; - - store_ctr_8way(%r12, %r11, RC1, RD1, RA1, RB1, RC2, RD2, RA2, RB2); - - popq %r12; - - FRAME_END - ret; -SYM_FUNC_END(twofish_ctr_8way)
arch/x86/crypto/twofish_avx_glue.c +0 -38
@@ -25,8 +25,6 @@ asmlinkage void twofish_ecb_enc_8way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void twofish_ecb_dec_8way(const void *ctx, u8 *dst, const u8 *src); asmlinkage void twofish_cbc_dec_8way(const void *ctx, u8 *dst, const u8 *src); -asmlinkage void twofish_ctr_8way(const void *ctx, u8 *dst, const u8 *src, -le128 *iv);static int twofish_setkey_skcipher (struct crypto_skcipher *tfm,const u8 *key,unsigned int keylen)
@@ -55,22 +53,6 @@ static const struct common_glue_ctx twofish_enc = { } } }; -static const struct common_glue_ctx twofish_ctr = { - .num_funcs =3 , - .fpu_blocks_limit =TWOFISH_PARALLEL_BLOCKS , - - .funcs = { { - .num_blocks =TWOFISH_PARALLEL_BLOCKS , - .fn_u = { .ctr = twofish_ctr_8way } - }, { - .num_blocks =3 , - .fn_u = { .ctr = twofish_enc_blk_ctr_3way } - }, { - .num_blocks =1 , - .fn_u = { .ctr = twofish_enc_blk_ctr } - } } -}; -static const struct common_glue_ctx twofish_dec = { .num_funcs =3 , .fpu_blocks_limit =TWOFISH_PARALLEL_BLOCKS ,
@@ -123,11 +105,6 @@ static int cbc_decrypt(struct skcipher_request *req)return glue_cbc_decrypt_req_128bit (&twofish_dec_cbc, req); } -static int ctr_crypt (struct skcipher_request *req) -{ -return glue_ctr_req_128bit (&twofish_ctr, req); -} -static struct skcipher_alg twofish_algs[] = { { .base.cra_name ="__ecb(twofish)" ,
@@ -156,21 +133,6 @@ static struct skcipher_alg twofish_algs[] = { .setkey = twofish_setkey_skcipher, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, - }, { - .base.cra_name = "__ctr(twofish)", - .base.cra_driver_name = "__ctr-twofish-avx", - .base.cra_priority = 400, - .base.cra_flags = CRYPTO_ALG_INTERNAL, - .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct twofish_ctx), - .base.cra_module = THIS_MODULE, - .min_keysize = TF_MIN_KEY_SIZE, - .max_keysize = TF_MAX_KEY_SIZE, - .ivsize = TF_BLOCK_SIZE, - .chunksize = TF_BLOCK_SIZE, - .setkey = twofish_setkey_skcipher, - .encrypt = ctr_crypt, - .decrypt = ctr_crypt, }, };
arch/x86/crypto/twofish_glue_3way.c +0 -72
96 lines not on this page · raw message
arch/x86/include/asm/crypto/twofish.h +0 -4
@@ -17,9 +17,5 @@ asmlinkage void twofish_dec_blk_3way(const void *ctx, u8 *dst, const u8 *src);/* helpers from twofish_x86_64-3way module */ extern void twofish_dec_blk_cbc_3way (const void *ctx,u8 *dst,const u8 *src); -extern void twofish_enc_blk_ctr (const void *ctx,u8 *dst,const u8 *src, -le128 *iv); -extern void twofish_enc_blk_ctr_3way (const void *ctx,u8 *dst,const u8 *src, -le128 *iv);#endif /* ASM_X86_TWOFISH_H */
Signature · 1 line
2.17.1
RFC PATCH 10/10 crypto: x86/glue-helper - drop CTR helper routines
In reply to Ard BiesheuvelThe glue helper's CTR routines are no longer used, so drop them.
--- arch/x86/crypto/glue_helper-asm-avx.S | 45 ------------ arch/x86/crypto/glue_helper-asm-avx2.S | 58 ---------------- arch/x86/crypto/glue_helper.c | 72 -------------------- arch/x86/include/asm/crypto/glue_helper.h | 32 --------- 4 files changed, 207 deletions(-)
arch/x86/crypto/glue_helper-asm-avx.S +0 -45
@@ -34,48 +34,3 @@ vpxor (5*16)(src), x6, x6; \ vpxor (6*16)(src), x7, x7; \ store_8way(dst, x0, x1, x2, x3, x4, x5, x6, x7); - -#define inc_le128(x, minus_one, tmp) \ - vpcmpeqq minus_one, x, tmp; \ - vpsubq minus_one, x, x; \ - vpslldq $8, tmp, tmp; \ - vpsubq tmp, x, x; - -#define load_ctr_8way(iv, bswap, x0, x1, x2, x3, x4, x5, x6, x7, t0, t1, t2) \ - vpcmpeqd t0, t0, t0; \ - vpsrldq $8, t0, t0; /* low: -1, high: 0 */ \ - vmovdqa bswap, t1; \ - \ - /* load IV and byteswap */ \ - vmovdqu (iv), x7; \ - vpshufb t1, x7, x0; \ - \ - /* construct IVs */ \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x1; \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x2; \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x3; \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x4; \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x5; \ - inc_le128(x7, t0, t2); \ - vpshufb t1, x7, x6; \ - inc_le128(x7, t0, t2); \ - vmovdqa x7, t2; \ - vpshufb t1, x7, x7; \ - inc_le128(t2, t0, t1); \ - vmovdqu t2, (iv); - -#define store_ctr_8way(src, dst, x0, x1, x2, x3, x4, x5, x6, x7) \ - vpxor (0*16)(src), x0, x0; \ - vpxor (1*16)(src), x1, x1; \ - vpxor (2*16)(src), x2, x2; \ - vpxor (3*16)(src), x3, x3; \ - vpxor (4*16)(src), x4, x4; \ - vpxor (5*16)(src), x5, x5; \ - vpxor (6*16)(src), x6, x6; \ - vpxor (7*16)(src), x7, x7; \ - store_8way(dst, x0, x1, x2, x3, x4, x5, x6, x7);
arch/x86/crypto/glue_helper-asm-avx2.S +0 -58
61 lines not on this page · raw message
arch/x86/crypto/glue_helper.c +0 -72
82 lines not on this page · raw message
arch/x86/include/asm/crypto/glue_helper.h +0 -32
@@ -9,19 +9,15 @@#include <crypto/internal/skcipher.h> #include <linux/kernel.h> #include <asm/fpu/api.h> -#include <crypto/b128ops.h> typedef void (*common_glue_func_t )(const void *ctx,u8 *dst,const u8 *src);typedef void (*common_glue_cbc_func_t )(const void *ctx,u8 *dst,const u8 *src); -typedef void (*common_glue_ctr_func_t )(const void *ctx,u8 *dst,const u8 *src, -le128 *iv);struct common_glue_func_entry {unsigned int num_blocks;/* number of blocks that @fn will process */ union {common_glue_func_t ecb;common_glue_cbc_func_t cbc; -common_glue_ctr_func_t ctr; } fn_u; };
@@ -66,31 +62,6 @@ static inline void glue_fpu_end(bool fpu_enabled)kernel_fpu_end (); } -static inline void le128_to_be128 (be128 *dst,const le128 *src) -{ - dst->a =cpu_to_be64 (le64_to_cpu (src->a)); - dst->b =cpu_to_be64 (le64_to_cpu (src->b)); -} - -static inline void be128_to_le128 (le128 *dst,const be128 *src) -{ - dst->a =cpu_to_le64 (be64_to_cpu (src->a)); - dst->b =cpu_to_le64 (be64_to_cpu (src->b)); -} - -static inline void le128_inc (le128 *i) -{ -u64 a =le64_to_cpu (i->a); -u64 b =le64_to_cpu (i->b); - - b++; -if (!b) - a++; - - i->a =cpu_to_le64 (a); - i->b =cpu_to_le64 (b); -} -extern int glue_ecb_req_128bit (const struct common_glue_ctx *gctx,struct skcipher_request *req);
@@ -100,7 +71,4 @@ extern int glue_cbc_encrypt_req_128bit(const common_glue_func_t fn,extern int glue_cbc_decrypt_req_128bit (const struct common_glue_ctx *gctx,struct skcipher_request *req); -extern int glue_ctr_req_128bit (const struct common_glue_ctx *gctx, -struct skcipher_request *req); -#endif /* _CRYPTO_GLUE_HELPER_H */
Signature · 1 line
2.17.1
Quoted 9 lines · Ard Biesheuvel
After applying my performance fixes for AES-NI in XTS mode, the only remaining users of the x86 glue helper module are the niche algorithms camellia, cast6, serpent and twofish.
It is not clear from the history why all these different versions of these algorithms in XTS and CTR modes were added in the first place: the only in-kernel references that seem to exist are to cbc(serpent), cbc(camellia) and cbc(twofish) in the IPsec stack. The XTS spec only mentions AES, and CTR modes don't seem to be widely used either.
FYI: Serpent, Camellia and Twofish are used in TrueCrypt/VeraCrypt implementation; cryptsetup and I perhaps even VeraCrypt itself tries to use native dm-crypt mapping. (They also added Russian GOST Kuznyechik with XTS, but this is not in mainline, but Debian packages it as gost-crypto-dkms).
Serpent and Twofish can be also used with LRW and CBC modes (for old containers only).
Cryptsetup uses crypto userspace API to decrypt the key from header, then it configures dm-crypt mapping for data. We need both use and in-kernel API here.
For reference, see this table (my independent implementation of TrueCrypt/VeraCrypt modes, it should be complete history though):
https://gitlab.com/cryptsetup/cryptsetup/-/blob/master/lib/tcrypt/tcrypt.c#L77
If the above still works (I would really like to have way to open old containers) it is ok to do whatever you want to change here :-)
I have no info that CTR is used anywhere related to dm-crypt (IIRC it can be tricked to be used there but it does not make any sense).
Thanks,
Milan
--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
Quoted 22 lines · Milan Broz
Quoted 9 lines · Ard Biesheuvel
After applying my performance fixes for AES-NI in XTS mode, the only remaining users of the x86 glue helper module are the niche algorithms camellia, cast6, serpent and twofish.
It is not clear from the history why all these different versions of these algorithms in XTS and CTR modes were added in the first place: the only in-kernel references that seem to exist are to cbc(serpent), cbc(camellia) and cbc(twofish) in the IPsec stack. The XTS spec only mentions AES, and CTR modes don't seem to be widely used either.
FYI: Serpent, Camellia and Twofish are used in TrueCrypt/VeraCrypt implementation; cryptsetup and I perhaps even VeraCrypt itself tries to use native dm-crypt mapping. (They also added Russian GOST Kuznyechik with XTS, but this is not in mainline, but Debian packages it as gost-crypto-dkms).
Serpent and Twofish can be also used with LRW and CBC modes (for old containers only).
Cryptsetup uses crypto userspace API to decrypt the key from header, then it configures dm-crypt mapping for data. We need both use and in-kernel API here.
Quoted 7 lines
For reference, see this table (my independent implementation of TrueCrypt/VeraCrypt modes, it should be complete history though):
https://gitlab.com/cryptsetup/cryptsetup/-/blob/master/lib/tcrypt/tcrypt.c#L77If the above still works (I would really like to have way to open old containers) it is ok to do whatever you want to change here :-)
Thanks Milan.
With the XTS code removed from these drivers, the XTS template will be used, which relies on the ECB mode helpers instead. So once we fix those to get rid of the indirect calls, I'd expect XTS to actually improve in performance for these algorithms.
Quoted 3 lines
I have no info that CTR is used anywhere related to dm-crypt (IIRC it can be tricked to be used there but it does not make any sense).
Yes, that was my assumption. Thanks for confirming.
Quoted 41 lines · Ard Biesheuvel
After applying my performance fixes for AES-NI in XTS mode, the only remaining users of the x86 glue helper module are the niche algorithms camellia, cast6, serpent and twofish.
It is not clear from the history why all these different versions of these algorithms in XTS and CTR modes were added in the first place: the only in-kernel references that seem to exist are to cbc(serpent), cbc(camellia) and cbc(twofish) in the IPsec stack. The XTS spec only mentions AES, and CTR modes don't seem to be widely used either.
Since the glue helper code relies heavily on indirect calls for small chunks of in/output, it needs some work to recover from the performance hit caused by the retpoline changes. However, it makes sense to only expend the effort for algorithms that are being used in the first place, and this does not seem to be the case for XTS and CTR.
CTR mode can simply be removed: it is not used in the kernel, and it is highly unlikely that it is being relied upon via
algif_skcipher. And even if it was, the generic CTR mode driver can still provide the CTR transforms if necessary.XTS mode may actually be in use by dm-crypt users, so we cannot simply drop this code entirely. However, as it turns out, the XTS template wrapped around the ECB mode skciphers perform roughly on par *, and so there is no need to retain all the complicated XTS helper logic. In the unlikely case that dm-crypt users are relying on xts(camellia) or xts(serpent) in the field, they should not be impacted by these changes at all.
As a follow-up, it makes sense to rework the ECB and CBC mode implementations to get rid of the indirect calls. Or perhaps we could drop [some of] these algorithms entirely ...
* tcrypt results for various XTS implementations below, captured on a Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
Cc Megha Dey <[email protected]>Cc Eric Biggers <[email protected]>Cc Herbert Xu <[email protected]>Cc Milan Broz <[email protected]>Cc Mike Snitzer <[email protected]>
Quoted 11 lines
Ard Biesheuvel (10):
crypto: x86/camellia - switch to XTS template
crypto: x86/cast6 - switch to XTS template
crypto: x86/serpent- switch to XTS template
crypto: x86/twofish - switch to XTS template
crypto: x86/glue-helper - drop XTS helper routines
crypto: x86/camellia - drop CTR mode implementation
crypto: x86/cast6 - drop CTR mode implementation
crypto: x86/serpent - drop CTR mode implementation
crypto: x86/twofish - drop CTR mode implementation
crypto: x86/glue-helper - drop CTR helper routines
- Eric