crypto: x86/aes-ni-xts - recover and improve performance
Subsystems (2)
cover crypto: x86/aes-ni-xts - recover and improve performance
The AES-NI implementation of XTS was impacted significantly by the retpoline changes, which is due to the fact that both its asm helper and the chaining mode glue library use indirect calls for processing small quantitities of data
So let's fix this, by:
- creating a minimal, backportable fix that recovers most of the performance, by reducing the number of indirect calls substantially;
- for future releases, rewrite the XTS implementation completely, and replace the glue helper with a core asm routine that is more flexible, making the C code wrapper much more straight-forward.
This results in a substantial performance improvement: around ~2x for 1k and 4k blocks, and more than 3x for ~1k blocks that require ciphertext stealing (benchmarked using tcrypt using 1420 byte blocks - full results below)
It also allows us to enable the same driver for i386.
Ard Biesheuvel (2):
crypto: x86/aes-ni-xts - use direct calls to and 4-way stride
crypto: x86/aes-ni-xts - rewrite and drop indirections via glue helper
arch/x86/crypto/aesni-intel_asm.S | 353 ++++++++++++++++---- arch/x86/crypto/aesni-intel_glue.c | 230 +++++++------ 2 files changed, 412 insertions(+), 171 deletions(-)
Signature · 37 lines
2.17.1 Benchmarked using tcrypt on a Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz. Requires patch below to get tcrypt to benchmark 1420 byte blocks. BEFORE testing speed of async xts(aes) (xts-aes-aesni) encryption tcrypt: test 0 (256 bit key, 16 byte blocks): 8030565 operations in 1 seconds (128489040 bytes) tcrypt: test 1 (256 bit key, 64 byte blocks): 4760527 operations in 1 seconds (304673728 bytes) tcrypt: test 2 (256 bit key, 256 byte blocks): 5250541 operations in 1 seconds (1344138496 bytes) tcrypt: test 3 (256 bit key, 1024 byte blocks): 2163398 operations in 1 seconds (2215319552 bytes) tcrypt: test 4 (256 bit key, 1420 byte blocks): 1036396 operations in 1 seconds (1471682320 bytes) tcrypt: test 5 (256 bit key, 4096 byte blocks): 568192 operations in 1 seconds (2327314432 bytes) tcrypt: test 6 (512 bit key, 16 byte blocks): 7916395 operations in 1 seconds (126662320 bytes) tcrypt: test 7 (512 bit key, 64 byte blocks): 4783114 operations in 1 seconds (306119296 bytes) tcrypt: test 8 (512 bit key, 256 byte blocks): 4916568 operations in 1 seconds (1258641408 bytes) tcrypt: test 9 (512 bit key, 1024 byte blocks): 1898349 operations in 1 seconds (1943909376 bytes) tcrypt: test 10 (512 bit key, 1420 byte blocks): 970328 operations in 1 seconds (1377865760 bytes) tcrypt: test 11 (512 bit key, 4096 byte blocks): 499687 operations in 1 seconds (2046717952 bytes) AFTER testing speed of async xts(aes) (xts-aes-aesni) encryption tcrypt: test 0 (256 bit key, 16 byte blocks): 11977048 operations in 1 seconds (191632768 bytes) tcrypt: test 1 (256 bit key, 64 byte blocks): 10504479 operations in 1 seconds (672286656 bytes) tcrypt: test 2 (256 bit key, 256 byte blocks): 7929809 operations in 1 seconds (2030031104 bytes) tcrypt: test 3 (256 bit key, 1024 byte blocks): 3992118 operations in 1 seconds (4087928832 bytes) tcrypt: test 4 (256 bit key, 1420 byte blocks): 3160481 operations in 1 seconds (4487883020 bytes) tcrypt: test 5 (256 bit key, 4096 byte blocks): 1240437 operations in 1 seconds (5080829952 bytes) tcrypt: test 6 (512 bit key, 16 byte blocks): 11694652 operations in 1 seconds (187114432 bytes) tcrypt: test 7 (512 bit key, 64 byte blocks): 9739536 operations in 1 seconds (623330304 bytes) tcrypt: test 8 (512 bit key, 256 byte blocks): 6833613 operations in 1 seconds (1749404928 bytes) tcrypt: test 9 (512 bit key, 1024 byte blocks): 3121421 operations in 1 seconds (3196335104 bytes) tcrypt: test 10 (512 bit key, 1420 byte blocks): 2421563 operations in 1 seconds (3438619460 bytes) tcrypt: test 11 (512 bit key, 4096 byte blocks): 941964 operations in 1 seconds (3858284544 bytes)
arch/x86/crypto/aesni-intel_glue.c +2 -1
@@ -994,12 +994,13 @@ static struct skcipher_alg aesni_skciphers[] = { .cra_driver_name = "__xts-aes-aesni", .cra_priority = 401, .cra_flags = CRYPTO_ALG_INTERNAL, - .cra_blocksize = AES_BLOCK_SIZE, + .cra_blocksize = 1,//AES_BLOCK_SIZE, .cra_ctxsize = XTS_AES_CTX_SIZE, .cra_module = THIS_MODULE, }, .min_keysize = 2 * AES_MIN_KEY_SIZE, .max_keysize = 2 * AES_MAX_KEY_SIZE, + .chunksize = AES_BLOCK_SIZE, .ivsize = AES_BLOCK_SIZE, .setkey = xts_aesni_setkey, .encrypt = xts_encrypt,
crypto/xts.c +2 -1
@@ -416,11 +416,12 @@ static int xts_create(struct crypto_template *tmpl, struct rtattr **tb) goto err_free_inst; inst->alg.base.cra_priority = alg->base.cra_priority; - inst->alg.base.cra_blocksize =XTS_BLOCK_SIZE ; + inst->alg.base.cra_blocksize =1 ,//XTS_BLOCK_SIZE; inst->alg.base.cra_alignmask = alg->base.cra_alignmask | (__alignof__(u64 ) -1 ); inst->alg.ivsize =XTS_BLOCK_SIZE ; + inst->alg.chunksize =XTS_BLOCK_SIZE ; inst->alg.min_keysize =crypto_skcipher_alg_min_keysize (alg) *2 ; inst->alg.max_keysize =crypto_skcipher_alg_max_keysize (alg) *2 ;
PATCH 1/2 crypto: x86/aes-ni-xts - use direct calls to and 4-way stride
The XTS asm helper arrangement is a bit odd: the 8-way stride helper consists of back-to-back calls to the 4-way core transforms, which are called indirectly, based on a boolean that indicates whether we are performing encryption or decryption.
Given how costly indirect calls are on x86, let's switch to direct calls, and given how the 8-way stride doesn't really add anything substantial, use a 4-way stride instead, and make the asm core routine deal with any multiple of 4 blocks. Since 512 byte sectors or 4 KB blocks are the typical quantities XTS operates on, increase the stride exported to the glue helper to 512 bytes as well.
As a result, the number of indirect calls is reduced from 3 per 64 bytes of in/output to 1 per 512 bytes of in/output, which produces a 65% speedup when operating on 1 KB blocks (measured on a Intel(R) Core(TM) i7-8650U CPU)
--- arch/x86/crypto/aesni-intel_asm.S | 115 ++++++++++++-------- arch/x86/crypto/aesni-intel_glue.c | 25 +++-- 2 files changed, 84 insertions(+), 56 deletions(-)
arch/x86/crypto/aesni-intel_asm.S +70 -45
@@ -2842,25 +2842,18 @@ SYM_FUNC_END(aesni_ctr_enc) pxor CTR, IV; /* - * void aesni_xts_crypt8(const struct crypto_aes_ctx *ctx, u8 *dst, - * const u8 *src, bool enc, le128 *iv) + * void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) */ -SYM_FUNC_START(aesni_xts_crypt8) +SYM_FUNC_START(aesni_xts_encrypt) FRAME_BEGIN - testb %cl, %cl - movl $0, %ecx - movl $240, %r10d - leaq _aesni_enc4, %r11 - leaq _aesni_dec4, %rax - cmovel %r10d, %ecx - cmoveq %rax, %r11 movdqa .Lgf128mul_x_ble_mask, GF128MUL_MASK movups (IVP), IV mov 480(KEYP), KLEN - addq %rcx, KEYP +.Lxts_enc_loop4: movdqa IV, STATE1 movdqu 0x00(INP), INC pxor INC, STATE1
@@ -2884,71 +2877,103 @@ SYM_FUNC_START(aesni_xts_crypt8) pxor INC, STATE4 movdqu IV, 0x30(OUTP) - CALL_NOSPEC r11 + call _aesni_enc4 movdqu 0x00(OUTP), INC pxor INC, STATE1 movdqu STATE1, 0x00(OUTP) - _aesni_gf128mul_x_ble() - movdqa IV, STATE1 - movdqu 0x40(INP), INC - pxor INC, STATE1 - movdqu IV, 0x40(OUTP) - movdqu 0x10(OUTP), INC pxor INC, STATE2 movdqu STATE2, 0x10(OUTP) - _aesni_gf128mul_x_ble() - movdqa IV, STATE2 - movdqu 0x50(INP), INC - pxor INC, STATE2 - movdqu IV, 0x50(OUTP) - movdqu 0x20(OUTP), INC pxor INC, STATE3 movdqu STATE3, 0x20(OUTP) - _aesni_gf128mul_x_ble() - movdqa IV, STATE3 - movdqu 0x60(INP), INC - pxor INC, STATE3 - movdqu IV, 0x60(OUTP) - movdqu 0x30(OUTP), INC pxor INC, STATE4 movdqu STATE4, 0x30(OUTP) _aesni_gf128mul_x_ble() - movdqa IV, STATE4 - movdqu 0x70(INP), INC - pxor INC, STATE4 - movdqu IV, 0x70(OUTP) - _aesni_gf128mul_x_ble() + add $64, INP + add $64, OUTP + sub $64, LEN + ja .Lxts_enc_loop4 + movups IV, (IVP) - CALL_NOSPEC r11 + FRAME_END + ret +SYM_FUNC_END(aesni_xts_encrypt) + +/* + * void aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) + */ +SYM_FUNC_START(aesni_xts_decrypt) + FRAME_BEGIN + + movdqa .Lgf128mul_x_ble_mask, GF128MUL_MASK + movups (IVP), IV + + mov 480(KEYP), KLEN + add $240, KEYP - movdqu 0x40(OUTP), INC +.Lxts_dec_loop4: + movdqa IV, STATE1 + movdqu 0x00(INP), INC pxor INC, STATE1 - movdqu STATE1, 0x40(OUTP) + movdqu IV, 0x00(OUTP) - movdqu 0x50(OUTP), INC + _aesni_gf128mul_x_ble() + movdqa IV, STATE2 + movdqu 0x10(INP), INC + pxor INC, STATE2 + movdqu IV, 0x10(OUTP) + + _aesni_gf128mul_x_ble() + movdqa IV, STATE3 + movdqu 0x20(INP), INC + pxor INC, STATE3 + movdqu IV, 0x20(OUTP) + + _aesni_gf128mul_x_ble() + movdqa IV, STATE4 + movdqu 0x30(INP), INC + pxor INC, STATE4 + movdqu IV, 0x30(OUTP) + + call _aesni_dec4 + + movdqu 0x00(OUTP), INC + pxor INC, STATE1 + movdqu STATE1, 0x00(OUTP) + + movdqu 0x10(OUTP), INC pxor INC, STATE2 - movdqu STATE2, 0x50(OUTP) + movdqu STATE2, 0x10(OUTP) - movdqu 0x60(OUTP), INC + movdqu 0x20(OUTP), INC pxor INC, STATE3 - movdqu STATE3, 0x60(OUTP) + movdqu STATE3, 0x20(OUTP) - movdqu 0x70(OUTP), INC + movdqu 0x30(OUTP), INC pxor INC, STATE4 - movdqu STATE4, 0x70(OUTP) + movdqu STATE4, 0x30(OUTP) + + _aesni_gf128mul_x_ble() + + add $64, INP + add $64, OUTP + sub $64, LEN + ja .Lxts_dec_loop4 + + movups IV, (IVP) FRAME_END ret -SYM_FUNC_END(aesni_xts_crypt8) +SYM_FUNC_END(aesni_xts_decrypt) #endif
arch/x86/crypto/aesni-intel_glue.c +14 -11
@@ -101,6 +101,12 @@ asmlinkage void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out,#define AVX_GEN2_OPTSIZE 640#define AVX_GEN4_OPTSIZE 4096 +asmlinkage void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, +const u8 *in,unsigned int len,u8 *iv); + +asmlinkage void aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, +const u8 *in,unsigned int len,u8 *iv); +#ifdef CONFIG_X86_64 static void (*aesni_ctr_enc_tfm)(struct crypto_aes_ctx *ctx, u8 *out,
@@ -108,9 +114,6 @@ static void (*aesni_ctr_enc_tfm)(struct crypto_aes_ctx *ctx, u8 *out, asmlinkage void aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out,const u8 *in,unsigned int len,u8 *iv); -asmlinkage void aesni_xts_crypt8(const struct crypto_aes_ctx *ctx, u8 *out, -const u8 *in,bool enc,le128 *iv); -/* Scatter / Gather routines, with args similar to above */ asmlinkage void aesni_gcm_init(void *ctx, struct gcm_context_data *gdata,
@@ -596,14 +599,14 @@ static void aesni_xts_dec(const void *ctx, u8 *dst, const u8 *src, le128 *iv)glue_xts_crypt_128bit_one (ctx, dst, src, iv, aesni_dec); } -static void aesni_xts_enc8 (const void *ctx,u8 *dst,const u8 *src,le128 *iv) +static void aesni_xts_enc32 (const void *ctx,u8 *dst,const u8 *src,le128 *iv) { -aesni_xts_crypt8 (ctx, dst, src, true, iv); +aesni_xts_encrypt (ctx, dst, src,32 *AES_BLOCK_SIZE , (u8 *)iv); } -static void aesni_xts_dec8 (const void *ctx,u8 *dst,const u8 *src,le128 *iv) +static void aesni_xts_dec32 (const void *ctx,u8 *dst,const u8 *src,le128 *iv) { -aesni_xts_crypt8 (ctx, dst, src, false, iv); +aesni_xts_decrypt (ctx, dst, src,32 *AES_BLOCK_SIZE , (u8 *)iv); }static const struct common_glue_ctx aesni_enc_xts = {
@@ -611,8 +614,8 @@ static const struct common_glue_ctx aesni_enc_xts = { .fpu_blocks_limit = 1, .funcs = { { - .num_blocks = 8, - .fn_u = { .xts = aesni_xts_enc8 } + .num_blocks = 32, + .fn_u = { .xts = aesni_xts_enc32 } }, { .num_blocks = 1, .fn_u = { .xts = aesni_xts_enc }
@@ -624,8 +627,8 @@ static const struct common_glue_ctx aesni_dec_xts = { .fpu_blocks_limit = 1, .funcs = { { - .num_blocks = 8, - .fn_u = { .xts = aesni_xts_dec8 } + .num_blocks = 32, + .fn_u = { .xts = aesni_xts_dec32 } }, { .num_blocks = 1, .fn_u = { .xts = aesni_xts_dec }
Signature · 1 line
2.17.1
PATCH 2/2 crypto: x86/aes-ni-xts - rewrite and drop indirections via glue helper
In reply to Ard BiesheuvelThe AES-NI driver implements XTS via the glue helper, which consumes a struct with sets of function pointers which are invoked on chunks of input data of the appropriate size, as annotated in the struct.
Let's get rid of this indirection, so that we can perform direct calls to the assembler helpers. Instead, let's adopt the arm64 strategy, i.e., provide a helper which can consume inputs of any size, provided that the penultimate, full block is passed via the last call if ciphertext stealing needs to be applied.
This also allows us to enable the XTS mode for i386.
--- arch/x86/crypto/aesni-intel_asm.S | 280 ++++++++++++++++---- arch/x86/crypto/aesni-intel_glue.c | 221 ++++++++------- 2 files changed, 357 insertions(+), 144 deletions(-)
arch/x86/crypto/aesni-intel_asm.S +232 -48
@@ -43,10 +43,6 @@ #ifdef __x86_64__ # constants in mergeable sections, linker can reorder and merge -.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 -.align 16 -.Lgf128mul_x_ble_mask: - .octa 0x00000000000000010000000000000087 .section .rodata.cst16.POLY, "aM", @progbits, 16 .align 16 POLY: .octa 0xC2000000000000000000000000000001
@@ -146,7 +142,7 @@ ALL_F: .octa 0xffffffffffffffffffffffffffffffff #define CTR %xmm11 #define INC %xmm12 -#define GF128MUL_MASK %xmm10 +#define GF128MUL_MASK %xmm7 #ifdef __x86_64__ #define AREG %rax
@@ -2823,6 +2819,14 @@ SYM_FUNC_START(aesni_ctr_enc) ret SYM_FUNC_END(aesni_ctr_enc) +#endif + +.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 +.align 16 +.Lgf128mul_x_ble_mask: + .octa 0x00000000000000010000000000000087 +.previous + /* * _aesni_gf128mul_x_ble: internal ABI * Multiply in GF(2^128) for XTS IVs
@@ -2835,11 +2839,11 @@ SYM_FUNC_END(aesni_ctr_enc) * CTR: == temporary value */ #define _aesni_gf128mul_x_ble() \ - pshufd $0x13, IV, CTR; \ + pshufd $0x13, IV, KEY; \ paddq IV, IV; \ - psrad $31, CTR; \ - pand GF128MUL_MASK, CTR; \ - pxor CTR, IV; + psrad $31, KEY; \ + pand GF128MUL_MASK, KEY; \ + pxor KEY, IV; /* * void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst,
@@ -2847,65 +2851,153 @@ SYM_FUNC_END(aesni_ctr_enc) */ SYM_FUNC_START(aesni_xts_encrypt) FRAME_BEGIN - +#ifndef __x86_64__ + pushl IVP + pushl LEN + pushl KEYP + pushl KLEN + movl (FRAME_OFFSET+20)(%esp), KEYP # ctx + movl (FRAME_OFFSET+24)(%esp), OUTP # dst + movl (FRAME_OFFSET+28)(%esp), INP # src + movl (FRAME_OFFSET+32)(%esp), LEN # len + movl (FRAME_OFFSET+36)(%esp), IVP # iv movdqa .Lgf128mul_x_ble_mask, GF128MUL_MASK +#else + movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK +#endif movups (IVP), IV mov 480(KEYP), KLEN .Lxts_enc_loop4: + sub $64, LEN + jl .Lxts_enc_1x + movdqa IV, STATE1 - movdqu 0x00(INP), INC - pxor INC, STATE1 + movdqu 0x00(INP), IN + pxor IN, STATE1 movdqu IV, 0x00(OUTP) _aesni_gf128mul_x_ble() movdqa IV, STATE2 - movdqu 0x10(INP), INC - pxor INC, STATE2 + movdqu 0x10(INP), IN + pxor IN, STATE2 movdqu IV, 0x10(OUTP) _aesni_gf128mul_x_ble() movdqa IV, STATE3 - movdqu 0x20(INP), INC - pxor INC, STATE3 + movdqu 0x20(INP), IN + pxor IN, STATE3 movdqu IV, 0x20(OUTP) _aesni_gf128mul_x_ble() movdqa IV, STATE4 - movdqu 0x30(INP), INC - pxor INC, STATE4 + movdqu 0x30(INP), IN + pxor IN, STATE4 movdqu IV, 0x30(OUTP) call _aesni_enc4 - movdqu 0x00(OUTP), INC - pxor INC, STATE1 + movdqu 0x00(OUTP), IN + pxor IN, STATE1 movdqu STATE1, 0x00(OUTP) - movdqu 0x10(OUTP), INC - pxor INC, STATE2 + movdqu 0x10(OUTP), IN + pxor IN, STATE2 movdqu STATE2, 0x10(OUTP) - movdqu 0x20(OUTP), INC - pxor INC, STATE3 + movdqu 0x20(OUTP), IN + pxor IN, STATE3 movdqu STATE3, 0x20(OUTP) - movdqu 0x30(OUTP), INC - pxor INC, STATE4 + movdqu 0x30(OUTP), IN + pxor IN, STATE4 movdqu STATE4, 0x30(OUTP) _aesni_gf128mul_x_ble() add $64, INP add $64, OUTP - sub $64, LEN - ja .Lxts_enc_loop4 + cmp $0, LEN + jnz .Lxts_enc_loop4 +.Lxts_enc_ret_iv: movups IV, (IVP) +.Lxts_enc_ret: +#ifndef __x86_64__ + popl KLEN + popl KEYP + popl LEN + popl IVP +#endif FRAME_END ret + +.Lxts_enc_1x: + add $64, LEN + jz .Lxts_enc_ret_iv + sub $16, LEN + jl .Lxts_enc_cts4 + +.Lxts_enc_loop1: + movdqu (INP), STATE + pxor IV, STATE + call _aesni_enc1 + pxor IV, STATE + _aesni_gf128mul_x_ble() + + cmp $0, LEN + jz .Lxts_enc_out + + add $16, INP + sub $16, LEN + jl .Lxts_enc_cts1 + + movdqu STATE, (OUTP) + add $16, OUTP + jmp .Lxts_enc_loop1 + +.Lxts_enc_out: + movdqu STATE, (OUTP) + jmp .Lxts_enc_ret_iv + +.Lxts_enc_cts4: + movdqa STATE4, STATE + sub $16, OUTP + +.Lxts_enc_cts1: +#ifndef __x86_64__ + lea .Lcts_permute_table, T1 +#else + lea .Lcts_permute_table(%rip), T1 +#endif + add LEN, INP /* rewind input pointer */ + add $16, LEN /* # bytes in final block */ + movups (INP), IN1 + + mov T1, IVP + add $32, IVP + add LEN, T1 + sub LEN, IVP + add OUTP, LEN + + movups (T1), %xmm4 + movaps STATE, IN2 + pshufb %xmm4, STATE + movups STATE, (LEN) + + movups (IVP), %xmm0 + pshufb %xmm0, IN1 + pblendvb IN2, IN1 + movaps IN1, STATE + + pxor IV, STATE + call _aesni_enc1 + pxor IV, STATE + + movups STATE, (OUTP) + jmp .Lxts_enc_ret SYM_FUNC_END(aesni_xts_encrypt) /*
@@ -2914,66 +3006,158 @@ SYM_FUNC_END(aesni_xts_encrypt) */ SYM_FUNC_START(aesni_xts_decrypt) FRAME_BEGIN - +#ifndef __x86_64__ + pushl IVP + pushl LEN + pushl KEYP + pushl KLEN + movl (FRAME_OFFSET+20)(%esp), KEYP # ctx + movl (FRAME_OFFSET+24)(%esp), OUTP # dst + movl (FRAME_OFFSET+28)(%esp), INP # src + movl (FRAME_OFFSET+32)(%esp), LEN # len + movl (FRAME_OFFSET+36)(%esp), IVP # iv movdqa .Lgf128mul_x_ble_mask, GF128MUL_MASK +#else + movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK +#endif movups (IVP), IV mov 480(KEYP), KLEN add $240, KEYP + test $15, LEN + jz .Lxts_dec_loop4 + sub $16, LEN + .Lxts_dec_loop4: + sub $64, LEN + jl .Lxts_dec_1x + movdqa IV, STATE1 - movdqu 0x00(INP), INC - pxor INC, STATE1 + movdqu 0x00(INP), IN + pxor IN, STATE1 movdqu IV, 0x00(OUTP) _aesni_gf128mul_x_ble() movdqa IV, STATE2 - movdqu 0x10(INP), INC - pxor INC, STATE2 + movdqu 0x10(INP), IN + pxor IN, STATE2 movdqu IV, 0x10(OUTP) _aesni_gf128mul_x_ble() movdqa IV, STATE3 - movdqu 0x20(INP), INC - pxor INC, STATE3 + movdqu 0x20(INP), IN + pxor IN, STATE3 movdqu IV, 0x20(OUTP) _aesni_gf128mul_x_ble() movdqa IV, STATE4 - movdqu 0x30(INP), INC - pxor INC, STATE4 + movdqu 0x30(INP), IN + pxor IN, STATE4 movdqu IV, 0x30(OUTP) call _aesni_dec4 - movdqu 0x00(OUTP), INC - pxor INC, STATE1 + movdqu 0x00(OUTP), IN + pxor IN, STATE1 movdqu STATE1, 0x00(OUTP) - movdqu 0x10(OUTP), INC - pxor INC, STATE2 + movdqu 0x10(OUTP), IN + pxor IN, STATE2 movdqu STATE2, 0x10(OUTP) - movdqu 0x20(OUTP), INC - pxor INC, STATE3 + movdqu 0x20(OUTP), IN + pxor IN, STATE3 movdqu STATE3, 0x20(OUTP) - movdqu 0x30(OUTP), INC - pxor INC, STATE4 + movdqu 0x30(OUTP), IN + pxor IN, STATE4 movdqu STATE4, 0x30(OUTP) _aesni_gf128mul_x_ble() add $64, INP add $64, OUTP - sub $64, LEN - ja .Lxts_dec_loop4 + cmp $0, LEN + jnz .Lxts_dec_loop4 +.Lxts_dec_ret_iv: movups IV, (IVP) +.Lxts_dec_ret: +#ifndef __x86_64__ + popl KLEN + popl KEYP + popl LEN + popl IVP +#endif FRAME_END ret -SYM_FUNC_END(aesni_xts_decrypt) +.Lxts_dec_1x: + add $64, LEN + jz .Lxts_dec_ret_iv + +.Lxts_dec_loop1: + movdqu (INP), STATE + + add $16, INP + sub $16, LEN + jl .Lxts_dec_cts1 + + pxor IV, STATE + call _aesni_dec1 + pxor IV, STATE + _aesni_gf128mul_x_ble() + + cmp $0, LEN + jz .Lxts_dec_out + + movdqu STATE, (OUTP) + add $16, OUTP + jmp .Lxts_dec_loop1 + +.Lxts_dec_out: + movdqu STATE, (OUTP) + jmp .Lxts_dec_ret_iv + +.Lxts_dec_cts1: + movdqa IV, STATE4 + _aesni_gf128mul_x_ble() + + pxor IV, STATE + call _aesni_dec1 + pxor IV, STATE + +#ifndef __x86_64__ + lea .Lcts_permute_table, T1 +#else + lea .Lcts_permute_table(%rip), T1 #endif + add LEN, INP /* rewind input pointer */ + add $16, LEN /* # bytes in final block */ + movups (INP), IN1 + + mov T1, IVP + add $32, IVP + add LEN, T1 + sub LEN, IVP + add OUTP, LEN + + movups (T1), %xmm4 + movaps STATE, IN2 + pshufb %xmm4, STATE + movups STATE, (LEN) + + movups (IVP), %xmm0 + pshufb %xmm0, IN1 + pblendvb IN2, IN1 + movaps IN1, STATE + + pxor STATE4, STATE + call _aesni_dec1 + pxor STATE4, STATE + + movups STATE, (OUTP) + jmp .Lxts_dec_ret +SYM_FUNC_END(aesni_xts_decrypt)
arch/x86/crypto/aesni-intel_glue.c +125 -96
254 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 22 lines · Ard Biesheuvel
The AES-NI implementation of XTS was impacted significantly by the retpoline changes, which is due to the fact that both its asm helper and the chaining mode glue library use indirect calls for processing small quantitities of data
So let's fix this, by:
- creating a minimal, backportable fix that recovers most of the performance, by reducing the number of indirect calls substantially;
- for future releases, rewrite the XTS implementation completely, and replace the glue helper with a core asm routine that is more flexible, making the C code wrapper much more straight-forward.This results in a substantial performance improvement: around ~2x for 1k and 4k blocks, and more than 3x for ~1k blocks that require ciphertext stealing (benchmarked using tcrypt using 1420 byte blocks - full results below)
It also allows us to enable the same driver for i386.
Cc Megha Dey <[email protected]>Cc Eric Biggers <[email protected]>Cc Herbert Xu <[email protected]>
Quoted 12 lines
Ard Biesheuvel (2):
crypto: x86/aes-ni-xts - use direct calls to and 4-way stride
crypto: x86/aes-ni-xts - rewrite and drop indirections via glue helperarch/x86/crypto/aesni-intel_asm.S | 353 ++++++++++++++++---- arch/x86/crypto/aesni-intel_glue.c | 230 +++++++------ 2 files changed, 412 insertions(+), 171 deletions(-)Signature · 3 lines
2.17.1 Benchmarked using tcrypt on a Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz.
Thanks for doing this! I didn't realize that there was such a big performance regression here. Getting rid of these indirect calls looks like the right approach; this all seems to have been written for a world where indirect calls are much faster...
I did some quick benchmarks on Zen ("AMD Ryzen Threadripper 1950X 16-Core Processor") with CONFIG_RETPOLINE=y and confirmed the speedup on 4096-byte blocks is around 2x there too. (It's over 2x for AES-128-XTS and AES-192-XTS, and a bit under 2x for AES-256-XTS. And most of the speedup comes from the first patch.) Also, the extra self-tests are passing.
So feel free to add:
Note that this patch series didn't apply cleanly, as it seems to depend on some other patches you've sent out recently. So I actually tested your "for-kernelci" branch instead of applying these directly.
- Eric
Quoted 48 lines · Eric Biggers
Quoted 26 lines · Ard Biesheuvel
The AES-NI implementation of XTS was impacted significantly by the retpoline changes, which is due to the fact that both its asm helper and the chaining mode glue library use indirect calls for processing small quantitities of data
So let's fix this, by:
- creating a minimal, backportable fix that recovers most of the performance, by reducing the number of indirect calls substantially;
- for future releases, rewrite the XTS implementation completely, and replace the glue helper with a core asm routine that is more flexible, making the C code wrapper much more straight-forward.This results in a substantial performance improvement: around ~2x for 1k and 4k blocks, and more than 3x for ~1k blocks that require ciphertext stealing (benchmarked using tcrypt using 1420 byte blocks - full results below)
It also allows us to enable the same driver for i386.
Cc Megha Dey <[email protected]>Cc Eric Biggers <[email protected]>Cc Herbert Xu <[email protected]>Ard Biesheuvel (2):
crypto: x86/aes-ni-xts - use direct calls to and 4-way stride
crypto: x86/aes-ni-xts - rewrite and drop indirections via glue helperQuoted 8 lines
arch/x86/crypto/aesni-intel_asm.S | 353 ++++++++++++++++---- arch/x86/crypto/aesni-intel_glue.c | 230 +++++++------ 2 files changed, 412 insertions(+), 171 deletions(-)--
2.17.1Benchmarked using tcrypt on a Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz.
Thanks for doing this! I didn't realize that there was such a big performance regression here. Getting rid of these indirect calls looks like the right approach; this all seems to have been written for a world where indirect calls are much faster...
I did some quick benchmarks on Zen ("AMD Ryzen Threadripper 1950X 16-Core Processor") with
CONFIG_RETPOLINE=yand confirmed the speedup on 4096-byte blocks is around 2x there too. (It's over 2x for AES-128-XTS and AES-192-XTS, and a bit under 2x for AES-256-XTS. And most of the speedup comes from the first patch.) Also, the extra self-tests are passing.
Quoted 8 lines
So feel free to add:
Note that this patch series didn't apply cleanly, as it seems to depend on some other patches you've sent out recently. So I actually tested your "for-kernelci" branch instead of applying these directly.
Thanks Eric. I have some other stuff queued up locally as well, so there are some non-functional conflicts there. The only prerequisite for this series is the one that adds CTS-CBC support to AES-NI, give that the XTS implementation reuses the permute table.
I will rebase and resend.