crypto: x86 - AES-CTR and AES-XCTR rewrite
Subsystems (2)
cover crypto: x86 - AES-CTR and AES-XCTR rewrite
This series adds new implementations of AES-CTR and AES-XCTR that are optimized for modern x86_64 CPUs, and it removes the existing implementations that are superseded or obsolete.
Changed in v2:
- Split the removal of the non-AVX implementation of AES-CTR into a separate patch, and removed the assembly code too.
- Made some minor tweaks to the new assembly file, including fixing a build error when aesni-intel is built as a module.
Eric Biggers (2):
crypto: x86/aes-ctr - rewrite AES-NI optimized CTR and add VAES
support
crypto: x86/aes-ctr - remove non-AVX implementation of AES-CTR
arch/x86/crypto/Makefile | 2 +- arch/x86/crypto/aes-ctr-avx-x86_64.S | 552 ++++++++++++++++++++++ arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 597 ------------------------ arch/x86/crypto/aesni-intel_asm.S | 125 ----- arch/x86/crypto/aesni-intel_glue.c | 450 ++++++++---------- 5 files changed, 756 insertions(+), 970 deletions(-) create mode 100644 arch/x86/crypto/aes-ctr-avx-x86_64.S delete mode 100644 arch/x86/crypto/aes_ctrby8_avx-x86_64.S
base-commit: 2014c95afecee3e76ca4a56956a936e23283f05b
Signature · 1 line
2.48.1
PATCH 1/2 crypto: x86/aes-ctr - rewrite AES-NI optimized CTR and add VAES support
From: Eric Biggers <[email protected]>
Delete aes_ctrby8_avx-x86_64.S and add a new assembly file aes-ctr-avx-x86_64.S which follows a similar approach to aes-xts-avx-x86_64.S in that it uses a "template" to provide AESNI+AVX, VAES+AVX2, VAES+AVX10/256, and VAES+AVX10/512 code, instead of just AESNI+AVX. Wire it up to the crypto API accordingly.
This greatly improves the performance of AES-CTR and AES-XCTR on VAES-capable CPUs, with the best case being AMD Zen 5 where an over 230% increase in throughput (i.e. over 3.3x faster) is seen on long messages. Performance on older CPUs remains about the same. There are some slight regressions (less than 10%) on some short message lengths on some CPUs; these are difficult to avoid, given how the previous code was so heavily unrolled by message length, and they are not particularly important. Detailed performance results are given in the tables below.
Both CTR and XCTR support is retained. The main loop remains 8-vector-wide, which differs from the 4-vector-wide main loops that are used in the XTS and GCM code. A wider loop is appropriate for CTR and XCTR since they have fewer other instructions (such as vpclmulqdq) to interleave with the AES instructions.
Similar to what was the case for AES-GCM, the new assembly code also has a much smaller binary size, as it fixes the excessive unrolling by data length and key length present in the old code. Specifically, the new assembly file compiles to about 9 KB of text vs. 28 KB for the old file. This is despite 4x as many implementations being included.
The tables below show the detailed performance results. The tables show percentage improvement in single-threaded throughput for repeated encryption of the given message length; an increase from 6000 MB/s to 12000 MB/s would be listed as 100%. They were collected by directly measuring the Linux crypto API performance using a custom kernel module. The tested CPUs were all server processors from Google Compute Engine except for Zen 5 which was a Ryzen 9 9950X desktop processor.
Table 1: AES-256-CTR throughput improvement,
CPU microarchitecture vs. message length in bytes:
| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
---------------------+-------+-------+-------+-------+-------+-------+
AMD Zen 5 | 232% | 203% | 212% | 143% | 71% | 95% | Intel Emerald Rapids | 116% | 116% | 117% | 91% | 78% | 79% | Intel Ice Lake | 109% | 103% | 107% | 81% | 54% | 56% | AMD Zen 4 | 109% | 91% | 100% | 70% | 43% | 59% | AMD Zen 3 | 92% | 78% | 87% | 57% | 32% | 43% | AMD Zen 2 | 9% | 8% | 14% | 12% | 8% | 21% | Intel Skylake | 7% | 7% | 8% | 5% | 3% | 8% |
| 300 | 200 | 64 | 63 | 16 | ---------------------+-------+-------+-------+-------+-------+ AMD Zen 5 | 57% | 39% | -9% | 7% | -7% | Intel Emerald Rapids | 37% | 42% | -0% | 13% | -8% | Intel Ice Lake | 39% | 30% | -1% | 14% | -9% | AMD Zen 4 | 42% | 38% | -0% | 18% | -3% | AMD Zen 3 | 38% | 35% | 6% | 31% | 5% | AMD Zen 2 | 24% | 23% | 5% | 30% | 3% | Intel Skylake | 9% | 1% | -4% | 10% | -7% |
Table 2: AES-256-XCTR throughput improvement,
CPU microarchitecture vs. message length in bytes:
| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
---------------------+-------+-------+-------+-------+-------+-------+
AMD Zen 5 | 240% | 201% | 216% | 151% | 75% | 108% | Intel Emerald Rapids | 100% | 99% | 102% | 91% | 94% | 104% | Intel Ice Lake | 93% | 89% | 92% | 74% | 50% | 64% | AMD Zen 4 | 86% | 75% | 83% | 60% | 41% | 52% | AMD Zen 3 | 73% | 63% | 69% | 45% | 21% | 33% | AMD Zen 2 | -2% | -2% | 2% | 3% | -1% | 11% | Intel Skylake | -1% | -1% | 1% | 2% | -1% | 9% |
| 300 | 200 | 64 | 63 | 16 | ---------------------+-------+-------+-------+-------+-------+ AMD Zen 5 | 78% | 56% | -4% | 38% | -2% | Intel Emerald Rapids | 61% | 55% | 4% | 32% | -5% | Intel Ice Lake | 57% | 42% | 3% | 44% | -4% | AMD Zen 4 | 35% | 28% | -1% | 17% | -3% | AMD Zen 3 | 26% | 23% | -3% | 11% | -6% | AMD Zen 2 | 13% | 24% | -1% | 14% | -3% | Intel Skylake | 16% | 8% | -4% | 35% | -3% |
--- arch/x86/crypto/Makefile | 2 +- arch/x86/crypto/aes-ctr-avx-x86_64.S | 552 ++++++++++++++++++++++ arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 597 ------------------------ arch/x86/crypto/aesni-intel_glue.c | 403 ++++++++-------- 4 files changed, 762 insertions(+), 792 deletions(-) create mode 100644 arch/x86/crypto/aes-ctr-avx-x86_64.S delete mode 100644 arch/x86/crypto/aes_ctrby8_avx-x86_64.S
arch/x86/crypto/Makefile +1 -1
@@ -46,11 +46,11 @@ obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha-x86_64.o chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha_glue.o chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o -aesni-intel-$(CONFIG_64BIT) += aes_ctrby8_avx-x86_64.o \ +aesni-intel-$(CONFIG_64BIT) += aes-ctr-avx-x86_64.o \ aes-gcm-aesni-x86_64.o \ aes-xts-avx-x86_64.o ifeq ($(CONFIG_AS_VAES)$(CONFIG_AS_VPCLMULQDQ),yy) aesni-intel-$(CONFIG_64BIT) += aes-gcm-avx10-x86_64.o endif
arch/x86/crypto/aes-ctr-avx-x86_64.S new file +552 -0
552 lines not on this page · raw message
arch/x86/crypto/aes_ctrby8_avx-x86_64.S deleted +0 -597
597 lines not on this page · raw message
arch/x86/crypto/aesni-intel_glue.c +209 -194
596 lines not on this page · raw message
Signature · 1 line
2.48.1
PATCH 2/2 crypto: x86/aes-ctr - remove non-AVX implementation of AES-CTR
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Nearly all x86_64 CPUs with AES-NI also support AVX. The exceptions are Intel Westmere from 2010, and the low-power Intel CPU microarchitectures Silvermont, Goldmont, and Tremont from 2013 through 2020. Tremont's successor, Gracemont (launched in 2021), supports AVX. It is unlikely that any more non-AVX-capable x86_64 CPUs will be released.
Supporting non-AVX x86_64 SIMD assembly code is a major burden, given the differences between VEX and non-VEX code. It is probably still worth doing for the most common algorithms like xts(aes) and gcm(aes). ctr(aes) seems unlikely to be one of these; it can be used in IPsec together with a standalone MAC if the better option of gcm(aes) is not being used, but it is not useful for much else in the kernel.
Therefore, let's drop the non-AVX implementation of ctr(aes).
--- arch/x86/crypto/aesni-intel_asm.S | 125 ----------------------------- arch/x86/crypto/aesni-intel_glue.c | 59 -------------- 2 files changed, 184 deletions(-)
arch/x86/crypto/aesni-intel_asm.S +0 -125
@@ -30,14 +30,10 @@ #define IN4 %xmm9 #define IN IN1 #define KEY %xmm2 #define IV %xmm3 -#define BSWAP_MASK %xmm10 -#define CTR %xmm11 -#define INC %xmm12 - #define GF128MUL_MASK %xmm7 #ifdef __x86_64__ #define AREG %rax #define KEYP %rdi
@@ -47,12 +43,10 @@ #define LEN %rcx #define IVP %r8 #define KLEN %r9d #define T1 %r10 #define TKEYP T1 -#define T2 %r11 -#define TCTR_LOW T2 #else #define AREG %eax #define KEYP %edi #define OUTP AREG #define UKEYP OUTP
@@ -1008,131 +1002,12 @@ SYM_FUNC_END(aesni_cts_cbc_dec) .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 -#ifdef __x86_64__ -.Lbswap_mask: - .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 -#endif .popsection -#ifdef __x86_64__ -/* - * _aesni_inc_init: internal ABI - * setup registers used by _aesni_inc - * input: - * IV - * output: - * CTR: == IV, in little endian - * TCTR_LOW: == lower qword of CTR - * INC: == 1, in little endian - * BSWAP_MASK == endian swapping mask - */ -SYM_FUNC_START_LOCAL(_aesni_inc_init) - movaps .Lbswap_mask(%rip), BSWAP_MASK - movaps IV, CTR - pshufb BSWAP_MASK, CTR - mov $1, TCTR_LOW - movq TCTR_LOW, INC - movq CTR, TCTR_LOW - RET -SYM_FUNC_END(_aesni_inc_init) - -/* - * _aesni_inc: internal ABI - * Increase IV by 1, IV is in big endian - * input: - * IV - * CTR: == IV, in little endian - * TCTR_LOW: == lower qword of CTR - * INC: == 1, in little endian - * BSWAP_MASK == endian swapping mask - * output: - * IV: Increase by 1 - * changed: - * CTR: == output IV, in little endian - * TCTR_LOW: == lower qword of CTR - */ -SYM_FUNC_START_LOCAL(_aesni_inc) - paddq INC, CTR - add $1, TCTR_LOW - jnc .Linc_low - pslldq $8, INC - paddq INC, CTR - psrldq $8, INC -.Linc_low: - movaps CTR, IV - pshufb BSWAP_MASK, IV - RET -SYM_FUNC_END(_aesni_inc) - -/* - * void aesni_ctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len, u8 *iv) - */ -SYM_FUNC_START(aesni_ctr_enc) - FRAME_BEGIN - cmp $16, LEN - jb .Lctr_enc_just_ret - mov 480(KEYP), KLEN - movups (IVP), IV - call _aesni_inc_init - cmp $64, LEN - jb .Lctr_enc_loop1 -.align 4 -.Lctr_enc_loop4: - movaps IV, STATE1 - call _aesni_inc - movups (INP), IN1 - movaps IV, STATE2 - call _aesni_inc - movups 0x10(INP), IN2 - movaps IV, STATE3 - call _aesni_inc - movups 0x20(INP), IN3 - movaps IV, STATE4 - call _aesni_inc - movups 0x30(INP), IN4 - call _aesni_enc4 - pxor IN1, STATE1 - movups STATE1, (OUTP) - pxor IN2, STATE2 - movups STATE2, 0x10(OUTP) - pxor IN3, STATE3 - movups STATE3, 0x20(OUTP) - pxor IN4, STATE4 - movups STATE4, 0x30(OUTP) - sub $64, LEN - add $64, INP - add $64, OUTP - cmp $64, LEN - jge .Lctr_enc_loop4 - cmp $16, LEN - jb .Lctr_enc_ret -.align 4 -.Lctr_enc_loop1: - movaps IV, STATE - call _aesni_inc - movups (INP), IN - call _aesni_enc1 - pxor IN, STATE - movups STATE, (OUTP) - sub $16, LEN - add $16, INP - add $16, OUTP - cmp $16, LEN - jge .Lctr_enc_loop1 -.Lctr_enc_ret: - movups IV, (IVP) -.Lctr_enc_just_ret: - FRAME_END - RET -SYM_FUNC_END(aesni_ctr_enc) - -#endif - .section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 .align 16 .Lgf128mul_x_ble_mask: .octa 0x00000000000000010000000000000087 .previous
arch/x86/crypto/aesni-intel_glue.c +0 -59
89 lines not on this page · raw message
Signature · 1 line
2.48.1
Re: [PATCH 2/2] crypto: x86/aes-ctr - remove non-AVX implementation of AES-CTR
Quoted 15 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Nearly all
x86_64CPUs with AES-NI also support AVX. The exceptions are Intel Westmere from 2010, and the low-power Intel CPU microarchitectures Silvermont, Goldmont, and Tremont from 2013 through 2020. Tremont's successor, Gracemont (launched in 2021), supports AVX. It is unlikely that any more non-AVX-capablex86_64CPUs will be released.Supporting non-AVX
x86_64SIMD assembly code is a major burden, given the differences between VEX and non-VEX code. It is probably still worth doing for the most common algorithms like xts(aes) and gcm(aes). ctr(aes) seems unlikely to be one of these; it can be used in IPsec together with a standalone MAC if the better option of gcm(aes) is not being used, but it is not useful for much else in the kernel.
Quoted 3 lines
Therefore, let's drop the non-AVX implementation of ctr(aes).
Signed-off-by Eric Biggers <[email protected]>
It was brought to my attention that the above does not list all the Intel CPUs that have AES-NI without AVX. The Pentiums and Celerons based on the Skylake, Kaby Lake, Coffee Lake, and Comet Lake microarchitectures have AVX fused off.
I'm leaning towards dropping this patch, and keeping the AES-NI only AES-CTR around for a couple years longer just in case. This patch would just be 184 deletions, so not a huge amount anyway, and I think we do need to keep some of the other modes in aesni-intel_asm.S like XTS anyway.
- Eric
Quoted 1 line · Eric Biggers
[PATCH 0/2] crypto: x86 - AES-CTR and AES-XCTR rewrite
To clarify, this is meant to be v2. I forgot to put it in the subject lines.
- Eric