Skip to contentSkip to conversation

crypto: x86/aes-ctr - rewrite AESNI+AVX optimized CTR and add VAES support

PATCH Superseded Eric Biggers 2 participants 2 messages Feb 8, 2025

Subsystems (2)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 crypto: x86/aes-ctr - rewrite AESNI+AVX optimized CTR and add VAES support

From: Eric Biggers <[email protected]>

Delete aes_ctrby8_avx-x86_64.S and add a new assembly file aes-ctr-avx-x86_64.S which follows a similar approach to aes-xts-avx-x86_64.S in that it uses a "template" to provide AESNI+AVX, VAES+AVX2, VAES+AVX10/256, and VAES+AVX10/512 code, instead of just AESNI+AVX. Wire it up to the crypto API accordingly.

This greatly improves the performance of AES-CTR and AES-XCTR on VAES-capable CPUs, with the best case being AMD Zen 5 where an over 230% increase in throughput is seen on long messages. Performance on non-VAES-capable CPUs remains about the same, and the non-AVX AES-CTR code (aesni_ctr_enc) is also kept as-is for now. There are some slight regressions (less than 10%) on some short message lengths on some CPUs; these are difficult to avoid, given how the previous code was so heavily unrolled by message length, and they are not particularly important. Detailed performance results are given in the tables below.

Both CTR and XCTR support is retained. The main loop remains 8-vector-wide, which differs from the 4-vector-wide main loops that are used in the XTS and GCM code. A wider loop is appropriate for CTR and XCTR since they have fewer other instructions (such as vpclmulqdq) to interleave with the AES instructions.

Similar to what was the case for AES-GCM, the new assembly code also has a much smaller binary size, as it fixes the excessive unrolling by data length and key length present in the old code. Specifically, the new assembly file compiles to about 9 KB of text vs. 28 KB for the old file. This is despite 4x as many implementations being included.

The tables below show the detailed performance results. The tables show percentage improvement in single-threaded throughput for repeated encryption of the given message length; an increase from 6000 MB/s to 12000 MB/s would be listed as 100%. They were collected by directly measuring the Linux crypto API performance using a custom kernel module. The tested CPUs were all server processors from Google Compute Engine except for Zen 5 which was a Ryzen 9 9950X desktop processor.

Table 1: AES-256-CTR throughput improvement,
CPU microarchitecture vs. message length in bytes:

| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
---------------------+-------+-------+-------+-------+-------+-------+

AMD Zen 5            |  232% |  203% |  212% |  143% |   71% |   95% |
Intel Emerald Rapids |  116% |  116% |  117% |   91% |   78% |   79% |
Intel Ice Lake       |  109% |  103% |  107% |   81% |   54% |   56% |
AMD Zen 4            |  109% |   91% |  100% |   70% |   43% |   59% |
AMD Zen 3            |   92% |   78% |   87% |   57% |   32% |   43% |
AMD Zen 2            |    9% |    8% |   14% |   12% |    8% |   21% |
Intel Skylake        |    7% |    7% |    8% |    5% |    3% |    8% |
                     |   300 |   200 |    64 |    63 |    16 |
---------------------+-------+-------+-------+-------+-------+
AMD Zen 5            |   57% |   39% |   -9% |    7% |   -7% |
Intel Emerald Rapids |   37% |   42% |   -0% |   13% |   -8% |
Intel Ice Lake       |   39% |   30% |   -1% |   14% |   -9% |
AMD Zen 4            |   42% |   38% |   -0% |   18% |   -3% |
AMD Zen 3            |   38% |   35% |    6% |   31% |    5% |
AMD Zen 2            |   24% |   23% |    5% |   30% |    3% |
Intel Skylake        |    9% |    1% |   -4% |   10% |   -7% |

Table 2: AES-256-XCTR throughput improvement,
CPU microarchitecture vs. message length in bytes:

| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
---------------------+-------+-------+-------+-------+-------+-------+

AMD Zen 5            |  240% |  201% |  216% |  151% |   75% |  108% |
Intel Emerald Rapids |  100% |   99% |  102% |   91% |   94% |  104% |
Intel Ice Lake       |   93% |   89% |   92% |   74% |   50% |   64% |
AMD Zen 4            |   86% |   75% |   83% |   60% |   41% |   52% |
AMD Zen 3            |   73% |   63% |   69% |   45% |   21% |   33% |
AMD Zen 2            |   -2% |   -2% |    2% |    3% |   -1% |   11% |
Intel Skylake        |   -1% |   -1% |    1% |    2% |   -1% |    9% |
                     |   300 |   200 |    64 |    63 |    16 |
---------------------+-------+-------+-------+-------+-------+
AMD Zen 5            |   78% |   56% |   -4% |   38% |   -2% |
Intel Emerald Rapids |   61% |   55% |    4% |   32% |   -5% |
Intel Ice Lake       |   57% |   42% |    3% |   44% |   -4% |
AMD Zen 4            |   35% |   28% |   -1% |   17% |   -3% |
AMD Zen 3            |   26% |   23% |   -3% |   11% |   -6% |
AMD Zen 2            |   13% |   24% |   -1% |   14% |   -3% |
Intel Skylake        |   16% |    8% |   -4% |   35% |   -3% |

---

Changed in v3:
- Dropped the patch removing the non-AVX AES-CTR for now
- Changed license of aes-ctr-avx-x86_64.S to Apache-2.0 OR BSD-2-Clause, same as what I used for the AES-GCM assembly files. I've received interest in this code possibly being reused in other projects.
- Updated commit message to remove an ambiguous statement
- Updated commit message to clarify that the non-AVX code is unchanged
- Added comment above ctr_crypt_aesni() noting that it's non-AVX

Changed in v2:
- Split the removal of the non-AVX implementation of AES-CTR into a separate patch, and removed the assembly code too.
- Made some minor tweaks to the new assembly file, including fixing a build error when aesni-intel is built as a module.

 arch/x86/crypto/Makefile                |   2 +-
 arch/x86/crypto/aes-ctr-avx-x86_64.S    | 592 +++++++++++++++++++++++
 arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 597 ------------------------
 arch/x86/crypto/aesni-intel_glue.c      | 404 ++++++++--------
 4 files changed, 803 insertions(+), 792 deletions(-)
 create mode 100644 arch/x86/crypto/aes-ctr-avx-x86_64.S
 delete mode 100644 arch/x86/crypto/aes_ctrby8_avx-x86_64.S
arch/x86/crypto/Makefile +1 -1
index 07b00bfca64b5..5d19f41bde583 100644
@@ -46,11 +46,11 @@ obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha-x86_64.o
 chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha_glue.o
 chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o
 
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
-aesni-intel-$(CONFIG_64BIT) += aes_ctrby8_avx-x86_64.o \
+aesni-intel-$(CONFIG_64BIT) += aes-ctr-avx-x86_64.o \
 			       aes-gcm-aesni-x86_64.o \
 			       aes-xts-avx-x86_64.o
 ifeq ($(CONFIG_AS_VAES)$(CONFIG_AS_VPCLMULQDQ),yy)
 aesni-intel-$(CONFIG_64BIT) += aes-gcm-avx10-x86_64.o
 endif
arch/x86/crypto/aes-ctr-avx-x86_64.S new file +592 -0

592 lines not on this page · raw message

arch/x86/crypto/aes_ctrby8_avx-x86_64.S deleted +0 -597

597 lines not on this page · raw message

arch/x86/crypto/aesni-intel_glue.c +210 -194

597 lines not on this page · raw message

base-commit: 2014c95afecee3e76ca4a56956a936e23283f05b

Signature · 1 line
2.48.1
#1 lore raw

Hi Eric,

Quoted 85 lines · Eric Biggers

From: Eric Biggers <[email protected]>

Delete aes_ctrby8_avx-x86_64.S and add a new assembly file aes-ctr-avx-x86_64.S which follows a similar approach to aes-xts-avx-x86_64.S in that it uses a "template" to provide AESNI+AVX, VAES+AVX2, VAES+AVX10/256, and VAES+AVX10/512 code, instead of just AESNI+AVX. Wire it up to the crypto API accordingly.

This greatly improves the performance of AES-CTR and AES-XCTR on VAES-capable CPUs, with the best case being AMD Zen 5 where an over 230% increase in throughput is seen on long messages. Performance on non-VAES-capable CPUs remains about the same, and the non-AVX AES-CTR code (aesni_ctr_enc) is also kept as-is for now. There are some slight regressions (less than 10%) on some short message lengths on some CPUs; these are difficult to avoid, given how the previous code was so heavily unrolled by message length, and they are not particularly important. Detailed performance results are given in the tables below.

Both CTR and XCTR support is retained. The main loop remains 8-vector-wide, which differs from the 4-vector-wide main loops that are used in the XTS and GCM code. A wider loop is appropriate for CTR and XCTR since they have fewer other instructions (such as vpclmulqdq) to interleave with the AES instructions.

Similar to what was the case for AES-GCM, the new assembly code also has a much smaller binary size, as it fixes the excessive unrolling by data length and key length present in the old code. Specifically, the new assembly file compiles to about 9 KB of text vs. 28 KB for the old file. This is despite 4x as many implementations being included.

The tables below show the detailed performance results. The tables show percentage improvement in single-threaded throughput for repeated encryption of the given message length; an increase from 6000 MB/s to 12000 MB/s would be listed as 100%. They were collected by directly measuring the Linux crypto API performance using a custom kernel module. The tested CPUs were all server processors from Google Compute Engine except for Zen 5 which was a Ryzen 9 9950X desktop processor.

Table 1: AES-256-CTR throughput improvement,
CPU microarchitecture vs. message length in bytes:

| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
---------------------+-------+-------+-------+-------+-------+-------+

AMD Zen 5            |  232% |  203% |  212% |  143% |   71% |   95% |
Intel Emerald Rapids |  116% |  116% |  117% |   91% |   78% |   79% |
Intel Ice Lake       |  109% |  103% |  107% |   81% |   54% |   56% |
AMD Zen 4            |  109% |   91% |  100% |   70% |   43% |   59% |
AMD Zen 3            |   92% |   78% |   87% |   57% |   32% |   43% |
AMD Zen 2            |    9% |    8% |   14% |   12% |    8% |   21% |
Intel Skylake        |    7% |    7% |    8% |    5% |    3% |    8% |
                     |   300 |   200 |    64 |    63 |    16 |
---------------------+-------+-------+-------+-------+-------+
AMD Zen 5            |   57% |   39% |   -9% |    7% |   -7% |
Intel Emerald Rapids |   37% |   42% |   -0% |   13% |   -8% |
Intel Ice Lake       |   39% |   30% |   -1% |   14% |   -9% |
AMD Zen 4            |   42% |   38% |   -0% |   18% |   -3% |
AMD Zen 3            |   38% |   35% |    6% |   31% |    5% |
AMD Zen 2            |   24% |   23% |    5% |   30% |    3% |
Intel Skylake        |    9% |    1% |   -4% |   10% |   -7% |

Table 2: AES-256-XCTR throughput improvement,
CPU microarchitecture vs. message length in bytes:

| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
---------------------+-------+-------+-------+-------+-------+-------+

AMD Zen 5            |  240% |  201% |  216% |  151% |   75% |  108% |
Intel Emerald Rapids |  100% |   99% |  102% |   91% |   94% |  104% |
Intel Ice Lake       |   93% |   89% |   92% |   74% |   50% |   64% |
AMD Zen 4            |   86% |   75% |   83% |   60% |   41% |   52% |
AMD Zen 3            |   73% |   63% |   69% |   45% |   21% |   33% |
AMD Zen 2            |   -2% |   -2% |    2% |    3% |   -1% |   11% |
Intel Skylake        |   -1% |   -1% |    1% |    2% |   -1% |    9% |
                     |   300 |   200 |    64 |    63 |    16 |
---------------------+-------+-------+-------+-------+-------+
AMD Zen 5            |   78% |   56% |   -4% |   38% |   -2% |
Intel Emerald Rapids |   61% |   55% |    4% |   32% |   -5% |
Intel Ice Lake       |   57% |   42% |    3% |   44% |   -4% |
AMD Zen 4            |   35% |   28% |   -1% |   17% |   -3% |
AMD Zen 3            |   26% |   23% |   -3% |   11% |   -6% |
AMD Zen 2            |   13% |   24% |   -1% |   14% |   -3% |
Intel Skylake        |   16% |    8% |   -4% |   35% |   -3% |
Quoted 3 lines

---

Very nice results! One remark below.
...

Quoted 227 lines
arch/x86/crypto/aes-ctr-avx-x86_64.S new file +221 -0
new file mode 100644
index 0000000000000..25cab1d8e63f9
@@ -0,0 +1,592 @@
+/* SPDX-License-Identifier: Apache-2.0 OR BSD-2-Clause */
+//
+// Copyright 2025 Google LLC
+//
+// Author: Eric Biggers <[email protected]>
+//
+// This file is dual-licensed, meaning that you can use it under your choice of
+// either of the following two licenses:
+//
+// Licensed under the Apache License 2.0 (the "License").  You may obtain a copy
+// of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+// or
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice,
+//    this list of conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright
+//    notice, this list of conditions and the following disclaimer in the
+//    documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+// POSSIBILITY OF SUCH DAMAGE.
+//
+//------------------------------------------------------------------------------
+//
+// This file contains x86_64 assembly implementations of AES-CTR and AES-XCTR
+// using the following sets of CPU features:
+//     - AES-NI && AVX
+//     - VAES && AVX2
+//     - VAES && (AVX10/256 || (AVX512BW && AVX512VL)) && BMI2
+//     - VAES && (AVX10/512 || (AVX512BW && AVX512VL)) && BMI2
+//
+// See the function definitions at the bottom of the file for more information.
+
+#include <linux/linkage.h>
+#include <linux/cfi_types.h>
+
+.section .rodata
+.p2align 4
+
+.Lbswap_mask:
+       .octa   0x000102030405060708090a0b0c0d0e0f
+
+.Lctr_pattern:
+       .quad   0, 0
+.Lone:
+       .quad   1, 0
+.Ltwo:
+       .quad   2, 0
+       .quad   3, 0
+
+.Lfour:
+       .quad   4, 0
+
+.text
+
+// Move a vector between memory and a register.
+// The register operand must be in the first 16 vector registers.
+.macro _vmovdqu        src, dst
+.if VL < 64
+       vmovdqu         \src, \dst
+.else
+       vmovdqu8        \src, \dst
+.endif
+.endm
+
+// Move a vector between registers.
+// The registers must be in the first 16 vector registers.
+.macro _vmovdqa        src, dst
+.if VL < 64
+       vmovdqa         \src, \dst
+.else
+       vmovdqa64       \src, \dst
+.endif
+.endm
+
+// Broadcast a 128-bit value from memory to all 128-bit lanes of a vector
+// register.  The register operand must be in the first 16 vector registers.
+.macro _vbroadcast128  src, dst
+.if VL == 16
+       vmovdqu         \src, \dst
+.elseif VL == 32
+       vbroadcasti128  \src, \dst
+.else
+       vbroadcasti32x4 \src, \dst
+.endif
+.endm
+
+// XOR two vectors together.
+// Any register operands must be in the first 16 vector registers.
+.macro _vpxor  src1, src2, dst
+.if VL < 64
+       vpxor           \src1, \src2, \dst
+.else
+       vpxord          \src1, \src2, \dst
+.endif
+.endm
+
+// Load 1 <= %ecx <= 15 bytes from the pointer \src into the xmm register \dst
+// and zeroize any remaining bytes.  Clobbers %rax, %rcx, and \tmp{64,32}.
+.macro _load_partial_block     src, dst, tmp64, tmp32
+       sub             $8, %ecx                // LEN - 8
+       jle             .Lle8\@
+
+       // Load 9 <= LEN <= 15 bytes.
+       vmovq           (\src), \dst            // Load first 8 bytes
+       mov             (\src, %rcx), %rax      // Load last 8 bytes
+       neg             %ecx
+       shl             $3, %ecx
+       shr             %cl, %rax               // Discard overlapping bytes
+       vpinsrq         $1, %rax, \dst, \dst
+       jmp             .Ldone\@
+
+.Lle8\@:
+       add             $4, %ecx                // LEN - 4
+       jl              .Llt4\@
+
+       // Load 4 <= LEN <= 8 bytes.
+       mov             (\src), %eax            // Load first 4 bytes
+       mov             (\src, %rcx), \tmp32    // Load last 4 bytes
+       jmp             .Lcombine\@
+
+.Llt4\@:
+       // Load 1 <= LEN <= 3 bytes.
+       add             $2, %ecx                // LEN - 2
+       movzbl          (\src), %eax            // Load first byte
+       jl              .Lmovq\@
+       movzwl          (\src, %rcx), \tmp32    // Load last 2 bytes
+.Lcombine\@:
+       shl             $3, %ecx
+       shl             %cl, \tmp64
+       or              \tmp64, %rax            // Combine the two parts
+.Lmovq\@:
+       vmovq           %rax, \dst
+.Ldone\@:
+.endm
+
+// Store 1 <= %ecx <= 15 bytes from the xmm register \src to the pointer \dst.
+// Clobbers %rax, %rcx, and \tmp{64,32}.
+.macro _store_partial_block    src, dst, tmp64, tmp32
+       sub             $8, %ecx                // LEN - 8
+       jl              .Llt8\@
+
+       // Store 8 <= LEN <= 15 bytes.
+       vpextrq         $1, \src, %rax
+       mov             %ecx, \tmp32
+       shl             $3, %ecx
+       ror             %cl, %rax
+       mov             %rax, (\dst, \tmp64)    // Store last LEN - 8 bytes
+       vmovq           \src, (\dst)            // Store first 8 bytes
+       jmp             .Ldone\@
+
+.Llt8\@:
+       add             $4, %ecx                // LEN - 4
+       jl              .Llt4\@
+
+       // Store 4 <= LEN <= 7 bytes.
+       vpextrd         $1, \src, %eax
+       mov             %ecx, \tmp32
+       shl             $3, %ecx
+       ror             %cl, %eax
+       mov             %eax, (\dst, \tmp64)    // Store last LEN - 4 bytes
+       vmovd           \src, (\dst)            // Store first 4 bytes
+       jmp             .Ldone\@
+
+.Llt4\@:
+       // Store 1 <= LEN <= 3 bytes.
+       vpextrb         $0, \src, 0(\dst)
+       cmp             $-2, %ecx               // LEN - 4 == -2, i.e. LEN == 2?
+       jl              .Ldone\@
+       vpextrb         $1, \src, 1(\dst)
+       je              .Ldone\@
+       vpextrb         $2, \src, 2(\dst)
+.Ldone\@:
+.endm
+
+// Prepare the next two vectors of AES inputs in AESDATA\i0 and AESDATA\i1, and
+// XOR each with the zero-th round key.  Also update LE_CTR if !\final.
+.macro _prepare_2_ctr_vecs     is_xctr, i0, i1, final=0
+.if \is_xctr
+  .if USE_AVX10
+       _vmovdqa        LE_CTR, AESDATA\i0
+       vpternlogd      $0x96, XCTR_IV, RNDKEY0, AESDATA\i0
+  .else
+       vpxor           XCTR_IV, LE_CTR, AESDATA\i0
+       vpxor           RNDKEY0, AESDATA\i0, AESDATA\i0
+  .endif
+       vpaddq          LE_CTR_INC1, LE_CTR, AESDATA\i1
+
+  .if USE_AVX10
+       vpternlogd      $0x96, XCTR_IV, RNDKEY0, AESDATA\i1
+  .else
+       vpxor           XCTR_IV, AESDATA\i1, AESDATA\i1
+       vpxor           RNDKEY0, AESDATA\i1, AESDATA\i1
+  .endif
+.else
+       vpshufb         BSWAP_MASK, LE_CTR, AESDATA\i0
+       _vpxor          RNDKEY0, AESDATA\i0, AESDATA\i0
+       vpaddq          LE_CTR_INC1, LE_CTR, AESDATA\i1
+       vpshufb         BSWAP_MASK, AESDATA\i1, AESDATA\i1

arch/x86/crypto/aes-ctr-avx-x86_64.S line 231

Quoted 10 lines
arch/x86/crypto/aes-ctr-avx-x86_64.S new file +10 -0
+       _vpxor          RNDKEY0, AESDATA\i1, AESDATA\i1
+.endif
+.if !\final
+       vpaddq          LE_CTR_INC2, LE_CTR, LE_CTR
+.endif
+.endm
+
+// Do all AES rounds on the data in the given AESDATA vectors, excluding the
+// zero-th and last rounds.
+.macro _aesenc_loop    vecs

If you make this vecs:vararg, you can drop the "" around the arguments in the callers.

Quoted 6 lines
arch/x86/crypto/aes-ctr-avx-x86_64.S new file +6 -0
+       mov             KEY, %rax
+1:
+       _vbroadcast128  (%rax), RNDKEY
+.irp i, \vecs
+       vaesenc         RNDKEY, AESDATA\i, AESDATA\i
+.endr

arch/x86/crypto/aes-ctr-avx-x86_64.S line 247

Quoted 10 lines
arch/x86/crypto/aes-ctr-avx-x86_64.S new file +10 -0
+       add             $16, %rax
+       cmp             %rax, RNDKEYLAST_PTR
+       jne             1b
+.endm
+
+// Finalize the keystream blocks in the given AESDATA vectors by doing the last
+// AES round, then XOR those keystream blocks with the corresponding data.
+// Reduce latency by doing the XOR before the vaesenclast, utilizing the
+// property vaesenclast(key, a) ^ b == vaesenclast(key ^ b, a).
+.macro _aesenclast_and_xor     vecs

Same here

Quoted 9 lines
arch/x86/crypto/aes-ctr-avx-x86_64.S new file +9 -0
+.irp i, \vecs
+       _vpxor          \i*VL(SRC), RNDKEYLAST, RNDKEY
+       vaesenclast     RNDKEY, AESDATA\i, AESDATA\i
+.endr
+.irp i, \vecs
+       _vmovdqu        AESDATA\i, \i*VL(DST)
+.endr
+.endm
+

...