Migrate x86 and RISC-V accelerated AES modes into library

PATCH Applied Eric Biggers 2 participants 23 messages Sep 27 – Sep 30

Subsystems (4)

x86 RISC-V Crypto API Crypto library

lore

Share & tools

lore

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #0 lore raw

v2 cover Migrate x86 and RISC-V accelerated AES modes into library

This series applies to v7.3-rc4. It can also be retrieved from:

git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git aes-lib-x86-riscv-v2

Patch 1 was already applied to libcrypto-fixes, but is resent to make this series applicable directly to v7.3-rc4 (ensuring that the Sashiko review runs). Patches 2-20 are targeting libcrypto-next for 7.4.

This series migrates the x86 and RISC-V accelerated implementations of the AES modes ECB, CBC, CBC-CTS, CTR, XCTR, and XTS into lib/crypto/.

This makes the corresponding library APIs be properly accelerated on these architectures, while still accelerating crypto_skcipher as well (via the library-based code in crypto/aes.c).

It removes a lot of redundant glue code, since crypto API boilerplate no longer needs to be duplicated per-architecture.

Finally, it fixes the longstanding issue where these optimizations were disabled by default.

In the case of RISC-V, this series handles all remaining AES code in arch/riscv/crypto/. In the case of x86, AES-GCM is still left in arch/x86/crypto/ for now; it will be handled later. Other architectures will be handled later as well.

For various reasons, aesni-intel_asm.S (the x86-accelerated ECB, CBC, CBC-CTS, CTR, and XTS code that doesn't use AVX or VAES) is replaced with new functions written from scratch. The other assembly functions are kept but are modified slightly for integration into the library.

Changed in v2:
- Fixed 32-bit x86 bisection hazards by making aesni-intel depend on CONFIG_64BIT earlier in the series.

- Fixed handling of lengths over S32_MAX in RISC-V CTR code.

- Restored selection of CRYPTO_SKCIPHER by CRYPTO_AES_NI_INTEL, since it is still needed.

- Made the 32-bit x86 XTS code spill the tweaks to the stack rather than to the destination buffer.

- Removed the no-longer-needed single block special case from the RISC-V CBC-CTS assembly code.

- Made x86 CBC-CTS encryption and decryption share more code.

- Reordered the function parameters in aes-xts-avx-x86_64.S and

     aes-ctr-avx-x86_64.S.

- Other miscellaneous cleanups.

Eric Biggers (20):
crypto: aes - Fix undesired override of some optimized AES modes lib/crypto: aes-xctr: Pass counter by value to aes_xctr_arch() lib/crypto: x86/aes: Clean up aes-aesni.S in preparation for AES modes lib/crypto: x86/aes-ecb: Add AES-NI optimization lib/crypto: x86/aes-cbc: Add AES-NI optimization lib/crypto: x86/aes-ctr: Add AES-NI optimization lib/crypto: x86/aes-xts: Add AES-NI optimization
crypto: x86/aes - Drop superseded 32-bit build support
crypto: x86/aes-ecb - Remove superseded ECB skcipher
crypto: x86/aes-cbc - Remove superseded CBC skciphers
crypto: x86/aes-ctr - Remove superseded CTR skcipher
crypto: x86/aes-xts - Remove superseded XTS skcipher lib/crypto: x86/aes-ctr: Migrate AVX-optimized code into library lib/crypto: x86/aes-xts: Migrate AVX-optimized code into library lib/crypto: riscv/aes: Copy aes-macros.S to library lib/crypto: riscv/aes: Pass key struct to assembly code lib/crypto: riscv/aes-ecb: Migrate optimized code into library lib/crypto: riscv/aes-cbc: Migrate optimized code into library lib/crypto: riscv/aes-ctr: Migrate optimized code into library lib/crypto: riscv/aes-xts: Migrate optimized code into library

 arch/riscv/crypto/Kconfig                     |   15 -
 arch/riscv/crypto/Makefile                    |    4 -
 arch/riscv/crypto/aes-riscv64-glue.c          |  566 -------
 arch/riscv/crypto/aes-riscv64-zvkned.S        |  312 ----
 arch/x86/crypto/Kconfig                       |   10 +-
 arch/x86/crypto/Makefile                      |   10 +-
 arch/x86/crypto/aesni-intel_asm.S             | 1338 -----------------
 arch/x86/crypto/aesni-intel_glue.c            |  792 +---------
 crypto/aes.c                                  |   53 +-
 lib/crypto/Makefile                           |   14 +
 lib/crypto/aes.c                              |    6 +-
 .../crypto => lib/crypto/riscv}/aes-macros.S  |   25 +-
 .../riscv}/aes-riscv64-zvkned-zvbb-zvkg.S     |   99 +-
 .../crypto/riscv}/aes-riscv64-zvkned-zvkb.S   |   23 +-
 lib/crypto/riscv/aes-riscv64-zvkned.S         |  305 +++-
 lib/crypto/riscv/aes.h                        |  244 ++-
 lib/crypto/x86/aes-aesni.S                    |  816 +++++++++-
 .../crypto/x86}/aes-ctr-avx-x86_64.S          |  103 +-
 .../crypto/x86}/aes-xts-avx-x86_64.S          |  179 +--
 lib/crypto/x86/aes.h                          |  381 ++++-
 20 files changed, 1852 insertions(+), 3443 deletions(-)
 delete mode 100644 arch/riscv/crypto/aes-riscv64-glue.c
 delete mode 100644 arch/riscv/crypto/aes-riscv64-zvkned.S
 delete mode 100644 arch/x86/crypto/aesni-intel_asm.S
 rename {arch/riscv/crypto => lib/crypto/riscv}/aes-macros.S (90%)
 rename {arch/riscv/crypto => lib/crypto/riscv}/aes-riscv64-zvkned-zvbb-zvkg.S (74%)
 rename {arch/riscv/crypto => lib/crypto/riscv}/aes-riscv64-zvkned-zvkb.S (93%)
 rename {arch/x86/crypto => lib/crypto/x86}/aes-ctr-avx-x86_64.S (88%)
 rename {arch/x86/crypto => lib/crypto/x86}/aes-xts-avx-x86_64.S (81%)

base-commit: 93f51579e7df248780214094418f205253383cc5

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #1 lore raw

PATCH v2 1/20 crypto: aes - Fix undesired override of some optimized AES modes

The new library APIs for AES encryption modes were wired up to the traditional crypto API via crypto/aes.c. However, for now the kernel is still in a transitional state where various architectures still have architecture-optimized implementations of AES modes in arch/*/crypto/, wired up to the traditional crypto API only. Because of that, the crypto/aes.c algorithms were given a cra_priority of only 110 to prevent them from overriding arch/*/crypto/ in the traditional crypto API.

However, because of how the traditional crypto API works, the cra_priority trick doesn't work in cases where the relevant algorithm isn't directly implemented by arch/*/crypto/ but rather is provided by a template instance using other code in arch/*/crypto/.

For example, x86 doesn't have its own "ccm(aes)" but rather relies on the "ccm" template constructing it from the x86-optimized "ctr(aes)". The existence of the library-based "ccm(aes)" prevents that, even though its priority is lower than what the template would produce.

Thus, "ccm(aes)" ends up using the slower single-block AES code.

Therefore, skip wiring up the relevant library-based code to the traditional crypto API on architectures where this problem can occur, as determined by what exists in arch/*/crypto/ for each architecture.

This is ugly, but it's also temporary: these conditions will go away as architecture-optimized implementations of AES modes are migrated into the library. But until then, we need to prevent performance regressions by ensuring that the optimized code continues to be used.

Note: "xts(aes)" is left alone. Though the "xts" template can use "ecb(aes)" as an inner algorithm, in practice this isn't very efficient and a dedicated "xts(aes)" is already provided in all the important cases anyway. (This omission is also consistent with the fact that the library isn't planned to provide a similar ECB-to-XTS "adapter".)

Fixes 20df21a482aa ("crypto: aes - Add CBC and CBC-CTS support using library")

Fixes 8ca62072faa1 ("crypto: aes - Add GCM support using library")

Fixes f70ad727d1d6 ("crypto: aes - Add CCM support using library")

Signed-off-by Eric Biggers <[email protected]>

---
 crypto/aes.c | 39 ++++++++++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)

crypto/aes.c +36 -3

index 94791f481e98..5046b887ac9a 100644
@@ -637,7 +637,17 @@ static struct skcipher_alg skcipher_algs[] = {
 		.decrypt = crypto_aes_cbc_decrypt,
 	},
 #endif
-#if IS_ENABLED(CONFIG_CRYPTO_CTS)
+#if IS_ENABLED(CONFIG_CRYPTO_CTS) && \
+	/*
+	 * Skip registering this when it might block a "better" implementation
+	 * from being instantiated via the "cts" template wrapping an arch-
+	 * optimized "cbc(aes)" that hasn't yet been migrated into the library.
+	 */ \
+	!(IS_ENABLED(CONFIG_ARM) || \
+	  IS_ENABLED(CONFIG_ARM64) || \
+	  IS_ENABLED(CONFIG_PPC) || \
+	  IS_ENABLED(CONFIG_S390) || \
+	  IS_ENABLED(CONFIG_SPARC))
 	{
 		.base.cra_name = "cts(cbc(aes))",
 		.base.cra_driver_name = "cts-cbc-aes-lib",

@@ -980,7 +990,18 @@ static __maybe_unused int crypto_aes_ccm_decrypt(struct aead_request *req)
 }
 
 static struct aead_alg aead_algs[] = {
-#if IS_ENABLED(CONFIG_CRYPTO_GCM)
+#if IS_ENABLED(CONFIG_CRYPTO_GCM) && \
+	/*
+	 * Skip registering these when they might block "better" implementations
+	 * from being instantiated via the corresponding templates using
+	 * arch-optimized code that hasn't yet been migrated into the library.
+	 */ \
+	!(IS_ENABLED(CONFIG_ARM) || \
+	  IS_ENABLED(CONFIG_ARM64) || \
+	  IS_ENABLED(CONFIG_PPC) || \
+	  IS_ENABLED(CONFIG_RISCV) || \
+	  IS_ENABLED(CONFIG_S390) || \
+	  IS_ENABLED(CONFIG_SPARC))
 	{
 		.base.cra_name = "gcm(aes)",
 		.base.cra_driver_name = "gcm-aes-lib",

@@ -1012,7 +1033,19 @@ static struct aead_alg aead_algs[] = {
 		.chunksize = AES_BLOCK_SIZE,
 	},
 #endif /* CONFIG_CRYPTO_GCM */
-#if IS_ENABLED(CONFIG_CRYPTO_CCM)
+#if IS_ENABLED(CONFIG_CRYPTO_CCM) && \
+	/*
+	 * Skip registering this when it might block a "better" implementation
+	 * from being instantiated via the "ccm" template wrapping an arch-
+	 * optimized "ctr(aes)" that hasn't yet been migrated into the library.
+	 */ \
+	!(IS_ENABLED(CONFIG_ARM) || \
+	  IS_ENABLED(CONFIG_ARM64) || \
+	  IS_ENABLED(CONFIG_PPC) || \
+	  IS_ENABLED(CONFIG_RISCV) || \
+	  IS_ENABLED(CONFIG_S390) || \
+	  IS_ENABLED(CONFIG_SPARC) || \
+	  IS_ENABLED(CONFIG_X86))
 	{
 		.base.cra_name = "ccm(aes)",
 		.base.cra_driver_name = "ccm-aes-lib",

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #2 lore raw

PATCH v2 2/20 lib/crypto: aes-xctr: Pass counter by value to aes_xctr_arch()

In reply to Eric Biggers

Update the calling convention for aes_xctr_arch() to pass the counter by value, then do the increment in generic code. This aligns better with the x86_64 and arm64 assembly code for XCTR, which takes the counter by value and thus has to be paired with an increment in C code anyway.

Reviewed-by Thomas Huth <[email protected]>

Signed-off-by Eric Biggers <[email protected]>

---
 lib/crypto/aes.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

lib/crypto/aes.c +4 -2

index f1549839b3de..22f096c50242 100644
@@ -1094,7 +1094,7 @@ static bool aes_ctr_arch(u8 *dst, const u8 *src, size_t len,
 }
 #endif
 #ifndef aes_xctr_arch
-static bool aes_xctr_arch(u8 *dst, const u8 *src, size_t len, u64 *ctr,
+static bool aes_xctr_arch(u8 *dst, const u8 *src, size_t len, u64 ctr,
 			  const u8 iv[AES_BLOCK_SIZE],
 			  const struct aes_enckey *key)
 {

@@ -1150,8 +1150,10 @@ void aes_xctr(u8 *dst, const u8 *src, size_t len, u64 *ctr,
 	__le64 aes_input[2];
 	u8 keystream[AES_BLOCK_SIZE] __aligned(__alignof__(long));
 
-	if (likely(aes_xctr_arch(dst, src, len, ctr, iv, key.enc_key)))
+	if (likely(aes_xctr_arch(dst, src, len, *ctr, iv, key.enc_key))) {
+		*ctr += DIV_ROUND_UP(len, AES_BLOCK_SIZE);
 		return;
+	}
 
 	aes_input[1] = get_unaligned((const __le64 *)&iv[8]);
 	/* Handle the full blocks. */

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #3 lore raw

PATCH v2 3/20 lib/crypto: x86/aes: Clean up aes-aesni.S in preparation for AES modes

In reply to Eric Biggers

Various miscellaneous updates to the assembly code in preparation for adding implementations of AES modes to the same file:

- Define macros for the function argument registers, function prologues, and function epilogues to centralize some of the handling of 32-bit vs 64-bit.

- Refactor the actual AES encryption and AES decryption logic into macros _do_aes and _do_aes_ecb so that some of the modes can reuse it.

- Rename mask to expandkey_mask to differentiate it from the bswap_mask that will be added.

- Update the prototypes of aes_encrypt_aesni() and aes_decrypt_aesni() to be dst, src, key so that they will match the mode functions.

Note that this means passing a pointer to the key struct instead of a (nrounds, rndkeys) pair, similar to what arch/x86/crypto/aes*.S do. Although this makes the assembly code depend on the format of the key struct, having one fewer argument makes it easier to accommodate 32-bit mode, and the C glue code becomes slightly simpler.

Signed-off-by Eric Biggers <[email protected]>

---
 lib/crypto/x86/aes-aesni.S | 249 +++++++++++++++++++++++++------------
 lib/crypto/x86/aes.h       |  28 +++--
 2 files changed, 185 insertions(+), 92 deletions(-)

lib/crypto/x86/aes-aesni.S +169 ~~-80~~

index b8c3e104a3be..a56fcb36da18 100644
@@ -8,25 +8,126 @@
 // AVX.  It does use up to SSE4.1, which all CPUs with AES-NI have.
 #include <linux/linkage.h>
 
-.section .rodata
 #ifdef __x86_64__
 #define RODATA(label)	label(%rip)
-#else
+
+#define ARG0		%rdi
+#define ARG1		%rsi
+#define ARG2		%rdx
+#define ARG2_32		%edx
+#define ARG3		%rcx
+#define ARG4		%r8
+#define TMP		%rax
+#define TMP_32		%eax
+#define TMP_16		%ax
+#define TMP_8		%al
+
+#else // __x86_64__
+
 #define RODATA(label)	label
-#endif
 
+// Caller-save GPRs and the first 3 function arguments, assuming -mregparm=3
+#define ARG0		%eax
+#define ARG1		%edx
+#define ARG2		%ecx
+#define ARG2_32		%ecx
+
+// *Callee*-save GPRs.
+#define ARG3		%edi
+#define ARG4		%esi
+#define TMP		%ebx
+#define TMP_32		%ebx
+#define TMP_16		%bx
+#define TMP_8		%bl
+#endif // !__x86_64__
+
+// Offsets in struct aes_key
+#define OFFSETOF_NROUNDS	4
+#define OFFSETOF_ROUNDKEYS	16
+#define OFFSETOF_INVROUNDKEYS	256
+
+.section .rodata
+.p2align 4
+.Lexpandkey_mask:
 	// A mask for pshufb that extracts the last dword, rotates it right by 8
 	// bits, and copies the result to all four dwords.
-.p2align 4
-.Lmask:
 	.byte	13, 14, 15, 12, 13, 14, 15, 12, 13, 14, 15, 12, 13, 14, 15, 12
 
-	// The AES round constants, used during key expansion
 .Lrcon:
+	// The AES round constants, used during key expansion
 	.long	0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36
 
 .text
 
+// In 32-bit mode, push certain callee-saved GPRs and optionally load function
+// arguments from the stack into them.  Do nothing in 64-bit mode.
+//
+// TMP is always made available as a temporary register.
+//
+// \uses_arg3 is 0 to not touch ARG3, 1 to make it available as a temporary
+// register only, or 2 to actually load it as an argument from the stack.
+// Likewise for \uses_arg4 and ARG4.  It's assumed the -mregparm=3 convention is
+// being used and %esp hasn't been modified since function entry.
+.macro _prologue	uses_arg3=0, uses_arg4=0
+#ifdef __i386__
+	.set	ARG3_OFFSET, 4	// Assuming -mregparm=3
+	.set	ARG3_PUSHED, \uses_arg3
+	.set	ARG4_PUSHED, \uses_arg4
+.if \uses_arg3
+	push	ARG3
+	.set	ARG3_OFFSET, ARG3_OFFSET + 4
+  .if \uses_arg3 == 2
+	mov	ARG3_OFFSET(%esp), ARG3
+  .endif
+.endif
+.if \uses_arg4
+	push	ARG4
+	.set	ARG3_OFFSET, ARG3_OFFSET + 4
+  .if \uses_arg4 == 2
+	mov	ARG3_OFFSET+4(%esp), ARG4
+  .endif
+.endif
+	push	TMP
+	.set	ARG3_OFFSET, ARG3_OFFSET + 4
+#endif // __i386__
+.endm
+
+.macro _reload_arg3
+#ifdef __i386__
+	mov	ARG3_OFFSET(%esp), ARG3
+#endif
+.endm
+
+// Undo any pushes that _prologue did, then return.
+.macro _epilogue
+#ifdef __i386__
+	pop	TMP
+.if ARG4_PUSHED
+	pop	ARG4
+.endif
+.if ARG3_PUSHED
+	pop	ARG3
+.endif
+#endif
+	RET
+.endm
+
+.macro	_aesenc		enc, rndkey, data
+.if \enc
+	aesenc		\rndkey, \data
+.else
+	aesdec		\rndkey, \data
+.endif
+.endm
+
+.macro	_aesenclast	enc, rndkeylast, data
+.if \enc
+	aesenclast	\rndkeylast, \data
+.else
+	aesdeclast	\rndkeylast, \data
+.endif
+.endm
+
 // Transform four dwords [a0, a1, a2, a3] in \a into
 // [a0, a0^a1, a0^a1^a2, a0^a1^a2^a3].  \tmp is a temporary xmm register.
 //

@@ -71,32 +172,18 @@
 .endm
 
 .macro	_aes_expandkey_aesni	is_aes128
-#ifdef __x86_64__
 	// Arguments
-	.set	RNDKEYS,	%rdi
-	.set	INV_RNDKEYS,	%rsi
-	.set	IN_KEY,		%rdx
+	.set	RNDKEYS,	ARG0
+	.set	INV_RNDKEYS,	ARG1
+	.set	IN_KEY,		ARG2
 
 	// Other local variables
-	.set	RCON_PTR,	%rcx
-	.set	COUNTER,	%eax
-#else
-	// Arguments, assuming -mregparm=3
-	.set	RNDKEYS,	%eax
-	.set	INV_RNDKEYS,	%edx
-	.set	IN_KEY,		%ecx
-
-	// Other local variables
-	.set	RCON_PTR,	%ebx
-	.set	COUNTER,	%esi
-#endif
+	.set	RCON_PTR,	ARG3
+	.set	COUNTER,	TMP_32
 	.set	RCON,		%xmm6
 	.set	MASK,		%xmm7
 
-#ifdef __i386__
-	push		%ebx
-	push		%esi
-#endif
+	_prologue	uses_arg3=1
 
 .if \is_aes128
 	// AES-128: the first round key is simply a copy of the raw key.

@@ -112,7 +199,7 @@
 .endif
 
 	// Generate the remaining round keys.
-	movdqa		RODATA(.Lmask), MASK
+	movdqa		RODATA(.Lexpandkey_mask), MASK
 .if \is_aes128
 	lea		RODATA(.Lrcon), RCON_PTR
 	mov		$10, COUNTER

@@ -176,11 +263,7 @@
 	movdqu		%xmm0, 16(INV_RNDKEYS)	// => Last inverse round key
 
 .Ldone\@:
-#ifdef __i386__
-	pop		%esi
-	pop		%ebx
-#endif
-	RET
+	_epilogue
 .endm
 
 // void aes128_expandkey_aesni(u32 rndkeys[], u32 *inv_rndkeys,

@@ -195,67 +278,73 @@ SYM_FUNC_START(aes256_expandkey_aesni)
 	_aes_expandkey_aesni	0
 SYM_FUNC_END(aes256_expandkey_aesni)
 
-.macro	_aes_crypt_aesni	enc
-#ifdef __x86_64__
-	.set	RNDKEYS,	%rdi
-	.set	NROUNDS,	%esi
-	.set	OUT,		%rdx
-	.set	IN,		%rcx
-#else
-	// Assuming -mregparm=3
-	.set	RNDKEYS,	%eax
-	.set	NROUNDS,	%edx
-	.set	OUT,		%ecx
-	.set	IN,		%ebx	// Passed on stack
-#endif
-
-#ifdef __i386__
-	push		%ebx
-	mov		8(%esp), %ebx
-#endif
-
-	// Zero-th round
-	movdqu		(IN), %xmm0
-	movdqu		(RNDKEYS), %xmm1
-	pxor		%xmm1, %xmm0
-
-	// Normal rounds
-	add		$16, RNDKEYS
+// AES-encrypt (\enc=1) or AES-decrypt (\enc=0) the AESDATA registers specified
+// in \vecs using the aes_enckey or aes_key pointed to by KEY.  RNDKEY must be
+// set to a temporary XMM register, NROUNDS to a temporary 32-bit GPR, and
+// RNDKEY_PTR to a temporary full-size GPR.
+.macro	_do_aes		enc, vecs:vararg
+	movl		OFFSETOF_NROUNDS(KEY), NROUNDS
 	dec		NROUNDS
-.Lnext_round\@:
-	movdqu		(RNDKEYS), %xmm1
 .if \enc
-	aesenc		%xmm1, %xmm0
+	.set		rndkey0_offs, OFFSETOF_ROUNDKEYS
 .else
-	aesdec		%xmm1, %xmm0
+	.set		rndkey0_offs, OFFSETOF_INVROUNDKEYS
 .endif
-	add		$16, RNDKEYS
+
+	// Do the zero-th AES round.
+	movdqu		rndkey0_offs(KEY), RNDKEY
+.irp i, \vecs
+	pxor		RNDKEY, AESDATA\i
+.endr
+	// Do the regular AES rounds.
+	lea		rndkey0_offs+16(KEY), RNDKEY_PTR
+.Lnext_round\@:
+	movdqu		(RNDKEY_PTR), RNDKEY
+	add		$16, RNDKEY_PTR
+.irp i, \vecs
+	_aesenc		\enc, RNDKEY, AESDATA\i
+.endr
 	dec		NROUNDS
-	jne		.Lnext_round\@
+	jnz		.Lnext_round\@
+	// Do the last AES round.
+	movdqu		(RNDKEY_PTR), RNDKEY
+.irp i, \vecs
+	_aesenclast	\enc, RNDKEY, AESDATA\i
+.endr
+.endm
 
-	// Last round
-	movdqu		(RNDKEYS), %xmm1
-.if \enc
-	aesenclast	%xmm1, %xmm0
-.else
-	aesdeclast	%xmm1, %xmm0
-.endif
-	movdqu		%xmm0, (OUT)
+.macro	_do_aes_ecb	enc, vecs:vararg
+.irp i, \vecs
+	movdqu		\i*16(SRC), AESDATA\i
+.endr
+	_do_aes		\enc, \vecs
+.irp i, \vecs
+	movdqu		AESDATA\i, \i*16(DST)
+.endr
+.endm
 
-#ifdef __i386__
-	pop		%ebx
-#endif
-	RET
+.macro	_aes_crypt_aesni	enc
+	.set	DST,		ARG0
+	.set	SRC,		ARG1
+	.set	KEY,		ARG2
+	.set	RNDKEY_PTR,	ARG3	// Temporary register for _do_aes
+	.set	NROUNDS,	TMP_32	// Temporary register for _do_aes
+	.set	AESDATA0,	%xmm0
+	.set	RNDKEY,		%xmm1	// Temporary register for _do_aes
+
+	_prologue	uses_arg3=1
+	_do_aes_ecb	\enc, 0
+	_epilogue
 .endm
 
-// void aes_encrypt_aesni(const u32 rndkeys[], int nrounds,
-//			  u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]);
+// void aes_encrypt_aesni(u8 dst[AES_BLOCK_SIZE], const u8 src[AES_BLOCK_SIZE],
+//			  const struct aes_enckey *key);
 SYM_FUNC_START(aes_encrypt_aesni)
 	_aes_crypt_aesni	1
 SYM_FUNC_END(aes_encrypt_aesni)
 
-// void aes_decrypt_aesni(const u32 inv_rndkeys[], int nrounds,
-//			  u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]);
+// void aes_decrypt_aesni(u8 dst[AES_BLOCK_SIZE], const u8 src[AES_BLOCK_SIZE],
+//			  const struct aes_key *key);
 SYM_FUNC_START(aes_decrypt_aesni)
 	_aes_crypt_aesni	0
 SYM_FUNC_END(aes_decrypt_aesni)

lib/crypto/x86/aes.h +16 ~~-12~~

index b047dee94f57..06146fef06be 100644
@@ -7,16 +7,21 @@
 
 #include <asm/fpu/api.h>
 
-static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_aes);
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_aesni);
+
+/* The assembly code assumes the following offsets. */
+static_assert(offsetof(struct aes_enckey, nrounds) == 4);
+static_assert(offsetof(struct aes_enckey, k.rndkeys) == 16);
+static_assert(offsetof(struct aes_key, inv_k.inv_rndkeys) == 256);
 
 void aes128_expandkey_aesni(u32 rndkeys[], u32 *inv_rndkeys,
 			    const u8 in_key[AES_KEYSIZE_128]);
 void aes256_expandkey_aesni(u32 rndkeys[], u32 *inv_rndkeys,
 			    const u8 in_key[AES_KEYSIZE_256]);
-void aes_encrypt_aesni(const u32 rndkeys[], int nrounds,
-		       u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]);
-void aes_decrypt_aesni(const u32 inv_rndkeys[], int nrounds,
-		       u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]);
+void aes_encrypt_aesni(u8 dst[AES_BLOCK_SIZE], const u8 src[AES_BLOCK_SIZE],
+		       const struct aes_enckey *key);
+void aes_decrypt_aesni(u8 dst[AES_BLOCK_SIZE], const u8 src[AES_BLOCK_SIZE],
+		       const struct aes_key *key);
 
 /*
  * Expand an AES key using AES-NI if supported and usable or generic code

@@ -36,7 +41,7 @@ static void aes_preparekey_arch(union aes_enckey_arch *k,
 	u32 *rndkeys = k->rndkeys;
 	u32 *inv_rndkeys = inv_k ? inv_k->inv_rndkeys : NULL;
 
-	if (static_branch_likely(&have_aes) && key_len != AES_KEYSIZE_192 &&
+	if (static_branch_likely(&have_aesni) && key_len != AES_KEYSIZE_192 &&
 	    irq_fpu_usable()) {
 		kernel_fpu_begin();
 		if (key_len == AES_KEYSIZE_128)

@@ -53,9 +58,9 @@ static void aes_encrypt_arch(const struct aes_enckey *key,
 			     u8 out[AES_BLOCK_SIZE],
 			     const u8 in[AES_BLOCK_SIZE])
 {
-	if (static_branch_likely(&have_aes) && irq_fpu_usable()) {
+	if (static_branch_likely(&have_aesni) && irq_fpu_usable()) {
 		kernel_fpu_begin();
-		aes_encrypt_aesni(key->k.rndkeys, key->nrounds, out, in);
+		aes_encrypt_aesni(out, in, key);
 		kernel_fpu_end();
 	} else {
 		aes_encrypt_generic(key->k.rndkeys, key->nrounds, out, in);

@@ -66,10 +71,9 @@ static void aes_decrypt_arch(const struct aes_key *key,
 			     u8 out[AES_BLOCK_SIZE],
 			     const u8 in[AES_BLOCK_SIZE])
 {
-	if (static_branch_likely(&have_aes) && irq_fpu_usable()) {
+	if (static_branch_likely(&have_aesni) && irq_fpu_usable()) {
 		kernel_fpu_begin();
-		aes_decrypt_aesni(key->inv_k.inv_rndkeys, key->nrounds,
-				  out, in);
+		aes_decrypt_aesni(out, in, key);
 		kernel_fpu_end();
 	} else {
 		aes_decrypt_generic(key->inv_k.inv_rndkeys, key->nrounds,

@@ -81,5 +85,5 @@ static void aes_decrypt_arch(const struct aes_key *key,
 static void aes_mod_init_arch(void)
 {
 	if (boot_cpu_has(X86_FEATURE_AES))
-		static_branch_enable(&have_aes);
+		static_branch_enable(&have_aesni);
 }

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #4 lore raw

PATCH v2 4/20 lib/crypto: x86/aes-ecb: Add AES-NI optimization

In reply to Eric Biggers

Optimize the crypto library's AES-ECB support with AES-NI, bringing its performance on par with the "ecb-aes-aesni" skcipher algorithm it will supersede.

The new assembly functions are written from scratch to fit well into the crypto library. However, they are functionally very similar to the functions in arch/x86/crypto/aesni-intel_asm.S that they will supersede and are intended to provide parity with those -- including supporting 32-bit mode, having the inner loops do 4 AES blocks per iteration, etc.

Signed-off-by Eric Biggers <[email protected]>

---
 crypto/aes.c               |  2 +-
 lib/crypto/x86/aes-aesni.S | 54 ++++++++++++++++++++++++++++++++++++++
 lib/crypto/x86/aes.h       | 35 ++++++++++++++++++++++++
 3 files changed, 90 insertions(+), 1 deletion(-)

crypto/aes.c +1 -1

index 5046b887ac9a..b1f71d2b2460 100644
@@ -610,7 +610,7 @@ static struct skcipher_alg skcipher_algs[] = {
 	{
 		.base.cra_name = "ecb(aes)",
 		.base.cra_driver_name = "ecb-aes-lib",
-		.base.cra_priority = 110,
+		.base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110,
 		.base.cra_blocksize = AES_BLOCK_SIZE,
 		.base.cra_ctxsize = sizeof(struct aes_key),
 		.base.cra_module = THIS_MODULE,

lib/crypto/x86/aes-aesni.S +54 -0

57 lines not on this page · raw message

lib/crypto/x86/aes.h +35 -0

index 06146fef06be..408625750ae9 100644
@@ -12,6 +12,8 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_aesni);
 /* The assembly code assumes the following offsets. */
 static_assert(offsetof(struct aes_enckey, nrounds) == 4);
 static_assert(offsetof(struct aes_enckey, k.rndkeys) == 16);
+static_assert(offsetof(struct aes_key, nrounds) == 4);
+static_assert(offsetof(struct aes_key, k.rndkeys) == 16);
 static_assert(offsetof(struct aes_key, inv_k.inv_rndkeys) == 256);
 
 void aes128_expandkey_aesni(u32 rndkeys[], u32 *inv_rndkeys,

@@ -81,6 +83,39 @@ static void aes_decrypt_arch(const struct aes_key *key,
 	}
 }
 
+#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_ECB)
+void aes_ecb_encrypt_aesni(u8 *dst, const u8 *src, long nblocks,
+			   const struct aes_enckey *key);
+void aes_ecb_decrypt_aesni(u8 *dst, const u8 *src, long nblocks,
+			   const struct aes_key *key);
+
+/* len is always a positive multiple of AES_BLOCK_SIZE here. */
+#define aes_ecb_encrypt_arch aes_ecb_encrypt_arch
+static bool aes_ecb_encrypt_arch(u8 *dst, const u8 *src, size_t len,
+				 const struct aes_enckey *key)
+{
+	if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable()))
+		return false;
+	kernel_fpu_begin();
+	aes_ecb_encrypt_aesni(dst, src, len / AES_BLOCK_SIZE, key);
+	kernel_fpu_end();
+	return true;
+}
+
+/* len is always a positive multiple of AES_BLOCK_SIZE here. */
+#define aes_ecb_decrypt_arch aes_ecb_decrypt_arch
+static bool aes_ecb_decrypt_arch(u8 *dst, const u8 *src, size_t len,
+				 const struct aes_key *key)
+{
+	if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable()))
+		return false;
+	kernel_fpu_begin();
+	aes_ecb_decrypt_aesni(dst, src, len / AES_BLOCK_SIZE, key);
+	kernel_fpu_end();
+	return true;
+}
+#endif /* CONFIG_CRYPTO_LIB_AES_ECB */
+
 #define aes_mod_init_arch aes_mod_init_arch
 static void aes_mod_init_arch(void)
 {

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #5 lore raw

PATCH v2 5/20 lib/crypto: x86/aes-cbc: Add AES-NI optimization

In reply to Eric Biggers

Optimize the crypto library's AES-CBC and AES-CBC-CTS support with AES-NI, bringing their performance on par with the "cbc-aes-aesni" and "cts-cbc-aes-aesni" skcipher algorithms they will supersede.

Signed-off-by Eric Biggers <[email protected]>

---
 crypto/aes.c               |   4 +-
 lib/crypto/x86/aes-aesni.S | 232 +++++++++++++++++++++++++++++++++++++
 lib/crypto/x86/aes.h       |  89 ++++++++++++++
 3 files changed, 323 insertions(+), 2 deletions(-)

crypto/aes.c +2 -2

index b1f71d2b2460..bf8acc553188 100644
@@ -625,7 +625,7 @@ static struct skcipher_alg skcipher_algs[] = {
 	{
 		.base.cra_name = "cbc(aes)",
 		.base.cra_driver_name = "cbc-aes-lib",
-		.base.cra_priority = 110,
+		.base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110,
 		.base.cra_blocksize = AES_BLOCK_SIZE,
 		.base.cra_ctxsize = sizeof(struct aes_key),
 		.base.cra_module = THIS_MODULE,

@@ -651,7 +651,7 @@ static struct skcipher_alg skcipher_algs[] = {
 	{
 		.base.cra_name = "cts(cbc(aes))",
 		.base.cra_driver_name = "cts-cbc-aes-lib",
-		.base.cra_priority = 110,
+		.base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110,
 		.base.cra_blocksize = AES_BLOCK_SIZE,
 		.base.cra_ctxsize = sizeof(struct aes_key),
 		.base.cra_module = THIS_MODULE,

lib/crypto/x86/aes-aesni.S +232 -0

241 lines not on this page · raw message

lib/crypto/x86/aes.h +89 -0

95 lines not on this page · raw message

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #6 lore raw

PATCH v2 6/20 lib/crypto: x86/aes-ctr: Add AES-NI optimization

In reply to Eric Biggers

Optimize the crypto library's AES-CTR support with AES-NI, bringing its performance on par with the "ctr-aes-aesni" skcipher algorithm it will supersede.

The new assembly function is written from scratch to fit well into the crypto library and to be more consistent with aes-ctr-avx-x86_64.S than the code in arch/x86/crypto/aesni-intel_asm.S that it will supersede. That includes using the "ctr64" convention, where the assembly code is simplified by making the C code handle incrementing the high 64 bits of the counter. Unlike the ECB, CBC, and XTS code, 32-bit support is *not* included for this one, as the existing CTR code didn't have it.

Note: the priority of ctr-aes-lib is left unchanged at 110 temporarily. It will be increased when the AVX-optimized code is migrated too.

Signed-off-by Eric Biggers <[email protected]>

---
 lib/crypto/x86/aes-aesni.S | 130 ++++++++++++++++++++++++++++++++++++-
 lib/crypto/x86/aes.h       |  54 +++++++++++++++
 2 files changed, 183 insertions(+), 1 deletion(-)

lib/crypto/x86/aes-aesni.S +129 -1

index 51c8522416d7..297fe21ba830 100644
@@ -4,7 +4,8 @@
 //
 // Copyright 2026 Google LLC
 //
-// The code in this file supports 32-bit and 64-bit CPUs, and it doesn't require
+// The code in this file supports 32-bit and 64-bit CPUs (except for
+// aes_ctr64_crypt_aesni() which supports 64-bit only), and it doesn't require
 // AVX.  It does use up to SSE4.1, which all CPUs with AES-NI have.
 #include <linux/linkage.h>

@@ -48,6 +49,12 @@
 
 .section .rodata
 .p2align 4
+#ifdef __x86_64__
+.Lbswap_mask:
+	// A mask for pshufb that byte-reflects the value.
+	.byte	15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0
+#endif
+
 .Lexpandkey_mask:
 	// A mask for pshufb that extracts the last dword, rotates it right by 8
 	// bits, and copies the result to all four dwords.

@@ -634,3 +641,124 @@ SYM_FUNC_END(aes_cbc_cts_encrypt_aesni)
 SYM_FUNC_START(aes_cbc_cts_decrypt_aesni)
 	_aes_cbc_cts_crypt	0
 SYM_FUNC_END(aes_cbc_cts_decrypt_aesni)
+
+#ifdef __x86_64__
+// void aes_ctr64_crypt_aesni(u8 *dst, const u8 *src, s64 len,
+//			      const u64 le_ctr[2],
+//			      const struct aes_enckey *key);
+SYM_FUNC_START(aes_ctr64_crypt_aesni)
+	// Arguments
+	.set	DST,		ARG0
+	.set	SRC,		ARG1
+	.set	LEN,		ARG2
+	.set	LEN32,		ARG2_32	// Used for improved code density
+	.set	LE_CTR_PTR,	ARG3	// Used as temp reg after LE_CTR is loaded
+	.set	KEY,		ARG4
+
+	// Other local variables
+	.set	AESDATA0,	%xmm0
+	.set	AESDATA1,	%xmm1
+	.set	AESDATA2,	%xmm2
+	.set	AESDATA3,	%xmm3
+	.set	LE_CTR,		%xmm4	// Current 128-bit little endian counter
+	.set	LE_CTR_INC,	%xmm5	// Initialized to (u64[])[1, 0]
+	.set	BSWAP_MASK,	%xmm6
+	.set	RNDKEY,		%xmm7
+	.set	RNDKEY_PTR,	LE_CTR_PTR  // Temporary register for _do_aes
+	.set	NROUNDS,	TMP_32	// Temporary register for _do_aes
+
+	// Initialize LE_CTR, BSWAP_MASK, and LE_CTR_INC.
+	movdqu		(LE_CTR_PTR), LE_CTR
+	movdqa		RODATA(.Lbswap_mask), BSWAP_MASK
+	mov		$1, TMP_32
+	movd		TMP_32, LE_CTR_INC
+
+	// Encrypt and XOR four blocks (64 bytes) at a time.
+	sub		$64, LEN
+	jl		.Lctr_loop4_done
+.p2align 5
+.Lctr_loop4:
+.irp i, 0,1,2,3
+	movdqa		LE_CTR, AESDATA\i
+	pshufb		BSWAP_MASK, AESDATA\i	// => big endian counter
+	paddq		LE_CTR_INC, LE_CTR
+.endr
+	_do_aes		1, 0,1,2,3
+	// AESDATA[0-3] now contain four keystream blocks.
+.irp i, 0,1,2,3
+	movdqu		\i*16(SRC), RNDKEY	// Use RNDKEY as temp register.
+	pxor		RNDKEY, AESDATA\i
+	movdqu		AESDATA\i, \i*16(DST)
+.endr
+	add		$64, SRC
+	add		$64, DST
+	sub		$64, LEN
+	jge		.Lctr_loop4
+.Lctr_loop4_done:
+	add		$64, LEN32
+	jz		.Lctr_done
+
+	// 1 <= LEN <= 63 bytes remain.  Prepare four more keystream blocks.
+.irp i, 0,1,2,3
+	movdqa		LE_CTR, AESDATA\i
+	pshufb		BSWAP_MASK, AESDATA\i	// => big endian counter
+  .if \i != 3
+	paddq		LE_CTR_INC, LE_CTR
+  .endif
+.endr
+	_do_aes		1, 0,1,2,3
+	// AESDATA[0-3] now contain four keystream blocks.
+
+	// XOR one block (16 bytes) at a time.
+	sub		$16, LEN32
+	jl		.Lctr_xor_partial
+.Lctr_xor1:
+	movdqu		(SRC), RNDKEY		// Use RNDKEY as temp register.
+	pxor		RNDKEY, AESDATA0
+	movdqu		AESDATA0, (DST)
+	movdqa		AESDATA1, AESDATA0
+	movdqa		AESDATA2, AESDATA1
+	movdqa		AESDATA3, AESDATA2
+	add		$16, SRC
+	add		$16, DST
+	sub		$16, LEN32
+	jge		.Lctr_xor1
+
+	// XOR the remaining LEN mod 16 bytes.
+.Lctr_xor_partial:
+	test		$8, LEN32
+	jz		1f
+	movq		AESDATA0, TMP
+	xor		(SRC), TMP	// XOR 8 bytes.
+	mov		TMP, (DST)
+	add		$8, SRC
+	add		$8, DST
+	psrldq		$8, AESDATA0
+1:
+	test		$4, LEN32
+	jz		2f
+	movd		AESDATA0, TMP_32
+	xor		(SRC), TMP_32	// XOR 4 bytes.
+	mov		TMP_32, (DST)
+	add		$4, SRC
+	add		$4, DST
+	psrldq		$4, AESDATA0
+2:
+	test		$2, LEN32
+	jz		3f
+	movd		AESDATA0, TMP_32
+	xor		(SRC), TMP_16	// XOR 2 bytes.
+	mov		TMP_16, (DST)
+	add		$2, SRC
+	add		$2, DST
+	psrldq		$2, AESDATA0
+3:
+	test		$1, LEN32
+	jz		.Lctr_done
+	movd		AESDATA0, TMP_32
+	xor		(SRC), TMP_8	// XOR 1 byte.
+	mov		TMP_8, (DST)
+.Lctr_done:
+	RET
+SYM_FUNC_END(aes_ctr64_crypt_aesni)
+#endif // __x86_64__

lib/crypto/x86/aes.h +54 -0

60 lines not on this page · raw message

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #7 lore raw

PATCH v2 7/20 lib/crypto: x86/aes-xts: Add AES-NI optimization

In reply to Eric Biggers

Optimize the crypto library's AES-XTS support with AES-NI, bringing its performance on par with the "xts-aes-aesni" skcipher algorithm it will supersede.

The new assembly code is written from scratch to fit well into the crypto library and to be more consistent with aes-xts-avx-x86_64.S than the code in arch/x86/crypto/aesni-intel_asm.S that it will supersede. At a high level it is quite similar though, including doing 4 blocks per iteration and supporting 32-bit mode for parity with the old code.

The new assembly code also fixes the flaw the old code had where the encrypted tweaks were spilled to the destination buffer, rather than kept entirely in registers (64-bit mode) or spilled to the stack (32-bit mode). The encrypted tweaks are secret values that should not be exposed to any code that doesn't have access to the key itself.

Note: the priority of xts-aes-lib is left unchanged at 110 temporarily. It will be increased when the AVX-optimized AES-XTS code is migrated too. Most systems use the AVX-optimized code; this commit just deals with support for older CPUs that have AES-NI but not AVX (and 32-bit).

Signed-off-by Eric Biggers <[email protected]>

---
 lib/crypto/x86/aes-aesni.S | 159 +++++++++++++++++++++++++++++++++++++
 lib/crypto/x86/aes.h       |  44 ++++++++++
 2 files changed, 203 insertions(+)

lib/crypto/x86/aes-aesni.S +159 -0

index 297fe21ba830..cbc0cc23f63e 100644
@@ -49,6 +49,17 @@
 
 .section .rodata
 .p2align 4
+.Lxts_gf_poly:
+	// For XTS: a constant used when advancing the tweak by one block by
+	// multiplying by the polynomial 'x' in GF(2^128).  The low 64 bits of
+	// this value represent the polynomial x^7 + x^2 + x + 1; it is the
+	// value that must be XOR'd into the low 64 bits of the tweak each time
+	// a 1 is carried out of the high 64 bits.
+	//
+	// The high 64 bits of this value is just the internal carry bit that
+	// exists when there's a carry out of the low 64 bits of the tweak.
+	.quad	0x87, 1
+
 #ifdef __x86_64__
 .Lbswap_mask:
 	// A mask for pshufb that byte-reflects the value.

@@ -762,3 +773,151 @@ SYM_FUNC_START(aes_ctr64_crypt_aesni)
 	RET
 SYM_FUNC_END(aes_ctr64_crypt_aesni)
 #endif // __x86_64__
+
+// Given a 128-bit XTS tweak in the xmm register \tweak, compute the next tweak
+// (by multiplying by the polynomial 'x') and write it back to \tweak.
+.macro	_next_tweak	tweak, tmp
+	pshufd		$0x13, \tweak, \tmp
+	paddq		\tweak, \tweak
+	psrad		$31, \tmp
+	pand		GF_POLY, \tmp
+	pxor		\tmp, \tweak
+.endm
+
+.macro	_aes_xts_crypt	enc
+	// Arguments
+	.set	DST,		ARG0
+	.set	SRC,		ARG1
+	.set	NBLOCKS,	ARG2
+	.set	NBLOCKS32,	ARG2_32	// Used for improved code density
+	.set	TWEAK_PTR,	ARG3
+	.set	KEY,		ARG4
+
+	// Other local variables
+#ifdef __x86_64__
+	.set	RNDKEY_PTR,	%r9
+#else
+	.set	RNDKEY_PTR,	TWEAK_PTR // TWEAK_PTR is clobbered and reloaded later.
+#endif
+	.set	NROUNDS,	TMP_32
+	.set	AESDATA0,	%xmm0
+	.set	AESDATA1,	%xmm1
+	.set	AESDATA2,	%xmm2
+	.set	AESDATA3,	%xmm3
+	.set	GF_POLY,	%xmm4
+	.set	RNDKEY,		%xmm5
+	.set	TWEAK,		%xmm6
+	.set	SAVED_TWEAK0,	%xmm7
+#ifdef __x86_64__
+	.set	SAVED_TWEAK1,	%xmm8
+	.set	SAVED_TWEAK2,	%xmm9
+#endif
+
+	_prologue	uses_arg3=2, uses_arg4=2
+#ifdef __i386__
+	// Reserve 16-byte aligned space to spill two tweaks.
+	push		%ebp
+	mov		%esp, %ebp
+	and		$~15, %esp
+	sub		$32, %esp
+#endif
+
+	movdqu		(TWEAK_PTR), TWEAK
+	movdqa		RODATA(.Lxts_gf_poly), GF_POLY
+
+	sub		$4, NBLOCKS
+	jl		.Lxts_loop4_done\@
+.p2align 5
+.Lxts_loop4\@:
+	// Load the next four source blocks into AESDATA[0-3] and XOR them with
+	// their tweaks, advancing the tweak three times in order to do so.
+	// Save the four tweaks for later; on 64-bit they all fit into
+	// registers, while on 32-bit two tweaks are spilled to the stack.
+.irp i, 0,1,2,3
+	movdqu		\i*16(SRC), AESDATA\i
+	pxor		TWEAK, AESDATA\i
+  .if \i != 3
+#ifdef __x86_64__
+	movdqa		TWEAK, SAVED_TWEAK\i
+#else
+    .if \i == 0
+	movdqa		TWEAK, SAVED_TWEAK0
+    .else
+	movdqa		TWEAK, (\i-1)*16(%esp)
+    .endif
+#endif
+	_next_tweak	TWEAK, RNDKEY
+  .endif
+.endr
+
+	// Encrypt or decrypt the blocks.
+	_do_aes		\enc, 0,1,2,3
+
+	// XOR the blocks with the saved tweaks.
+	pxor		SAVED_TWEAK0, AESDATA0
+#ifdef __x86_64__
+	pxor		SAVED_TWEAK1, AESDATA1
+	pxor		SAVED_TWEAK2, AESDATA2
+#else
+	pxor		(%esp), AESDATA1
+	pxor		16(%esp), AESDATA2
+#endif
+	pxor		TWEAK, AESDATA3
+
+	// Store the encrypted or decrypted blocks.
+.irp i, 0,1,2,3
+	movdqu		AESDATA\i, \i*16(DST)
+.endr
+
+	_next_tweak	TWEAK, RNDKEY
+	add		$64, SRC
+	add		$64, DST
+	sub		$4, NBLOCKS
+	jge		.Lxts_loop4\@
+.Lxts_loop4_done\@:
+	add		$4, NBLOCKS32
+	jz		.Lxts_done\@
+
+.Lxts_loop1\@:
+	movdqu		(SRC), AESDATA0
+	pxor		TWEAK, AESDATA0
+	_do_aes		\enc, 0
+	pxor		TWEAK, AESDATA0
+	movdqu		AESDATA0, (DST)
+	_next_tweak	TWEAK, RNDKEY
+	add		$16, SRC
+	add		$16, DST
+	dec		NBLOCKS32
+	jnz		.Lxts_loop1\@
+
+.Lxts_done\@:
+#ifdef __i386__
+	// Zeroize the stack buffer that tweaks were spilled to.
+	pxor		AESDATA0, AESDATA0
+	movdqa		AESDATA0, (%esp)
+	movdqa		AESDATA0, 16(%esp)
+	mov		%ebp, %esp
+	pop		%ebp
+#endif
+	// Store the next tweak.  On 32-bit, reload TWEAK_PTR from stack first.
+	_reload_arg3
+	movdqu		TWEAK, (TWEAK_PTR)
+	_epilogue
+.endm
+
+// void aes_xts_encrypt_aesni(u8 *dst, const u8 *src, long nblocks,
+//			      u8 tweak[AES_BLOCK_SIZE],
+//			      const struct aes_key *key);
+// void aes_xts_decrypt_aesni(u8 *dst, const u8 *src, long nblocks,
+//			      u8 tweak[AES_BLOCK_SIZE],
+//			      const struct aes_key *key);
+//
+// `tweak` must have already been encrypted by the tweak key; `key` is just the
+// main key.  To allow incremental computation, `tweak` is updated to contain
+// the next tweak.
+SYM_FUNC_START(aes_xts_encrypt_aesni)
+	_aes_xts_crypt	1
+SYM_FUNC_END(aes_xts_encrypt_aesni)
+SYM_FUNC_START(aes_xts_decrypt_aesni)
+	_aes_xts_crypt	0
+SYM_FUNC_END(aes_xts_decrypt_aesni)

lib/crypto/x86/aes.h +44 -0

50 lines not on this page · raw message

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #8 lore raw

PATCH v2 8/20 crypto: x86/aes - Drop superseded 32-bit build support

In reply to Eric Biggers

Now that bare AES, AES-ECB, AES-CBC, AES-CBC-CTS, and AES-XTS are accelerated on 32-bit x86 kernels via libaes, the equivalent code in aesni-intel is redundant. Only the 64-bit-only code in aesni-intel is still needed. Thus, drop the 32-bit build support from aesni-intel. These AES modes remain accelerated on 32-bit kernels via libaes.

aesni-intel_asm.S is left untouched, as later commits remove that file entirely one mode at a time anyway. This commit just drops 32-bit support right away to prevent bisection hazards during that.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/x86/crypto/Kconfig            |  6 ++----
 arch/x86/crypto/Makefile           | 13 +++++++------
 arch/x86/crypto/aesni-intel_glue.c | 22 +---------------------
 3 files changed, 10 insertions(+), 31 deletions(-)

arch/x86/crypto/Kconfig +2 -4

index f65d7b83702f..3c4c582e7f2d 100644
@@ -4,6 +4,7 @@ menu "Accelerated Cryptographic Algorithms for CPU (x86)"
 
 config CRYPTO_AES_NI_INTEL
 	tristate "Ciphers: AES, modes: ECB, CBC, CTS, CTR, XCTR, XTS, GCM (AES-NI/VAES)"
+	depends on 64BIT
 	select CRYPTO_AEAD
 	select CRYPTO_LIB_AES
 	select CRYPTO_LIB_GF128MUL

@@ -12,13 +13,10 @@ config CRYPTO_AES_NI_INTEL
 	  AEAD cipher: AES with GCM
 	  Length-preserving ciphers: AES with ECB, CBC, CTS, CTR, XCTR, XTS
 
-	  Architecture: x86 (32-bit and 64-bit) using:
+	  Architecture: x86_64 using:
 	  - AES-NI (AES new instructions)
 	  - VAES (Vector AES)
 
-	  Some algorithm implementations are supported only in 64-bit builds,
-	  and some have additional prerequisites such as AVX2 or AVX512.
-
 config CRYPTO_BLOWFISH_X86_64
 	tristate "Ciphers: Blowfish, modes: ECB, CBC"
 	depends on 64BIT

arch/x86/crypto/Makefile +7 -6

index e04ff8718d6b..1c3feb6d72b7 100644
@@ -40,12 +40,13 @@ obj-$(CONFIG_CRYPTO_AEGIS128_AESNI_SSE2) += aegis128-aesni.o
 aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o
 
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
-aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
-aesni-intel-$(CONFIG_64BIT) += aes-ctr-avx-x86_64.o \
-			       aes-gcm-aesni-x86_64.o \
-			       aes-gcm-vaes-avx2.o \
-			       aes-gcm-vaes-avx512.o \
-			       aes-xts-avx-x86_64.o
+aesni-intel-y := aesni-intel_asm.o \
+		 aesni-intel_glue.o \
+		 aes-ctr-avx-x86_64.o \
+		 aes-gcm-aesni-x86_64.o \
+		 aes-gcm-vaes-avx2.o \
+		 aes-gcm-vaes-avx512.o \
+		 aes-xts-avx-x86_64.o
 
 obj-$(CONFIG_CRYPTO_SM4_AESNI_AVX_X86_64) += sm4-aesni-avx-x86_64.o
 sm4-aesni-avx-x86_64-y := sm4-aesni-avx-asm_64.o sm4_aesni_avx_glue.o

arch/x86/crypto/aesni-intel_glue.c +1 ~~-21~~

index f522fff9231e..259e319ff92b 100644
@@ -80,10 +80,8 @@ asmlinkage void aesni_xts_enc(const struct crypto_aes_ctx *ctx, u8 *out,
 asmlinkage void aesni_xts_dec(const struct crypto_aes_ctx *ctx, u8 *out,
 			      const u8 *in, unsigned int len, u8 *iv);
 
-#ifdef CONFIG_X86_64
 asmlinkage void aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out,
 			      const u8 *in, unsigned int len, u8 *iv);
-#endif
 
 static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx)
 {

@@ -319,7 +317,6 @@ static int cts_cbc_decrypt(struct skcipher_request *req)
 	return skcipher_walk_done(&walk, 0);
 }
 
-#ifdef CONFIG_X86_64
 /* This is the non-AVX version. */
 static int ctr_crypt_aesni(struct skcipher_request *req)
 {

@@ -353,7 +350,6 @@ static int ctr_crypt_aesni(struct skcipher_request *req)
 	}
 	return err;
 }
-#endif
 
 static int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key,
 			    unsigned int keylen)

@@ -469,8 +465,7 @@ xts_crypt(struct skcipher_request *req, xts_encrypt_iv_func encrypt_iv,
 	 * single-scatterlist-element messages as efficiently as possible.  The
 	 * code is 64-bit specific, as it assumes no page mapping is needed.
 	 */
-	if (IS_ENABLED(CONFIG_X86_64) &&
-	    likely(req->src->length >= req->cryptlen &&
+	if (likely(req->src->length >= req->cryptlen &&
 		   req->dst->length >= req->cryptlen)) {
 		(*crypt_func)(&ctx->crypt_ctx, sg_virt(req->src),
 			      sg_virt(req->dst), req->cryptlen, req->iv);

@@ -557,7 +552,6 @@ static struct skcipher_alg aesni_skciphers[] = {
 		.setkey		= aesni_skcipher_setkey,
 		.encrypt	= cts_cbc_encrypt,
 		.decrypt	= cts_cbc_decrypt,
-#ifdef CONFIG_X86_64
 	}, {
 		.base = {
 			.cra_name		= "ctr(aes)",

@@ -574,7 +568,6 @@ static struct skcipher_alg aesni_skciphers[] = {
 		.setkey		= aesni_skcipher_setkey,
 		.encrypt	= ctr_crypt_aesni,
 		.decrypt	= ctr_crypt_aesni,
-#endif
 	}, {
 		.base = {
 			.cra_name		= "xts(aes)",

@@ -594,7 +587,6 @@ static struct skcipher_alg aesni_skciphers[] = {
 	}
 };
 
-#ifdef CONFIG_X86_64
 asmlinkage void aes_xts_encrypt_iv(const struct crypto_aes_ctx *tweak_key,
 				   u8 iv[AES_BLOCK_SIZE]);

@@ -1604,18 +1596,6 @@ static void unregister_avx_algs(void)
 	unregister_aeads(aes_gcm_algs_vaes_avx2);
 	unregister_aeads(aes_gcm_algs_vaes_avx512);
 }
-#else /* CONFIG_X86_64 */
-static struct aead_alg aes_gcm_algs_aesni[0];
-
-static int __init register_avx_algs(void)
-{
-	return 0;
-}
-
-static void unregister_avx_algs(void)
-{
-}
-#endif /* !CONFIG_X86_64 */
 
 static const struct x86_cpu_id aesni_cpu_id[] = {
 	X86_MATCH_FEATURE(X86_FEATURE_AES, NULL),

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #9 lore raw

PATCH v2 9/20 crypto: x86/aes-ecb - Remove superseded ECB skcipher

In reply to Eric Biggers

Now that the AES-ECB library functions are optimized with dedicated AES-NI code and exposed via the skcipher API via crypto/aes.c, the similar implementation in aesni-intel is redundant. Remove it.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/x86/crypto/Kconfig            |   4 +-
 arch/x86/crypto/aesni-intel_asm.S  | 121 -----------------------------
 arch/x86/crypto/aesni-intel_glue.c |  62 ---------------
 3 files changed, 2 insertions(+), 185 deletions(-)

arch/x86/crypto/Kconfig +2 -2

index 3c4c582e7f2d..f7c4243dd187 100644
@@ -3,7 +3,7 @@
 menu "Accelerated Cryptographic Algorithms for CPU (x86)"
 
 config CRYPTO_AES_NI_INTEL
-	tristate "Ciphers: AES, modes: ECB, CBC, CTS, CTR, XCTR, XTS, GCM (AES-NI/VAES)"
+	tristate "Ciphers: AES, modes: CBC, CTS, CTR, XCTR, XTS, GCM (AES-NI/VAES)"
 	depends on 64BIT
 	select CRYPTO_AEAD
 	select CRYPTO_LIB_AES

@@ -11,7 +11,7 @@ config CRYPTO_AES_NI_INTEL
 	select CRYPTO_SKCIPHER
 	help
 	  AEAD cipher: AES with GCM
-	  Length-preserving ciphers: AES with ECB, CBC, CTS, CTR, XCTR, XTS
+	  Length-preserving ciphers: AES with CBC, CTS, CTR, XCTR, XTS
 
 	  Architecture: x86_64 using:
 	  - AES-NI (AES new instructions)

arch/x86/crypto/aesni-intel_asm.S +0 ~~-121~~

127 lines not on this page · raw message

arch/x86/crypto/aesni-intel_glue.c +0 ~~-62~~

80 lines not on this page · raw message

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #10 lore raw

PATCH v2 10/20 crypto: x86/aes-cbc - Remove superseded CBC skciphers

In reply to Eric Biggers

Now that the AES-CBC (and AES-CBC-CTS) library functions are optimized with dedicated AES-NI code and exposed via the skcipher API via crypto/aes.c, the similar implementation in aesni-intel is redundant. Remove it.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/x86/crypto/Kconfig            |   4 +-
 arch/x86/crypto/aesni-intel_asm.S  | 255 -----------------------------
 arch/x86/crypto/aesni-intel_glue.c | 195 ----------------------
 3 files changed, 2 insertions(+), 452 deletions(-)

arch/x86/crypto/Kconfig +2 -2

index f7c4243dd187..79e610b5e5bc 100644
@@ -3,7 +3,7 @@
 menu "Accelerated Cryptographic Algorithms for CPU (x86)"
 
 config CRYPTO_AES_NI_INTEL
-	tristate "Ciphers: AES, modes: CBC, CTS, CTR, XCTR, XTS, GCM (AES-NI/VAES)"
+	tristate "Ciphers: AES, modes: CTR, XCTR, XTS, GCM (AES-NI/VAES)"
 	depends on 64BIT
 	select CRYPTO_AEAD
 	select CRYPTO_LIB_AES

@@ -11,7 +11,7 @@ config CRYPTO_AES_NI_INTEL
 	select CRYPTO_SKCIPHER
 	help
 	  AEAD cipher: AES with GCM
-	  Length-preserving ciphers: AES with CBC, CTS, CTR, XCTR, XTS
+	  Length-preserving ciphers: AES with CTR, XCTR, XTS
 
 	  Architecture: x86_64 using:
 	  - AES-NI (AES new instructions)

arch/x86/crypto/aesni-intel_asm.S +0 ~~-255~~

261 lines not on this page · raw message

arch/x86/crypto/aesni-intel_glue.c +0 ~~-195~~

213 lines not on this page · raw message

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #11 lore raw

PATCH v2 11/20 crypto: x86/aes-ctr - Remove superseded CTR skcipher

In reply to Eric Biggers

Now that the crypto library's AES-CTR support is optimized with AES-NI (without AVX), the similar code in aesni-intel is redundant. Remove it.

This only affects the non-AVX implementation ("ctr-aes-aesni"), not aes-ctr-avx-x86_64.S which is handled later.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/x86/crypto/aesni-intel_asm.S  | 125 -----------------------------
 arch/x86/crypto/aesni-intel_glue.c |  54 -------------
 2 files changed, 179 deletions(-)

arch/x86/crypto/aesni-intel_asm.S +0 ~~-125~~

index 16c406781b24..c4d54c4a2c23 100644
@@ -33,10 +33,6 @@
 #define KEY	%xmm2
 #define IV	%xmm3
 
-#define BSWAP_MASK %xmm10
-#define CTR	%xmm11
-#define INC	%xmm12
-
 #define GF128MUL_MASK %xmm7
 
 #ifdef __x86_64__

@@ -51,7 +47,6 @@
 #define T1	%r10
 #define TKEYP	T1
 #define T2	%r11
-#define TCTR_LOW T2
 #else
 #define AREG	%eax
 #define KEYP	%edi

@@ -610,128 +605,8 @@ SYM_FUNC_END(_aesni_dec4)
 	.byte		0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f
 	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
 	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
-#ifdef __x86_64__
-.Lbswap_mask:
-	.byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0
-#endif
 .popsection
 
-#ifdef __x86_64__
-/*
- * _aesni_inc_init:	internal ABI
- *	setup registers used by _aesni_inc
- * input:
- *	IV
- * output:
- *	CTR:	== IV, in little endian
- *	TCTR_LOW: == lower qword of CTR
- *	INC:	== 1, in little endian
- *	BSWAP_MASK == endian swapping mask
- */
-SYM_FUNC_START_LOCAL(_aesni_inc_init)
-	movaps .Lbswap_mask(%rip), BSWAP_MASK
-	movaps IV, CTR
-	pshufb BSWAP_MASK, CTR
-	mov $1, TCTR_LOW
-	movq TCTR_LOW, INC
-	movq CTR, TCTR_LOW
-	RET
-SYM_FUNC_END(_aesni_inc_init)
-
-/*
- * _aesni_inc:		internal ABI
- *	Increase IV by 1, IV is in big endian
- * input:
- *	IV
- *	CTR:	== IV, in little endian
- *	TCTR_LOW: == lower qword of CTR
- *	INC:	== 1, in little endian
- *	BSWAP_MASK == endian swapping mask
- * output:
- *	IV:	Increase by 1
- * changed:
- *	CTR:	== output IV, in little endian
- *	TCTR_LOW: == lower qword of CTR
- */
-SYM_FUNC_START_LOCAL(_aesni_inc)
-	paddq INC, CTR
-	add $1, TCTR_LOW
-	jnc .Linc_low
-	pslldq $8, INC
-	paddq INC, CTR
-	psrldq $8, INC
-.Linc_low:
-	movaps CTR, IV
-	pshufb BSWAP_MASK, IV
-	RET
-SYM_FUNC_END(_aesni_inc)
-
-/*
- * void aesni_ctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
- *		      size_t len, u8 *iv)
- */
-SYM_FUNC_START(aesni_ctr_enc)
-	ANNOTATE_NOENDBR
-	FRAME_BEGIN
-	cmp $16, LEN
-	jb .Lctr_enc_just_ret
-	mov 480(KEYP), KLEN
-	movups (IVP), IV
-	call _aesni_inc_init
-	cmp $64, LEN
-	jb .Lctr_enc_loop1
-.align 4
-.Lctr_enc_loop4:
-	movaps IV, STATE1
-	call _aesni_inc
-	movups (INP), IN1
-	movaps IV, STATE2
-	call _aesni_inc
-	movups 0x10(INP), IN2
-	movaps IV, STATE3
-	call _aesni_inc
-	movups 0x20(INP), IN3
-	movaps IV, STATE4
-	call _aesni_inc
-	movups 0x30(INP), IN4
-	call _aesni_enc4
-	pxor IN1, STATE1
-	movups STATE1, (OUTP)
-	pxor IN2, STATE2
-	movups STATE2, 0x10(OUTP)
-	pxor IN3, STATE3
-	movups STATE3, 0x20(OUTP)
-	pxor IN4, STATE4
-	movups STATE4, 0x30(OUTP)
-	sub $64, LEN
-	add $64, INP
-	add $64, OUTP
-	cmp $64, LEN
-	jge .Lctr_enc_loop4
-	cmp $16, LEN
-	jb .Lctr_enc_ret
-.align 4
-.Lctr_enc_loop1:
-	movaps IV, STATE
-	call _aesni_inc
-	movups (INP), IN
-	call _aesni_enc1
-	pxor IN, STATE
-	movups STATE, (OUTP)
-	sub $16, LEN
-	add $16, INP
-	add $16, OUTP
-	cmp $16, LEN
-	jge .Lctr_enc_loop1
-.Lctr_enc_ret:
-	movups IV, (IVP)
-.Lctr_enc_just_ret:
-	FRAME_END
-	RET
-SYM_FUNC_END(aesni_ctr_enc)
-
-#endif
-
 .section	.rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16
 .align 16
 .Lgf128mul_x_ble_mask:

arch/x86/crypto/aesni-intel_glue.c +0 ~~-54~~

78 lines not on this page · raw message

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #12 lore raw

PATCH v2 12/20 crypto: x86/aes-xts - Remove superseded XTS skcipher

In reply to Eric Biggers

Now that the crypto library's AES-XTS support is optimized with AES-NI (without AVX), the similar code in aesni-intel is redundant. Remove it.

This only affects the non-AVX implementation ("xts-aes-aesni"), not aes-xts-avx-x86_64.S which is handled later.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/x86/crypto/aesni-intel_asm.S  | 617 -----------------------------
 arch/x86/crypto/aesni-intel_glue.c |  69 +---
 2 files changed, 1 insertion(+), 685 deletions(-)

arch/x86/crypto/aesni-intel_asm.S +0 ~~-617~~

index c4d54c4a2c23..b12a0f2bf006 100644
@@ -20,30 +20,11 @@
 #include <linux/objtool.h>
 #include <asm/frame.h>
 
-#define STATE1	%xmm0
-#define STATE2	%xmm4
-#define STATE3	%xmm5
-#define STATE4	%xmm6
-#define STATE	STATE1
-#define IN1	%xmm1
-#define IN2	%xmm7
-#define IN3	%xmm8
-#define IN4	%xmm9
-#define IN	IN1
-#define KEY	%xmm2
-#define IV	%xmm3
-
-#define GF128MUL_MASK %xmm7
-
 #ifdef __x86_64__
 #define AREG	%rax
 #define KEYP	%rdi
 #define OUTP	%rsi
 #define UKEYP	OUTP
-#define INP	%rdx
-#define LEN	%rcx
-#define IVP	%r8
-#define KLEN	%r9d
 #define T1	%r10
 #define TKEYP	T1
 #define T2	%r11

@@ -52,10 +33,6 @@
 #define KEYP	%edi
 #define OUTP	AREG
 #define UKEYP	OUTP
-#define INP	%edx
-#define LEN	%esi
-#define IVP	%ebp
-#define KLEN	%ebx
 #define T1	%ecx
 #define TKEYP	T1
 #endif

@@ -241,597 +218,3 @@ SYM_FUNC_START(aesni_set_key)
 	FRAME_END
 	RET
 SYM_FUNC_END(aesni_set_key)
-
-/*
- * void aesni_enc(const void *ctx, u8 *dst, const u8 *src)
- */
-SYM_FUNC_START(aesni_enc)
-	FRAME_BEGIN
-#ifndef __x86_64__
-	pushl KEYP
-	pushl KLEN
-	movl (FRAME_OFFSET+12)(%esp), KEYP	# ctx
-	movl (FRAME_OFFSET+16)(%esp), OUTP	# dst
-	movl (FRAME_OFFSET+20)(%esp), INP	# src
-#endif
-	movl 480(KEYP), KLEN		# key length
-	movups (INP), STATE		# input
-	call _aesni_enc1
-	movups STATE, (OUTP)		# output
-#ifndef __x86_64__
-	popl KLEN
-	popl KEYP
-#endif
-	FRAME_END
-	RET
-SYM_FUNC_END(aesni_enc)
-
-/*
- * _aesni_enc1:		internal ABI
- * input:
- *	KEYP:		key struct pointer
- *	KLEN:		round count
- *	STATE:		initial state (input)
- * output:
- *	STATE:		finial state (output)
- * changed:
- *	KEY
- *	TKEYP (T1)
- */
-SYM_FUNC_START_LOCAL(_aesni_enc1)
-	movaps (KEYP), KEY		# key
-	mov KEYP, TKEYP
-	pxor KEY, STATE		# round 0
-	add $0x30, TKEYP
-	cmp $24, KLEN
-	jb .Lenc128
-	lea 0x20(TKEYP), TKEYP
-	je .Lenc192
-	add $0x20, TKEYP
-	movaps -0x60(TKEYP), KEY
-	aesenc KEY, STATE
-	movaps -0x50(TKEYP), KEY
-	aesenc KEY, STATE
-.align 4
-.Lenc192:
-	movaps -0x40(TKEYP), KEY
-	aesenc KEY, STATE
-	movaps -0x30(TKEYP), KEY
-	aesenc KEY, STATE
-.align 4
-.Lenc128:
-	movaps -0x20(TKEYP), KEY
-	aesenc KEY, STATE
-	movaps -0x10(TKEYP), KEY
-	aesenc KEY, STATE
-	movaps (TKEYP), KEY
-	aesenc KEY, STATE
-	movaps 0x10(TKEYP), KEY
-	aesenc KEY, STATE
-	movaps 0x20(TKEYP), KEY
-	aesenc KEY, STATE
-	movaps 0x30(TKEYP), KEY
-	aesenc KEY, STATE
-	movaps 0x40(TKEYP), KEY
-	aesenc KEY, STATE
-	movaps 0x50(TKEYP), KEY
-	aesenc KEY, STATE
-	movaps 0x60(TKEYP), KEY
-	aesenc KEY, STATE
-	movaps 0x70(TKEYP), KEY
-	aesenclast KEY, STATE
-	RET
-SYM_FUNC_END(_aesni_enc1)
-
-/*
- * _aesni_enc4:	internal ABI
- * input:
- *	KEYP:		key struct pointer
- *	KLEN:		round count
- *	STATE1:		initial state (input)
- *	STATE2
- *	STATE3
- *	STATE4
- * output:
- *	STATE1:		finial state (output)
- *	STATE2
- *	STATE3
- *	STATE4
- * changed:
- *	KEY
- *	TKEYP (T1)
- */
-SYM_FUNC_START_LOCAL(_aesni_enc4)
-	movaps (KEYP), KEY		# key
-	mov KEYP, TKEYP
-	pxor KEY, STATE1		# round 0
-	pxor KEY, STATE2
-	pxor KEY, STATE3
-	pxor KEY, STATE4
-	add $0x30, TKEYP
-	cmp $24, KLEN
-	jb .L4enc128
-	lea 0x20(TKEYP), TKEYP
-	je .L4enc192
-	add $0x20, TKEYP
-	movaps -0x60(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-	movaps -0x50(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-#.align 4
-.L4enc192:
-	movaps -0x40(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-	movaps -0x30(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-#.align 4
-.L4enc128:
-	movaps -0x20(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-	movaps -0x10(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-	movaps (TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-	movaps 0x10(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-	movaps 0x20(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-	movaps 0x30(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-	movaps 0x40(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-	movaps 0x50(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-	movaps 0x60(TKEYP), KEY
-	aesenc KEY, STATE1
-	aesenc KEY, STATE2
-	aesenc KEY, STATE3
-	aesenc KEY, STATE4
-	movaps 0x70(TKEYP), KEY
-	aesenclast KEY, STATE1		# last round
-	aesenclast KEY, STATE2
-	aesenclast KEY, STATE3
-	aesenclast KEY, STATE4
-	RET
-SYM_FUNC_END(_aesni_enc4)
-
-/*
- * _aesni_dec1:		internal ABI
- * input:
- *	KEYP:		key struct pointer
- *	KLEN:		key length
- *	STATE:		initial state (input)
- * output:
- *	STATE:		finial state (output)
- * changed:
- *	KEY
- *	TKEYP (T1)
- */
-SYM_FUNC_START_LOCAL(_aesni_dec1)
-	movaps (KEYP), KEY		# key
-	mov KEYP, TKEYP
-	pxor KEY, STATE		# round 0
-	add $0x30, TKEYP
-	cmp $24, KLEN
-	jb .Ldec128
-	lea 0x20(TKEYP), TKEYP
-	je .Ldec192
-	add $0x20, TKEYP
-	movaps -0x60(TKEYP), KEY
-	aesdec KEY, STATE
-	movaps -0x50(TKEYP), KEY
-	aesdec KEY, STATE
-.align 4
-.Ldec192:
-	movaps -0x40(TKEYP), KEY
-	aesdec KEY, STATE
-	movaps -0x30(TKEYP), KEY
-	aesdec KEY, STATE
-.align 4
-.Ldec128:
-	movaps -0x20(TKEYP), KEY
-	aesdec KEY, STATE
-	movaps -0x10(TKEYP), KEY
-	aesdec KEY, STATE
-	movaps (TKEYP), KEY
-	aesdec KEY, STATE
-	movaps 0x10(TKEYP), KEY
-	aesdec KEY, STATE
-	movaps 0x20(TKEYP), KEY
-	aesdec KEY, STATE
-	movaps 0x30(TKEYP), KEY
-	aesdec KEY, STATE
-	movaps 0x40(TKEYP), KEY
-	aesdec KEY, STATE
-	movaps 0x50(TKEYP), KEY
-	aesdec KEY, STATE
-	movaps 0x60(TKEYP), KEY
-	aesdec KEY, STATE
-	movaps 0x70(TKEYP), KEY
-	aesdeclast KEY, STATE
-	RET
-SYM_FUNC_END(_aesni_dec1)
-
-/*
- * _aesni_dec4:	internal ABI
- * input:
- *	KEYP:		key struct pointer
- *	KLEN:		key length
- *	STATE1:		initial state (input)
- *	STATE2
- *	STATE3
- *	STATE4
- * output:
- *	STATE1:		finial state (output)
- *	STATE2
- *	STATE3
- *	STATE4
- * changed:
- *	KEY
- *	TKEYP (T1)
- */
-SYM_FUNC_START_LOCAL(_aesni_dec4)
-	movaps (KEYP), KEY		# key
-	mov KEYP, TKEYP
-	pxor KEY, STATE1		# round 0
-	pxor KEY, STATE2
-	pxor KEY, STATE3
-	pxor KEY, STATE4
-	add $0x30, TKEYP
-	cmp $24, KLEN
-	jb .L4dec128
-	lea 0x20(TKEYP), TKEYP
-	je .L4dec192
-	add $0x20, TKEYP
-	movaps -0x60(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-	movaps -0x50(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-.align 4
-.L4dec192:
-	movaps -0x40(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-	movaps -0x30(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-.align 4
-.L4dec128:
-	movaps -0x20(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-	movaps -0x10(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-	movaps (TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-	movaps 0x10(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-	movaps 0x20(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-	movaps 0x30(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-	movaps 0x40(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-	movaps 0x50(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-	movaps 0x60(TKEYP), KEY
-	aesdec KEY, STATE1
-	aesdec KEY, STATE2
-	aesdec KEY, STATE3
-	aesdec KEY, STATE4
-	movaps 0x70(TKEYP), KEY
-	aesdeclast KEY, STATE1		# last round
-	aesdeclast KEY, STATE2
-	aesdeclast KEY, STATE3
-	aesdeclast KEY, STATE4
-	RET
-SYM_FUNC_END(_aesni_dec4)
-
-.pushsection .rodata
-.align 16
-.Lcts_permute_table:
-	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
-	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
-	.byte		0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07
-	.byte		0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f
-	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
-	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
-.popsection
-
-.section	.rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16
-.align 16
-.Lgf128mul_x_ble_mask:
-	.octa 0x00000000000000010000000000000087
-.previous
-
-/*
- * _aesni_gf128mul_x_ble: Multiply in GF(2^128) for XTS IVs
- * input:
- *	IV:	current IV
- *	GF128MUL_MASK == mask with 0x87 and 0x01
- * output:
- *	IV:	next IV
- * changed:
- *	KEY:	== temporary value
- */
-.macro _aesni_gf128mul_x_ble
-	pshufd $0x13, IV, KEY
-	paddq IV, IV
-	psrad $31, KEY
-	pand GF128MUL_MASK, KEY
-	pxor KEY, IV
-.endm
-
-.macro	_aesni_xts_crypt	enc
-	FRAME_BEGIN
-#ifndef __x86_64__
-	pushl IVP
-	pushl LEN
-	pushl KEYP
-	pushl KLEN
-	movl (FRAME_OFFSET+20)(%esp), KEYP	# ctx
-	movl (FRAME_OFFSET+24)(%esp), OUTP	# dst
-	movl (FRAME_OFFSET+28)(%esp), INP	# src
-	movl (FRAME_OFFSET+32)(%esp), LEN	# len
-	movl (FRAME_OFFSET+36)(%esp), IVP	# iv
-	movdqa .Lgf128mul_x_ble_mask, GF128MUL_MASK
-#else
-	movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK
-#endif
-	movups (IVP), IV
-
-	mov 480(KEYP), KLEN
-.if !\enc
-	add $240, KEYP
-
-	test $15, LEN
-	jz .Lxts_loop4\@
-	sub $16, LEN
-.endif
-
-.Lxts_loop4\@:
-	sub $64, LEN
-	jl .Lxts_1x\@
-
-	movdqa IV, STATE1
-	movdqu 0x00(INP), IN
-	pxor IN, STATE1
-	movdqu IV, 0x00(OUTP)
-
-	_aesni_gf128mul_x_ble
-	movdqa IV, STATE2
-	movdqu 0x10(INP), IN
-	pxor IN, STATE2
-	movdqu IV, 0x10(OUTP)
-
-	_aesni_gf128mul_x_ble
-	movdqa IV, STATE3
-	movdqu 0x20(INP), IN
-	pxor IN, STATE3
-	movdqu IV, 0x20(OUTP)
-
-	_aesni_gf128mul_x_ble
-	movdqa IV, STATE4
-	movdqu 0x30(INP), IN
-	pxor IN, STATE4
-	movdqu IV, 0x30(OUTP)
-
-.if \enc
-	call _aesni_enc4
-.else
-	call _aesni_dec4
-.endif
-
-	movdqu 0x00(OUTP), IN
-	pxor IN, STATE1
-	movdqu STATE1, 0x00(OUTP)
-
-	movdqu 0x10(OUTP), IN
-	pxor IN, STATE2
-	movdqu STATE2, 0x10(OUTP)
-
-	movdqu 0x20(OUTP), IN
-	pxor IN, STATE3
-	movdqu STATE3, 0x20(OUTP)
-
-	movdqu 0x30(OUTP), IN
-	pxor IN, STATE4
-	movdqu STATE4, 0x30(OUTP)
-
-	_aesni_gf128mul_x_ble
-
-	add $64, INP
-	add $64, OUTP
-	test LEN, LEN
-	jnz .Lxts_loop4\@
-
-.Lxts_ret_iv\@:
-	movups IV, (IVP)
-
-.Lxts_ret\@:
-#ifndef __x86_64__
-	popl KLEN
-	popl KEYP
-	popl LEN
-	popl IVP
-#endif
-	FRAME_END
-	RET
-
-.Lxts_1x\@:
-	add $64, LEN
-	jz .Lxts_ret_iv\@
-.if \enc
-	sub $16, LEN
-	jl .Lxts_cts4\@
-.endif
-
-.Lxts_loop1\@:
-	movdqu (INP), STATE
-.if \enc
-	pxor IV, STATE
-	call _aesni_enc1
-.else
-	add $16, INP
-	sub $16, LEN
-	jl .Lxts_cts1\@
-	pxor IV, STATE
-	call _aesni_dec1
-.endif
-	pxor IV, STATE
-	_aesni_gf128mul_x_ble
-
-	test LEN, LEN
-	jz .Lxts_out\@
-
-.if \enc
-	add $16, INP
-	sub $16, LEN
-	jl .Lxts_cts1\@
-.endif
-
-	movdqu STATE, (OUTP)
-	add $16, OUTP
-	jmp .Lxts_loop1\@
-
-.Lxts_out\@:
-	movdqu STATE, (OUTP)
-	jmp .Lxts_ret_iv\@
-
-.if \enc
-.Lxts_cts4\@:
-	movdqa STATE4, STATE
-	sub $16, OUTP
-.Lxts_cts1\@:
-.else
-.Lxts_cts1\@:
-	movdqa IV, STATE4
-	_aesni_gf128mul_x_ble
-
-	pxor IV, STATE
-	call _aesni_dec1
-	pxor IV, STATE
-.endif
-#ifndef __x86_64__
-	lea .Lcts_permute_table, T1
-#else
-	lea .Lcts_permute_table(%rip), T1
-#endif
-	add LEN, INP		/* rewind input pointer */
-	add $16, LEN		/* # bytes in final block */
-	movups (INP), IN1
-
-	mov T1, IVP
-	add $32, IVP
-	add LEN, T1
-	sub LEN, IVP
-	add OUTP, LEN
-
-	movups (T1), %xmm4
-	movaps STATE, IN2
-	pshufb %xmm4, STATE
-	movups STATE, (LEN)
-
-	movups (IVP), %xmm0
-	pshufb %xmm0, IN1
-	pblendvb IN2, IN1
-	movaps IN1, STATE
-
-.if \enc
-	pxor IV, STATE
-	call _aesni_enc1
-	pxor IV, STATE
-.else
-	pxor STATE4, STATE
-	call _aesni_dec1
-	pxor STATE4, STATE
-.endif
-
-	movups STATE, (OUTP)
-	jmp .Lxts_ret\@
-.endm
-
-/*
- * void aesni_xts_enc(const struct crypto_aes_ctx *ctx, u8 *dst,
- *		      const u8 *src, unsigned int len, le128 *iv)
- */
-SYM_FUNC_START(aesni_xts_enc)
-	_aesni_xts_crypt	1
-SYM_FUNC_END(aesni_xts_enc)
-
-/*
- * void aesni_xts_dec(const struct crypto_aes_ctx *ctx, u8 *dst,
- *		      const u8 *src, unsigned int len, le128 *iv)
- */
-SYM_FUNC_START(aesni_xts_dec)
-	_aesni_xts_crypt	0
-SYM_FUNC_END(aesni_xts_dec)

arch/x86/crypto/aesni-intel_glue.c +1 ~~-68~~

102 lines not on this page · raw message

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #13 lore raw

PATCH v2 13/20 lib/crypto: x86/aes-ctr: Migrate AVX-optimized code into library

In reply to Eric Biggers

Migrate aes-ctr-avx-x86_64.S into lib/crypto/, wiring it up to the CTR and XCTR library functions instead of the crypto_skcipher API. It still remains available through crypto_skcipher via crypto/aes.c.

Some slight adjustments to the assembly code were needed:

- Take 'struct aes_enckey' instead of 'struct crypto_aes_ctx'.

- Upgrade the length argument from 32-bit to 64-bit so that it's compatible with the library's use of size_t (at least assuming no lengths over S64_MAX, which seems quite safe to assume...)

- Remove the CFI stubs, as the functions are now called directly.

- Adjust the argument order to match the caller. Not strictly required, but it's easiest to handle this now when changing the function prototypes anyway and adding the new glue code.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/x86/crypto/Kconfig                       |   4 +-
 arch/x86/crypto/Makefile                      |   1 -
 arch/x86/crypto/aesni-intel_glue.c            | 153 ------------------
 crypto/aes.c                                  |   7 +-
 lib/crypto/Makefile                           |   4 +
 .../crypto/x86}/aes-ctr-avx-x86_64.S          | 103 ++++++------
 lib/crypto/x86/aes.h                          |  80 ++++++++-
 7 files changed, 140 insertions(+), 212 deletions(-)
 rename {arch/x86/crypto => lib/crypto/x86}/aes-ctr-avx-x86_64.S (88%)

arch/x86/crypto/Kconfig +2 -2

index 79e610b5e5bc..778ab164c571 100644
@@ -3,7 +3,7 @@
 menu "Accelerated Cryptographic Algorithms for CPU (x86)"
 
 config CRYPTO_AES_NI_INTEL
-	tristate "Ciphers: AES, modes: CTR, XCTR, XTS, GCM (AES-NI/VAES)"
+	tristate "Ciphers: AES, modes: XTS, GCM (AES-NI/VAES)"
 	depends on 64BIT
 	select CRYPTO_AEAD
 	select CRYPTO_LIB_AES

@@ -11,7 +11,7 @@ config CRYPTO_AES_NI_INTEL
 	select CRYPTO_SKCIPHER
 	help
 	  AEAD cipher: AES with GCM
-	  Length-preserving ciphers: AES with CTR, XCTR, XTS
+	  Length-preserving ciphers: AES with XTS
 
 	  Architecture: x86_64 using:
 	  - AES-NI (AES new instructions)

arch/x86/crypto/Makefile +0 -1

index 1c3feb6d72b7..0c016ba87373 100644
@@ -42,7 +42,6 @@ aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 aesni-intel-y := aesni-intel_asm.o \
 		 aesni-intel_glue.o \
-		 aes-ctr-avx-x86_64.o \
 		 aes-gcm-aesni-x86_64.o \
 		 aes-gcm-vaes-avx2.o \
 		 aes-gcm-vaes-avx512.o \

arch/x86/crypto/aesni-intel_glue.c +0 ~~-153~~

189 lines not on this page · raw message

crypto/aes.c +3 -4

index bf8acc553188..568a900bfec8 100644
@@ -667,7 +667,7 @@ static struct skcipher_alg skcipher_algs[] = {
 	{
 		.base.cra_name = "ctr(aes)",
 		.base.cra_driver_name = "ctr-aes-lib",
-		.base.cra_priority = 110,
+		.base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110,
 		.base.cra_blocksize = 1,
 		.base.cra_ctxsize = sizeof(struct aes_enckey),
 		.base.cra_module = THIS_MODULE,

@@ -684,7 +684,7 @@ static struct skcipher_alg skcipher_algs[] = {
 	{
 		.base.cra_name = "xctr(aes)",
 		.base.cra_driver_name = "xctr-aes-lib",
-		.base.cra_priority = 110,
+		.base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110,
 		.base.cra_blocksize = 1,
 		.base.cra_ctxsize = sizeof(struct aes_enckey),
 		.base.cra_module = THIS_MODULE,

@@ -1044,8 +1044,7 @@ static struct aead_alg aead_algs[] = {
 	  IS_ENABLED(CONFIG_PPC) || \
 	  IS_ENABLED(CONFIG_RISCV) || \
 	  IS_ENABLED(CONFIG_S390) || \
-	  IS_ENABLED(CONFIG_SPARC) || \
-	  IS_ENABLED(CONFIG_X86))
+	  IS_ENABLED(CONFIG_SPARC))
 	{
 		.base.cra_name = "ccm(aes)",
 		.base.cra_driver_name = "ccm-aes-lib",

lib/crypto/Makefile +4 -0

index ca068df1f71f..5d5484fc78ea 100644
@@ -52,7 +52,11 @@ endif # CONFIG_PPC
 
 libaes-$(CONFIG_RISCV) += riscv/aes-riscv64-zvkned.o
 libaes-$(CONFIG_SPARC) += sparc/aes_asm.o
+
 libaes-$(CONFIG_X86) += x86/aes-aesni.o
+ifneq ($(CONFIG_CRYPTO_LIB_AES_CTR),)
+libaes-$(CONFIG_X86_64) += x86/aes-ctr-avx-x86_64.o
+endif
 endif # CONFIG_CRYPTO_LIB_AES_ARCH
 
 # clean-files must be defined unconditionally

lib/crypto/x86/aes-ctr-avx-x86_64.S renamed from arch/x86/crypto/aes-ctr-avx-x86_64.S +54 ~~-49~~

244 lines not on this page · raw message

lib/crypto/x86/aes.h +77 -3

108 lines not on this page · raw message

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #14 lore raw

PATCH v2 14/20 lib/crypto: x86/aes-xts: Migrate AVX-optimized code into library

In reply to Eric Biggers

Migrate aes-xts-avx-x86_64.S into lib/crypto/, wiring it up to the XTS library functions instead of the crypto_skcipher API. It still remains available through crypto_skcipher via crypto/aes.c.

Some adjustments to the assembly code were needed:

- Take 'struct aes_key' instead of 'struct crypto_aes_ctx'.

- Remove the ciphertext stealing support from the assembly code, as the library implements it in a generic way instead. (This does slightly reduce performance when the length isn't a multiple of 16 bytes; however, that case seems to never be reached in practice in the kernel. So it makes sense to not extensively optimize for it yet.)

- Change 'int len' to 'long nblocks' for compatibility with the library's use of size_t lengths.

- Remove the CFI stubs, as the functions are now called directly.

- Adjust the argument order to match the caller. Not strictly required, but it's easiest to handle this now when changing the function prototypes anyway and adding the new glue code.

This makes the remaining code in aesni-intel_asm.S (which just handled key expansion) unused, so remove that too.

Don't remove the selection of CRYPTO_SKCIPHER from CRYPTO_AES_NI_INTEL yet, since it's still needed for the "skcipher_walk" functions.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/x86/crypto/Kconfig                       |   6 +-
 arch/x86/crypto/Makefile                      |   6 +-
 arch/x86/crypto/aesni-intel_asm.S             | 220 ----------------
 arch/x86/crypto/aesni-intel_glue.c            | 239 +-----------------
 crypto/aes.c                                  |   2 +-
 lib/crypto/Makefile                           |   3 +
 .../crypto/x86}/aes-xts-avx-x86_64.S          | 179 ++++---------
 lib/crypto/x86/aes.h                          |  67 ++++-
 8 files changed, 121 insertions(+), 601 deletions(-)
 delete mode 100644 arch/x86/crypto/aesni-intel_asm.S
 rename {arch/x86/crypto => lib/crypto/x86}/aes-xts-avx-x86_64.S (81%)

arch/x86/crypto/Kconfig +4 -2

index 778ab164c571..c1842c2d7055 100644
@@ -3,7 +3,7 @@
 menu "Accelerated Cryptographic Algorithms for CPU (x86)"
 
 config CRYPTO_AES_NI_INTEL
-	tristate "Ciphers: AES, modes: XTS, GCM (AES-NI/VAES)"
+	tristate "Ciphers: AES, modes: GCM (AES-NI/VAES)"
 	depends on 64BIT
 	select CRYPTO_AEAD
 	select CRYPTO_LIB_AES

@@ -11,12 +11,14 @@ config CRYPTO_AES_NI_INTEL
 	select CRYPTO_SKCIPHER
 	help
 	  AEAD cipher: AES with GCM
-	  Length-preserving ciphers: AES with XTS
 
 	  Architecture: x86_64 using:
 	  - AES-NI (AES new instructions)
 	  - VAES (Vector AES)
 
+	  Note: this option no longer provides the accelerated XTS, CBC, CTR,
+	  and ECB code.  For those just use CRYPTO_XTS, CRYPTO_CBC, etc.
+
 config CRYPTO_BLOWFISH_X86_64
 	tristate "Ciphers: Blowfish, modes: ECB, CBC"
 	depends on 64BIT

arch/x86/crypto/Makefile +2 -4

index 0c016ba87373..09bdea45bc3e 100644
@@ -40,12 +40,10 @@ obj-$(CONFIG_CRYPTO_AEGIS128_AESNI_SSE2) += aegis128-aesni.o
 aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o
 
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
-aesni-intel-y := aesni-intel_asm.o \
-		 aesni-intel_glue.o \
+aesni-intel-y := aesni-intel_glue.o \
 		 aes-gcm-aesni-x86_64.o \
 		 aes-gcm-vaes-avx2.o \
-		 aes-gcm-vaes-avx512.o \
-		 aes-xts-avx-x86_64.o
+		 aes-gcm-vaes-avx512.o
 
 obj-$(CONFIG_CRYPTO_SM4_AESNI_AVX_X86_64) += sm4-aesni-avx-x86_64.o
 sm4-aesni-avx-x86_64-y := sm4-aesni-avx-asm_64.o sm4_aesni_avx_glue.o

arch/x86/crypto/aesni-intel_asm.S deleted +0 ~~-220~~

220 lines not on this page · raw message

arch/x86/crypto/aesni-intel_glue.c +4 ~~-235~~

296 lines not on this page · raw message

crypto/aes.c +1 -1

index 568a900bfec8..756fce2157be 100644
@@ -701,7 +701,7 @@ static struct skcipher_alg skcipher_algs[] = {
 	{
 		.base.cra_name = "xts(aes)",
 		.base.cra_driver_name = "xts-aes-lib",
-		.base.cra_priority = 110,
+		.base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110,
 		.base.cra_blocksize = AES_BLOCK_SIZE,
 		.base.cra_ctxsize = sizeof(struct aes_xts_key),
 		.base.cra_module = THIS_MODULE,

lib/crypto/Makefile +3 -0

index 5d5484fc78ea..02d89a226377 100644
@@ -57,6 +57,9 @@ libaes-$(CONFIG_X86) += x86/aes-aesni.o
 ifneq ($(CONFIG_CRYPTO_LIB_AES_CTR),)
 libaes-$(CONFIG_X86_64) += x86/aes-ctr-avx-x86_64.o
 endif
+ifneq ($(CONFIG_CRYPTO_LIB_AES_XTS),)
+libaes-$(CONFIG_X86_64) += x86/aes-xts-avx-x86_64.o
+endif
 endif # CONFIG_CRYPTO_LIB_AES_ARCH
 
 # clean-files must be defined unconditionally

lib/crypto/x86/aes-xts-avx-x86_64.S renamed from arch/x86/crypto/aes-xts-avx-x86_64.S +46 ~~-133~~

294 lines not on this page · raw message

lib/crypto/x86/aes.h +61 -6

85 lines not on this page · raw message

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #15 lore raw

PATCH v2 15/20 lib/crypto: riscv/aes: Copy aes-macros.S to library

In reply to Eric Biggers

Copy arch/riscv/crypto/aes-macros.S to lib/crypto/riscv/aes-macros.S and make lib/crypto/riscv/aes-riscv64-zvkned.S include the latter copy.

This makes it possible to change these macros without interfering with the remaining code in arch/riscv/crypto/.

Of course, the copy in arch/riscv/crypto/ will be removed once the rest of the AES code there is migrated to the library.

Signed-off-by Eric Biggers <[email protected]>

---
 lib/crypto/riscv/aes-macros.S         | 166 ++++++++++++++++++++++++++
 lib/crypto/riscv/aes-riscv64-zvkned.S |   2 +-
 2 files changed, 167 insertions(+), 1 deletion(-)
 create mode 100644 lib/crypto/riscv/aes-macros.S

lib/crypto/riscv/aes-macros.S new file +166 -0

new file mode 100644
index 000000000000..1384164621a5
@@ -0,0 +1,166 @@
+/* SPDX-License-Identifier: Apache-2.0 OR BSD-2-Clause */
+//
+// This file is dual-licensed, meaning that you can use it under your
+// choice of either of the following two licenses:
+//
+// Copyright 2023 The OpenSSL Project Authors. All Rights Reserved.
+//
+// Licensed under the Apache License 2.0 (the "License"). You can obtain
+// a copy in the file LICENSE in the source distribution or at
+// https://www.openssl.org/source/license.html
+//
+// or
+//
+// Copyright (c) 2023, Christoph Müllner <[email protected]>
+// Copyright (c) 2023, Phoebe Chen <[email protected]>
+// Copyright (c) 2023, Jerry Shih <[email protected]>
+// Copyright 2024 Google LLC
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+//    notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+//    notice, this list of conditions and the following disclaimer in the
+//    documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+// This file contains macros that are shared by the other aes-*.S files.  The
+// generated code of these macros depends on the following RISC-V extensions:
+// - RV64I
+// - RISC-V Vector ('V') with VLEN >= 128
+// - RISC-V Vector AES block cipher extension ('Zvkned')
+
+// Loads the AES round keys from \keyp into vector registers and jumps to code
+// specific to the length of the key.  Specifically:
+//   - If AES-128, loads round keys into v1-v11 and jumps to \label128.
+//   - If AES-192, loads round keys into v1-v13 and jumps to \label192.
+//   - If AES-256, loads round keys into v1-v15 and continues onwards.
+//
+// Also sets vl=4 and vtype=e32,m1,ta,ma.  Clobbers t0 and t1.
+.macro	aes_begin	keyp, label128, label192, key_len
+.ifb \key_len
+	lwu		t0, 480(\keyp)	// t0 = key length in bytes
+.endif
+	li		t1, 24		// t1 = key length for AES-192
+	vsetivli	zero, 4, e32, m1, ta, ma
+	vle32.v		v1, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v2, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v3, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v4, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v5, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v6, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v7, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v8, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v9, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v10, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v11, (\keyp)
+.ifb \key_len
+	blt		t0, t1, \label128	// If AES-128, goto label128.
+.else
+	blt		\key_len, t1, \label128	// If AES-128, goto label128.
+.endif
+	addi		\keyp, \keyp, 16
+	vle32.v		v12, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v13, (\keyp)
+.ifb \key_len
+	beq		t0, t1, \label192	// If AES-192, goto label192.
+.else
+	beq		\key_len, t1, \label192	// If AES-192, goto label192.
+.endif
+	// Else, it's AES-256.
+	addi		\keyp, \keyp, 16
+	vle32.v		v14, (\keyp)
+	addi		\keyp, \keyp, 16
+	vle32.v		v15, (\keyp)
+.endm
+
+// Encrypts \data using zvkned instructions, using the round keys loaded into
+// v1-v11 (for AES-128), v1-v13 (for AES-192), or v1-v15 (for AES-256).  \keylen
+// is the AES key length in bits.  vl and vtype must already be set
+// appropriately.  Note that if vl > 4, multiple blocks are encrypted.
+.macro	aes_encrypt	data, keylen
+	vaesz.vs	\data, v1
+	vaesem.vs	\data, v2
+	vaesem.vs	\data, v3
+	vaesem.vs	\data, v4
+	vaesem.vs	\data, v5
+	vaesem.vs	\data, v6
+	vaesem.vs	\data, v7
+	vaesem.vs	\data, v8
+	vaesem.vs	\data, v9
+	vaesem.vs	\data, v10
+.if \keylen == 128
+	vaesef.vs	\data, v11
+.elseif \keylen == 192
+	vaesem.vs	\data, v11
+	vaesem.vs	\data, v12
+	vaesef.vs	\data, v13
+.else
+	vaesem.vs	\data, v11
+	vaesem.vs	\data, v12
+	vaesem.vs	\data, v13
+	vaesem.vs	\data, v14
+	vaesef.vs	\data, v15
+.endif
+.endm
+
+// Same as aes_encrypt, but decrypts instead of encrypts.
+.macro	aes_decrypt	data, keylen
+.if \keylen == 128
+	vaesz.vs	\data, v11
+.elseif \keylen == 192
+	vaesz.vs	\data, v13
+	vaesdm.vs	\data, v12
+	vaesdm.vs	\data, v11
+.else
+	vaesz.vs	\data, v15
+	vaesdm.vs	\data, v14
+	vaesdm.vs	\data, v13
+	vaesdm.vs	\data, v12
+	vaesdm.vs	\data, v11
+.endif
+	vaesdm.vs	\data, v10
+	vaesdm.vs	\data, v9
+	vaesdm.vs	\data, v8
+	vaesdm.vs	\data, v7
+	vaesdm.vs	\data, v6
+	vaesdm.vs	\data, v5
+	vaesdm.vs	\data, v4
+	vaesdm.vs	\data, v3
+	vaesdm.vs	\data, v2
+	vaesdf.vs	\data, v1
+.endm
+
+// Expands to aes_encrypt or aes_decrypt according to \enc, which is 1 or 0.
+.macro	aes_crypt	data, enc, keylen
+.if \enc
+	aes_encrypt	\data, \keylen
+.else
+	aes_decrypt	\data, \keylen
+.endif
+.endm

lib/crypto/riscv/aes-riscv64-zvkned.S +1 -1

index 0d988bc3d37b..7a52ea6c669d 100644
@@ -48,7 +48,7 @@
 .text
 .option arch, +zvkned
 
-#include "../../arch/riscv/crypto/aes-macros.S"
+#include "aes-macros.S"
 
 #define RNDKEYS		a0
 #define KEY_LEN		a1

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #16 lore raw

PATCH v2 16/20 lib/crypto: riscv/aes: Pass key struct to assembly code

In reply to Eric Biggers

Make the assembly code take the AES key struct directly, rather than the round keys pointer and key length separately. Make the aes_begin macro assume this convention, and remove support for the legacy 'struct crypto_aes_ctx' from it since that isn't used here.

This aligns with the convention that is being used (and will continue to be used) for the AES modes, it makes the C glue code slightly simpler, and it avoids the unnecessary shuffling around of arguments.

Signed-off-by Eric Biggers <[email protected]>

---
 lib/crypto/riscv/aes-macros.S         | 25 ++++++++++---------------
 lib/crypto/riscv/aes-riscv64-zvkned.S | 17 ++++++++---------
 lib/crypto/riscv/aes.h                | 18 ++++++++++++------
 3 files changed, 30 insertions(+), 30 deletions(-)

lib/crypto/riscv/aes-macros.S +10 ~~-15~~

index 1384164621a5..1ab18e358474 100644
@@ -44,17 +44,20 @@
 // - RISC-V Vector ('V') with VLEN >= 128
 // - RISC-V Vector AES block cipher extension ('Zvkned')
 
-// Loads the AES round keys from \keyp into vector registers and jumps to code
-// specific to the length of the key.  Specifically:
+// Offsets in struct aes_enckey
+#define OFFSETOF_KEYLEN		0
+#define OFFSETOF_RNDKEYS	16
+
+// Loads the AES round keys from the struct aes_enckey \keyp into vector
+// registers and jumps to code specific to the length of the key.  Specifically:
 //   - If AES-128, loads round keys into v1-v11 and jumps to \label128.
 //   - If AES-192, loads round keys into v1-v13 and jumps to \label192.
 //   - If AES-256, loads round keys into v1-v15 and continues onwards.
 //
-// Also sets vl=4 and vtype=e32,m1,ta,ma.  Clobbers t0 and t1.
-.macro	aes_begin	keyp, label128, label192, key_len
-.ifb \key_len
-	lwu		t0, 480(\keyp)	// t0 = key length in bytes
-.endif
+// Also sets vl=4 and vtype=e32,m1,ta,ma.  Clobbers \keyp, t0, and t1.
+.macro	aes_begin	keyp, label128, label192
+	lwu		t0, OFFSETOF_KEYLEN(\keyp) // t0 = key length in bytes
+	addi		\keyp, \keyp, OFFSETOF_RNDKEYS
 	li		t1, 24		// t1 = key length for AES-192
 	vsetivli	zero, 4, e32, m1, ta, ma
 	vle32.v		v1, (\keyp)

@@ -78,20 +81,12 @@
 	vle32.v		v10, (\keyp)
 	addi		\keyp, \keyp, 16
 	vle32.v		v11, (\keyp)
-.ifb \key_len
 	blt		t0, t1, \label128	// If AES-128, goto label128.
-.else
-	blt		\key_len, t1, \label128	// If AES-128, goto label128.
-.endif
 	addi		\keyp, \keyp, 16
 	vle32.v		v12, (\keyp)
 	addi		\keyp, \keyp, 16
 	vle32.v		v13, (\keyp)
-.ifb \key_len
 	beq		t0, t1, \label192	// If AES-192, goto label192.
-.else
-	beq		\key_len, t1, \label192	// If AES-192, goto label192.
-.endif
 	// Else, it's AES-256.
 	addi		\keyp, \keyp, 16
 	vle32.v		v14, (\keyp)

lib/crypto/riscv/aes-riscv64-zvkned.S +8 -9

index 7a52ea6c669d..fb35f694b5ac 100644
@@ -50,10 +50,9 @@
 
 #include "aes-macros.S"
 
-#define RNDKEYS		a0
-#define KEY_LEN		a1
-#define OUTP		a2
-#define INP		a3
+#define KEYP		a0
+#define OUTP		a1
+#define INP		a2
 
 .macro	__aes_crypt_zvkned	enc, keybits
 	vle32.v		v16, (INP)

@@ -63,7 +62,7 @@
 .endm
 
 .macro	aes_crypt_zvkned	enc
-	aes_begin	RNDKEYS, 128f, 192f, KEY_LEN
+	aes_begin	KEYP, 128f, 192f
 	__aes_crypt_zvkned	\enc, 256
 128:
 	__aes_crypt_zvkned	\enc, 128

@@ -71,14 +70,14 @@
 	__aes_crypt_zvkned	\enc, 192
 .endm
 
-// void aes_encrypt_zvkned(const u32 rndkeys[], int key_len,
-//			   u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]);
+// void aes_encrypt_zvkned(const struct aes_enckey *key, u8 out[AES_BLOCK_SIZE],
+//			   const u8 in[AES_BLOCK_SIZE]);
 SYM_FUNC_START(aes_encrypt_zvkned)
 	aes_crypt_zvkned	1
 SYM_FUNC_END(aes_encrypt_zvkned)
 
-// void aes_decrypt_zvkned(const u32 rndkeys[], int key_len,
-//			   u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]);
+// void aes_decrypt_zvkned(const struct aes_key *key, u8 out[AES_BLOCK_SIZE],
+//			   const u8 in[AES_BLOCK_SIZE]);
 SYM_FUNC_START(aes_decrypt_zvkned)
 	aes_crypt_zvkned	0
 SYM_FUNC_END(aes_decrypt_zvkned)

lib/crypto/riscv/aes.h +12 -6

index 0b26f58faf2b..9de9dbd1e887 100644
@@ -10,10 +10,16 @@
 
 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_zvkned);
 
-void aes_encrypt_zvkned(const u32 rndkeys[], int key_len,
-			u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]);
-void aes_decrypt_zvkned(const u32 rndkeys[], int key_len,
-			u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]);
+/* The assembly code assumes the following offsets. */
+static_assert(offsetof(struct aes_enckey, len) == 0);
+static_assert(offsetof(struct aes_enckey, k.rndkeys) == 16);
+static_assert(offsetof(struct aes_key, len) == 0);
+static_assert(offsetof(struct aes_key, k.rndkeys) == 16);
+
+void aes_encrypt_zvkned(const struct aes_enckey *key, u8 out[AES_BLOCK_SIZE],
+			const u8 in[AES_BLOCK_SIZE]);
+void aes_decrypt_zvkned(const struct aes_key *key, u8 out[AES_BLOCK_SIZE],
+			const u8 in[AES_BLOCK_SIZE]);
 
 static void aes_preparekey_arch(union aes_enckey_arch *k,
 				union aes_invkey_arch *inv_k,

@@ -29,7 +35,7 @@ static void aes_encrypt_arch(const struct aes_enckey *key,
 {
 	if (static_branch_likely(&have_zvkned) && likely(may_use_simd())) {
 		kernel_vector_begin();
-		aes_encrypt_zvkned(key->k.rndkeys, key->len, out, in);
+		aes_encrypt_zvkned(key, out, in);
 		kernel_vector_end();
 	} else {
 		aes_encrypt_generic(key->k.rndkeys, key->nrounds, out, in);

@@ -46,7 +52,7 @@ static void aes_decrypt_arch(const struct aes_key *key,
 	 */
 	if (static_branch_likely(&have_zvkned) && likely(may_use_simd())) {
 		kernel_vector_begin();
-		aes_decrypt_zvkned(key->k.rndkeys, key->len, out, in);
+		aes_decrypt_zvkned(key, out, in);
 		kernel_vector_end();
 	} else {
 		aes_decrypt_generic(key->inv_k.inv_rndkeys, key->nrounds,

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #17 lore raw

PATCH v2 17/20 lib/crypto: riscv/aes-ecb: Migrate optimized code into library

In reply to Eric Biggers

Instead of exposing the riscv-optimized AES-ECB code via a riscv-specific crypto_skcipher algorithm, just implement the AES-ECB library functions. This is simpler, it makes the AES-ECB library functions be riscv-optimized, and it also fixes the longstanding issue where the riscv-optimized AES-ECB code was disabled by default. AES-ECB support still remains available through crypto_skcipher via crypto/aes.c, but individual architectures no longer need to handle it.

To match what the library expects, update the assembly functions to operate on struct aes_enckey or struct aes_key rather than struct crypto_aes_ctx, and adjust the argument order.

Bump up the priority of the corresponding library-based algorithm on riscv now that it no longer has to be lower than arch/riscv/crypto/.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/riscv/crypto/Kconfig              |  4 +-
 arch/riscv/crypto/aes-riscv64-glue.c   | 61 +-------------------------
 arch/riscv/crypto/aes-riscv64-zvkned.S | 39 ----------------
 crypto/aes.c                           |  3 +-
 lib/crypto/riscv/aes-riscv64-zvkned.S  | 51 +++++++++++++++++++++
 lib/crypto/riscv/aes.h                 | 33 ++++++++++++++
 6 files changed, 89 insertions(+), 102 deletions(-)

arch/riscv/crypto/Kconfig +2 -2

index 6905232ddb03..84c41824b433 100644
@@ -3,13 +3,13 @@
 menu "Accelerated Cryptographic Algorithms for CPU (riscv)"
 
 config CRYPTO_AES_RISCV64
-	tristate "Ciphers: AES, modes: ECB, CBC, CTS, CTR, XTS"
+	tristate "Ciphers: AES, modes: CBC, CTS, CTR, XTS"
 	depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \
 		   RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS
 	select CRYPTO_LIB_AES
 	select CRYPTO_SKCIPHER
 	help
-	  Length-preserving ciphers: AES with ECB, CBC, CTS, CTR, XTS
+	  Length-preserving ciphers: AES with CBC, CTS, CTR, XTS
 
 	  Architecture: riscv64 using:
 	  - Zvkned vector crypto extension

arch/riscv/crypto/aes-riscv64-glue.c +1 ~~-60~~

88 lines not on this page · raw message

arch/riscv/crypto/aes-riscv64-zvkned.S +0 ~~-39~~

index d0fc4581a380..00f8a06596d3 100644
@@ -56,45 +56,6 @@
 #define LEN		a3
 #define IVP		a4
 
-.macro	__aes_ecb_crypt	enc, keylen
-	srli		t0, LEN, 2
-	// t0 is the remaining length in 32-bit words.  It's a multiple of 4.
-1:
-	vsetvli		t1, t0, e32, m8, ta, ma
-	sub		t0, t0, t1	// Subtract number of words processed
-	slli		t1, t1, 2	// Words to bytes
-	vle32.v		v16, (INP)
-	aes_crypt	v16, \enc, \keylen
-	vse32.v		v16, (OUTP)
-	add		INP, INP, t1
-	add		OUTP, OUTP, t1
-	bnez		t0, 1b
-
-	ret
-.endm
-
-.macro	aes_ecb_crypt	enc
-	aes_begin	KEYP, 128f, 192f
-	__aes_ecb_crypt	\enc, 256
-128:
-	__aes_ecb_crypt	\enc, 128
-192:
-	__aes_ecb_crypt	\enc, 192
-.endm
-
-// void aes_ecb_encrypt_zvkned(const struct crypto_aes_ctx *key,
-//			       const u8 *in, u8 *out, size_t len);
-//
-// |len| must be nonzero and a multiple of 16 (AES_BLOCK_SIZE).
-SYM_FUNC_START(aes_ecb_encrypt_zvkned)
-	aes_ecb_crypt	1
-SYM_FUNC_END(aes_ecb_encrypt_zvkned)
-
-// Same prototype and calling convention as the encryption function
-SYM_FUNC_START(aes_ecb_decrypt_zvkned)
-	aes_ecb_crypt	0
-SYM_FUNC_END(aes_ecb_decrypt_zvkned)
-
 .macro	aes_cbc_encrypt	keylen
 	vle32.v		v16, (IVP)	// Load IV
 1:

crypto/aes.c +2 -1

index 756fce2157be..9b9e5b413fc4 100644
@@ -610,7 +610,8 @@ static struct skcipher_alg skcipher_algs[] = {
 	{
 		.base.cra_name = "ecb(aes)",
 		.base.cra_driver_name = "ecb-aes-lib",
-		.base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110,
+		.base.cra_priority = (IS_ENABLED(CONFIG_RISCV) ||
+				      IS_ENABLED(CONFIG_X86)) ? 300 : 110,
 		.base.cra_blocksize = AES_BLOCK_SIZE,
 		.base.cra_ctxsize = sizeof(struct aes_key),
 		.base.cra_module = THIS_MODULE,

lib/crypto/riscv/aes-riscv64-zvkned.S +51 -0

54 lines not on this page · raw message

lib/crypto/riscv/aes.h +33 -0

index 9de9dbd1e887..5b36d0147d93 100644
@@ -60,6 +60,39 @@ static void aes_decrypt_arch(const struct aes_key *key,
 	}
 }
 
+#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_ECB)
+void aes_ecb_encrypt_zvkned(u8 *dst, const u8 *src, size_t len,
+			    const struct aes_enckey *key);
+void aes_ecb_decrypt_zvkned(u8 *dst, const u8 *src, size_t len,
+			    const struct aes_key *key);
+
+/* len is always a positive multiple of AES_BLOCK_SIZE here. */
+#define aes_ecb_encrypt_arch aes_ecb_encrypt_arch
+static bool aes_ecb_encrypt_arch(u8 *dst, const u8 *src, size_t len,
+				 const struct aes_enckey *key)
+{
+	if (!static_branch_likely(&have_zvkned) || unlikely(!may_use_simd()))
+		return false;
+	kernel_vector_begin();
+	aes_ecb_encrypt_zvkned(dst, src, len, key);
+	kernel_vector_end();
+	return true;
+}
+
+/* len is always a positive multiple of AES_BLOCK_SIZE here. */
+#define aes_ecb_decrypt_arch aes_ecb_decrypt_arch
+static bool aes_ecb_decrypt_arch(u8 *dst, const u8 *src, size_t len,
+				 const struct aes_key *key)
+{
+	if (!static_branch_likely(&have_zvkned) || unlikely(!may_use_simd()))
+		return false;
+	kernel_vector_begin();
+	aes_ecb_decrypt_zvkned(dst, src, len, key);
+	kernel_vector_end();
+	return true;
+}
+#endif /* CONFIG_CRYPTO_LIB_AES_ECB */
+
 #define aes_mod_init_arch aes_mod_init_arch
 static void aes_mod_init_arch(void)
 {

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #18 lore raw

PATCH v2 18/20 lib/crypto: riscv/aes-cbc: Migrate optimized code into library

In reply to Eric Biggers

Instead of exposing the riscv-optimized AES-CBC and AES-CBC-CTS code via a riscv-specific crypto_skcipher algorithm, just implement the AES-CBC and AES-CBC-CTS library functions. This is simpler, it makes the AES-CBC and AES-CBC-CTS library functions be riscv-optimized, and it also fixes the longstanding issue where the riscv-optimized AES-CBC and AES-CBC-CTS code was disabled by default. AES-CBC and AES-CBC-CTS support still remains available through crypto_skcipher via crypto/aes.c, but individual architectures no longer need to handle it.

To match what the library expects, update the assembly functions to operate on struct aes_enckey or struct aes_key rather than struct crypto_aes_ctx, and adjust the argument order.

Remove the single-block special case from the assembly code, since it's already handled by lib/crypto/aes.c before calling aes_cbc_cts_*_arch().

Bump up the priority of the corresponding library-based algorithms on riscv now that they no longer have to be lower than arch/riscv/crypto/.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/riscv/crypto/Kconfig              |   4 +-
 arch/riscv/crypto/Makefile             |   2 +-
 arch/riscv/crypto/aes-riscv64-glue.c   | 167 +--------------
 arch/riscv/crypto/aes-riscv64-zvkned.S | 273 -------------------------
 crypto/aes.c                           |   6 +-
 lib/crypto/riscv/aes-riscv64-zvkned.S  | 235 +++++++++++++++++++++
 lib/crypto/riscv/aes.h                 |  68 ++++++
 7 files changed, 312 insertions(+), 443 deletions(-)
 delete mode 100644 arch/riscv/crypto/aes-riscv64-zvkned.S

arch/riscv/crypto/Kconfig +2 -2

index 84c41824b433..0a3f87ad384e 100644
@@ -3,13 +3,13 @@
 menu "Accelerated Cryptographic Algorithms for CPU (riscv)"
 
 config CRYPTO_AES_RISCV64
-	tristate "Ciphers: AES, modes: CBC, CTS, CTR, XTS"
+	tristate "Ciphers: AES, modes: CTR, XTS"
 	depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \
 		   RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS
 	select CRYPTO_LIB_AES
 	select CRYPTO_SKCIPHER
 	help
-	  Length-preserving ciphers: AES with CBC, CTS, CTR, XTS
+	  Length-preserving ciphers: AES with CTR, XTS
 
 	  Architecture: riscv64 using:
 	  - Zvkned vector crypto extension

arch/riscv/crypto/Makefile +1 -1

index 8cf31db57fc4..d8b85afa6d0b 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 obj-$(CONFIG_CRYPTO_AES_RISCV64) += aes-riscv64.o
-aes-riscv64-y := aes-riscv64-glue.o aes-riscv64-zvkned.o \
+aes-riscv64-y := aes-riscv64-glue.o \
 		 aes-riscv64-zvkned-zvbb-zvkg.o aes-riscv64-zvkned-zvkb.o
 
 obj-$(CONFIG_CRYPTO_SM4_RISCV64) += sm4-riscv64.o

arch/riscv/crypto/aes-riscv64-glue.c +2 ~~-165~~

214 lines not on this page · raw message

arch/riscv/crypto/aes-riscv64-zvkned.S deleted +0 ~~-273~~

273 lines not on this page · raw message

crypto/aes.c +4 -2

index 9b9e5b413fc4..1daaa59d0cdf 100644
@@ -626,7 +626,8 @@ static struct skcipher_alg skcipher_algs[] = {
 	{
 		.base.cra_name = "cbc(aes)",
 		.base.cra_driver_name = "cbc-aes-lib",
-		.base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110,
+		.base.cra_priority = (IS_ENABLED(CONFIG_RISCV) ||
+				      IS_ENABLED(CONFIG_X86)) ? 300 : 110,
 		.base.cra_blocksize = AES_BLOCK_SIZE,
 		.base.cra_ctxsize = sizeof(struct aes_key),
 		.base.cra_module = THIS_MODULE,

@@ -652,7 +653,8 @@ static struct skcipher_alg skcipher_algs[] = {
 	{
 		.base.cra_name = "cts(cbc(aes))",
 		.base.cra_driver_name = "cts-cbc-aes-lib",
-		.base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110,
+		.base.cra_priority = (IS_ENABLED(CONFIG_RISCV) ||
+				      IS_ENABLED(CONFIG_X86)) ? 300 : 110,
 		.base.cra_blocksize = AES_BLOCK_SIZE,
 		.base.cra_ctxsize = sizeof(struct aes_key),
 		.base.cra_module = THIS_MODULE,

lib/crypto/riscv/aes-riscv64-zvkned.S +235 -0

238 lines not on this page · raw message

lib/crypto/riscv/aes.h +68 -0

74 lines not on this page · raw message

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #19 lore raw

PATCH v2 19/20 lib/crypto: riscv/aes-ctr: Migrate optimized code into library

In reply to Eric Biggers

Instead of exposing the riscv-optimized AES-CTR code via a riscv-specific crypto_skcipher algorithm, just implement the AES-CTR library functions. This is simpler, it makes the AES-CTR library functions be riscv-optimized, and it also fixes the longstanding issue where the riscv-optimized AES-CTR code was disabled by default. AES-CTR support still remains available through crypto_skcipher via crypto/aes.c, but individual architectures no longer need to handle it.

To match what the library expects, update the assembly functions to operate on struct aes_enckey rather than struct crypto_aes_ctx, and adjust the argument order.

Bump up the priority of the corresponding library-based algorithm on riscv now that it no longer has to be lower than arch/riscv/crypto/. Also re-enable the library-based "ccm(aes)" and "gcm(aes)".

Signed-off-by Eric Biggers <[email protected]>

---
 arch/riscv/crypto/Kconfig                     |   5 +-
 arch/riscv/crypto/Makefile                    |   2 +-
 arch/riscv/crypto/aes-riscv64-glue.c          | 114 +-----------------
 crypto/aes.c                                  |   5 +-
 lib/crypto/Makefile                           |   4 +
 .../crypto/riscv}/aes-riscv64-zvkned-zvkb.S   |  23 ++--
 lib/crypto/riscv/aes.h                        |  72 ++++++++++-
 7 files changed, 94 insertions(+), 131 deletions(-)
 rename {arch/riscv/crypto => lib/crypto/riscv}/aes-riscv64-zvkned-zvkb.S (93%)

arch/riscv/crypto/Kconfig +2 -3

index 0a3f87ad384e..0733d4894401 100644
@@ -3,18 +3,17 @@
 menu "Accelerated Cryptographic Algorithms for CPU (riscv)"
 
 config CRYPTO_AES_RISCV64
-	tristate "Ciphers: AES, modes: CTR, XTS"
+	tristate "Ciphers: AES, modes: XTS"
 	depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \
 		   RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS
 	select CRYPTO_LIB_AES
 	select CRYPTO_SKCIPHER
 	help
-	  Length-preserving ciphers: AES with CTR, XTS
+	  Length-preserving ciphers: AES with XTS
 
 	  Architecture: riscv64 using:
 	  - Zvkned vector crypto extension
 	  - Zvbb vector extension (XTS)
-	  - Zvkb vector crypto extension (CTR)
 	  - Zvkg vector crypto extension (XTS)
 
 config CRYPTO_SM4_RISCV64

arch/riscv/crypto/Makefile +1 -1

index d8b85afa6d0b..08904603fc94 100644
@@ -2,7 +2,7 @@
 
 obj-$(CONFIG_CRYPTO_AES_RISCV64) += aes-riscv64.o
 aes-riscv64-y := aes-riscv64-glue.o \
-		 aes-riscv64-zvkned-zvbb-zvkg.o aes-riscv64-zvkned-zvkb.o
+		 aes-riscv64-zvkned-zvbb-zvkg.o
 
 obj-$(CONFIG_CRYPTO_SM4_RISCV64) += sm4-riscv64.o
 sm4-riscv64-y := sm4-riscv64-glue.o sm4-riscv64-zvksed-zvkb.o

arch/riscv/crypto/aes-riscv64-glue.c +3 ~~-111~~

156 lines not on this page · raw message

crypto/aes.c +2 -3

index 1daaa59d0cdf..1b84953c5deb 100644
@@ -670,7 +670,8 @@ static struct skcipher_alg skcipher_algs[] = {
 	{
 		.base.cra_name = "ctr(aes)",
 		.base.cra_driver_name = "ctr-aes-lib",
-		.base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110,
+		.base.cra_priority = (IS_ENABLED(CONFIG_RISCV) ||
+				      IS_ENABLED(CONFIG_X86)) ? 300 : 110,
 		.base.cra_blocksize = 1,
 		.base.cra_ctxsize = sizeof(struct aes_enckey),
 		.base.cra_module = THIS_MODULE,

@@ -1002,7 +1003,6 @@ static struct aead_alg aead_algs[] = {
 	!(IS_ENABLED(CONFIG_ARM) || \
 	  IS_ENABLED(CONFIG_ARM64) || \
 	  IS_ENABLED(CONFIG_PPC) || \
-	  IS_ENABLED(CONFIG_RISCV) || \
 	  IS_ENABLED(CONFIG_S390) || \
 	  IS_ENABLED(CONFIG_SPARC))
 	{

@@ -1045,7 +1045,6 @@ static struct aead_alg aead_algs[] = {
 	!(IS_ENABLED(CONFIG_ARM) || \
 	  IS_ENABLED(CONFIG_ARM64) || \
 	  IS_ENABLED(CONFIG_PPC) || \
-	  IS_ENABLED(CONFIG_RISCV) || \
 	  IS_ENABLED(CONFIG_S390) || \
 	  IS_ENABLED(CONFIG_SPARC))
 	{

lib/crypto/Makefile +4 -0

index 02d89a226377..ff34aeda37ba 100644
@@ -51,6 +51,10 @@ endif # !CONFIG_SPE
 endif # CONFIG_PPC
 
 libaes-$(CONFIG_RISCV) += riscv/aes-riscv64-zvkned.o
+ifneq ($(CONFIG_CRYPTO_LIB_AES_CTR),)
+libaes-$(CONFIG_RISCV) += riscv/aes-riscv64-zvkned-zvkb.o
+endif
+
 libaes-$(CONFIG_SPARC) += sparc/aes_asm.o
 
 libaes-$(CONFIG_X86) += x86/aes-aesni.o

lib/crypto/riscv/aes-riscv64-zvkned-zvkb.S renamed from arch/riscv/crypto/aes-riscv64-zvkned-zvkb.S +11 ~~-12~~

similarity index 93%
rename from arch/riscv/crypto/aes-riscv64-zvkned-zvkb.S
rename to lib/crypto/riscv/aes-riscv64-zvkned-zvkb.S
index 9962d4500587..2981d8182de6 100644
@@ -49,11 +49,11 @@
 
 #include "aes-macros.S"
 
-#define KEYP		a0
-#define INP		a1
-#define OUTP		a2
-#define LEN		a3
-#define IVP		a4
+#define DST		a0
+#define SRC		a1
+#define LEN		a2
+#define IVP		a3
+#define KEYP		a4
 
 #define LEN32		a5
 #define VL_E32		a6

@@ -110,13 +110,13 @@
 
 	// XOR the data with the keystream.
 	vsetvli		t0, LEN, e8, m4, ta, ma
-	vle8.v		v20, (INP)
+	vle8.v		v20, (SRC)
 	vxor.vv		v20, v20, v24
-	vse8.v		v20, (OUTP)
+	vse8.v		v20, (DST)
 
 	// Advance the pointers and update the remaining length.
-	add		INP, INP, t0
-	add		OUTP, OUTP, t0
+	add		SRC, SRC, t0
+	add		DST, DST, t0
 	sub		LEN, LEN, t0
 	sub		LEN32, LEN32, VL_E32
 	srli		VL_BLOCKS, VL_E32, 2

@@ -133,9 +133,8 @@
 	ret
 .endm
 
-// void aes_ctr32_crypt_zvkned_zvkb(const struct crypto_aes_ctx *key,
-//				    const u8 *in, u8 *out, size_t len,
-//				    u8 iv[16]);
+// void aes_ctr32_crypt_zvkned_zvkb(u8 *dst, const u8 *src, size_t len,
+//				    u8 iv[16], const struct aes_enckey *key);
 SYM_FUNC_START(aes_ctr32_crypt_zvkned_zvkb)
 	aes_begin	KEYP, 128f, 192f
 	aes_ctr32_crypt	256

lib/crypto/riscv/aes.h +71 -1

87 lines not on this page · raw message

Signature · 1 line

2.55.0

Eric Biggers

author maintainer

2026-09-27 22:44 UTC #20 lore raw

PATCH v2 20/20 lib/crypto: riscv/aes-xts: Migrate optimized code into library

In reply to Eric Biggers

Instead of exposing the riscv-optimized AES-XTS code via a riscv-specific crypto_skcipher algorithm, just implement the AES-XTS library functions. This is simpler, it makes the AES-XTS library functions be riscv-optimized, and it also fixes the longstanding issue where the riscv-optimized AES-XTS code was disabled by default. AES-XTS support still remains available through crypto_skcipher via crypto/aes.c, but individual architectures no longer need to handle it.

To match what the library expects, update the assembly functions to operate on struct aes_key rather than struct crypto_aes_ctx, adjust the argument order, and remove the redundant ciphertext stealing support which is already implemented in a generic way in the library.

Bump up the priority of the corresponding library-based algorithm on riscv now that it no longer has to be lower than arch/riscv/crypto/.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/riscv/crypto/Kconfig                     |  14 --
 arch/riscv/crypto/Makefile                    |   4 -
 arch/riscv/crypto/aes-macros.S                | 166 ------------
 arch/riscv/crypto/aes-riscv64-glue.c          | 236 ------------------
 crypto/aes.c                                  |   3 +-
 lib/crypto/Makefile                           |   3 +
 .../riscv}/aes-riscv64-zvkned-zvbb-zvkg.S     |  99 ++------
 lib/crypto/riscv/aes.h                        |  53 ++++
 8 files changed, 74 insertions(+), 504 deletions(-)
 delete mode 100644 arch/riscv/crypto/aes-macros.S
 delete mode 100644 arch/riscv/crypto/aes-riscv64-glue.c
 rename {arch/riscv/crypto => lib/crypto/riscv}/aes-riscv64-zvkned-zvbb-zvkg.S (74%)

arch/riscv/crypto/Kconfig +0 ~~-14~~

index 0733d4894401..614f93214862 100644
@@ -2,20 +2,6 @@
 
 menu "Accelerated Cryptographic Algorithms for CPU (riscv)"
 
-config CRYPTO_AES_RISCV64
-	tristate "Ciphers: AES, modes: XTS"
-	depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \
-		   RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS
-	select CRYPTO_LIB_AES
-	select CRYPTO_SKCIPHER
-	help
-	  Length-preserving ciphers: AES with XTS
-
-	  Architecture: riscv64 using:
-	  - Zvkned vector crypto extension
-	  - Zvbb vector extension (XTS)
-	  - Zvkg vector crypto extension (XTS)
-
 config CRYPTO_SM4_RISCV64
 	tristate "Ciphers: SM4 (ShangMi 4)"
 	depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \

arch/riscv/crypto/Makefile +0 -4

index 08904603fc94..9f6956cf50b1 100644
@@ -1,8 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
-obj-$(CONFIG_CRYPTO_AES_RISCV64) += aes-riscv64.o
-aes-riscv64-y := aes-riscv64-glue.o \
-		 aes-riscv64-zvkned-zvbb-zvkg.o
-
 obj-$(CONFIG_CRYPTO_SM4_RISCV64) += sm4-riscv64.o
 sm4-riscv64-y := sm4-riscv64-glue.o sm4-riscv64-zvksed-zvkb.o

arch/riscv/crypto/aes-macros.S deleted +0 ~~-166~~

166 lines not on this page · raw message

arch/riscv/crypto/aes-riscv64-glue.c deleted +0 ~~-236~~

236 lines not on this page · raw message

crypto/aes.c +2 -1

index 1b84953c5deb..2e61146f6fbb 100644
@@ -705,7 +705,8 @@ static struct skcipher_alg skcipher_algs[] = {
 	{
 		.base.cra_name = "xts(aes)",
 		.base.cra_driver_name = "xts-aes-lib",
-		.base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110,
+		.base.cra_priority = (IS_ENABLED(CONFIG_RISCV) ||
+				      IS_ENABLED(CONFIG_X86)) ? 300 : 110,
 		.base.cra_blocksize = AES_BLOCK_SIZE,
 		.base.cra_ctxsize = sizeof(struct aes_xts_key),
 		.base.cra_module = THIS_MODULE,

lib/crypto/Makefile +3 -0

index ff34aeda37ba..d683b8520f55 100644
@@ -54,6 +54,9 @@ libaes-$(CONFIG_RISCV) += riscv/aes-riscv64-zvkned.o
 ifneq ($(CONFIG_CRYPTO_LIB_AES_CTR),)
 libaes-$(CONFIG_RISCV) += riscv/aes-riscv64-zvkned-zvkb.o
 endif
+ifneq ($(CONFIG_CRYPTO_LIB_AES_XTS),)
+libaes-$(CONFIG_RISCV) += riscv/aes-riscv64-zvkned-zvbb-zvkg.o
+endif
 
 libaes-$(CONFIG_SPARC) += sparc/aes_asm.o

lib/crypto/riscv/aes-riscv64-zvkned-zvbb-zvkg.S renamed from arch/riscv/crypto/aes-riscv64-zvkned-zvbb-zvkg.S +16 ~~-83~~

151 lines not on this page · raw message

lib/crypto/riscv/aes.h +53 -0

75 lines not on this page · raw message

Signature · 1 line

2.55.0

Ard Biesheuvel

maintainer Reviewed-by on the v2 series

2026-09-30 13:16 UTC #21 lore raw

In reply to Eric Biggers

Quoted 68 lines · Eric Biggers

This series applies to v7.3-rc4. It can also be retrieved from:
git fetch
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git
aes-lib-x86-riscv-v2
Patch 1 was already applied to libcrypto-fixes, but is resent to make this series applicable directly to v7.3-rc4 (ensuring that the Sashiko review runs). Patches 2-20 are targeting libcrypto-next for 7.4.
This series migrates the x86 and RISC-V accelerated implementations of the AES modes ECB, CBC, CBC-CTS, CTR, XCTR, and XTS into lib/crypto/.
This makes the corresponding library APIs be properly accelerated on these architectures, while still accelerating crypto_skcipher as well (via the library-based code in crypto/aes.c).
It removes a lot of redundant glue code, since crypto API boilerplate no longer needs to be duplicated per-architecture.
Finally, it fixes the longstanding issue where these optimizations were disabled by default.
In the case of RISC-V, this series handles all remaining AES code in arch/riscv/crypto/. In the case of x86, AES-GCM is still left in arch/x86/crypto/ for now; it will be handled later. Other architectures will be handled later as well.
For various reasons, aesni-intel_asm.S (the x86-accelerated ECB, CBC, CBC-CTS, CTR, and XTS code that doesn't use AVX or VAES) is replaced with new functions written from scratch. The other assembly functions are kept but are modified slightly for integration into the library.
Changed in v2:
- Fixed 32-bit x86 bisection hazards by making aesni-intel depend on CONFIG_64BIT earlier in the series.
- Fixed handling of lengths over S32_MAX in RISC-V CTR code.
- Restored selection of CRYPTO_SKCIPHER by CRYPTO_AES_NI_INTEL, since it is still needed.
- Made the 32-bit x86 XTS code spill the tweaks to the stack rather than to the destination buffer.
- Removed the no-longer-needed single block special case from the RISC-V CBC-CTS assembly code.
- Made x86 CBC-CTS encryption and decryption share more code.
- Reordered the function parameters in aes-xts-avx-x86_64.S and
     aes-ctr-avx-x86_64.S.
- Other miscellaneous cleanups.
Eric Biggers (20):
crypto: aes - Fix undesired override of some optimized AES modes lib/crypto: aes-xctr: Pass counter by value to aes_xctr_arch() lib/crypto: x86/aes: Clean up aes-aesni.S in preparation for AES modes lib/crypto: x86/aes-ecb: Add AES-NI optimization lib/crypto: x86/aes-cbc: Add AES-NI optimization lib/crypto: x86/aes-ctr: Add AES-NI optimization lib/crypto: x86/aes-xts: Add AES-NI optimization
crypto: x86/aes - Drop superseded 32-bit build support
crypto: x86/aes-ecb - Remove superseded ECB skcipher
crypto: x86/aes-cbc - Remove superseded CBC skciphers
crypto: x86/aes-ctr - Remove superseded CTR skcipher
crypto: x86/aes-xts - Remove superseded XTS skcipher

Quoted 9 lines

lib/crypto: x86/aes-ctr: Migrate AVX-optimized code into library lib/crypto: x86/aes-xts: Migrate AVX-optimized code into library lib/crypto: riscv/aes: Copy aes-macros.S to library lib/crypto: riscv/aes: Pass key struct to assembly code lib/crypto: riscv/aes-ecb: Migrate optimized code into library lib/crypto: riscv/aes-cbc: Migrate optimized code into library lib/crypto: riscv/aes-ctr: Migrate optimized code into library lib/crypto: riscv/aes-xts: Migrate optimized code into library

For the series,

Reviewed-by Ard Biesheuvel <[email protected]>

Eric Biggers

author maintainer Applied

2026-09-30 16:56 UTC #22 lore raw

In reply to Eric Biggers

Quoted 26 lines · Eric Biggers

This series applies to v7.3-rc4. It can also be retrieved from:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git aes-lib-x86-riscv-v2
Patch 1 was already applied to libcrypto-fixes, but is resent to make this series applicable directly to v7.3-rc4 (ensuring that the Sashiko review runs). Patches 2-20 are targeting libcrypto-next for 7.4.
This series migrates the x86 and RISC-V accelerated implementations of the AES modes ECB, CBC, CBC-CTS, CTR, XCTR, and XTS into lib/crypto/.
This makes the corresponding library APIs be properly accelerated on these architectures, while still accelerating crypto_skcipher as well (via the library-based code in crypto/aes.c).
It removes a lot of redundant glue code, since crypto API boilerplate no longer needs to be duplicated per-architecture.
Finally, it fixes the longstanding issue where these optimizations were disabled by default.
In the case of RISC-V, this series handles all remaining AES code in arch/riscv/crypto/. In the case of x86, AES-GCM is still left in arch/x86/crypto/ for now; it will be handled later. Other architectures will be handled later as well.

Quoted 5 lines

For various reasons, aesni-intel_asm.S (the x86-accelerated ECB, CBC, CBC-CTS, CTR, and XTS code that doesn't use AVX or VAES) is replaced with new functions written from scratch. The other assembly functions are kept but are modified slightly for integration into the library.

Applied to https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=libcrypto-next

- Eric