Skip to contentSkip to conversation

x86: Support Key Locker

PATCH In review Chang S. Bae 14 participants 154 messages Jan 12, 2022 – Jun 11, 2024 Page 2 of 2

Subsystems (7)
Documentation x86 Crypto API KVM x86 +3 more subsystems
lore
Share & tools
lore
#142 lore raw

PATCH v9 12/14 x86/Kconfig: Add symbols for Key Locker

In reply to Chang S. Bae

Add CONFIG_X86_KEYLOCKER to control whether Key Locker is initialized at boot. Additionally, add the AS_KEYLOCKER config symbol to indicate whether the assembler supports Key Locker.

The former will be selected, and the latter will be referenced by the Key Locker cipher module CRYPTO_AES_KL, to be added in a later patch.

---
Changes from v9:
* Include AS_KEYLOCKER symbol (Eric Biggers).
* Revoke the earlier tag.

---
 arch/x86/Kconfig           | 3 +++
 arch/x86/Kconfig.assembler | 5 +++++
 2 files changed, 8 insertions(+)
arch/x86/Kconfig +3 -0
index 1d7122a1883e..ce4e4c1641da 100644
@@ -1881,6 +1881,9 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS
 
 	  If unsure, say y.
 
+config X86_KEYLOCKER
+	bool
+
 choice
 	prompt "TSX enable mode"
 	depends on CPU_SUP_INTEL
arch/x86/Kconfig.assembler +5 -0
index 59aedf32c4ea..e6ce80d23113 100644
@@ -35,6 +35,11 @@ config AS_VPCLMULQDQ
 	help
 	  Supported by binutils >= 2.30 and LLVM integrated assembler
 
+config AS_KEYLOCKER
+	def_bool $(as-instr,encodekey256 %eax$(comma)%eax)
+	help
+	  Supported by binutils >= 2.36 and LLVM integrated assembler >= V12
+
 config AS_WRUSS
 	def_bool $(as-instr,wrussq %rax$(comma)(%rbx))
 	help
Signature · 1 line
2.34.1
#143 lore raw

PATCH v9 13/14 crypto: x86/aes - Prepare for new AES-XTS implementation

In reply to Chang S. Bae

The key Locker's AES instruction set ('AES-KL') shares a similar programming interface with AES-NI. The internal ABI in the assembly code will have the same prototype as AES-NI, and the glue code will also be identical.

The upcoming AES code will exclusively support the XTS mode as disk encryption is the only intended use case.

Refactor the XTS-related code to eliminate code duplication and relocate certain constant values to make them shareable. Also, introduce wrappers for data transformation functions to return an error code, as AES-KL may populate it.

Introduce union x86_aes_ctx as AES-KL will reference an encoded form instead of an expanded AES key. This allows different AES context formats in the shared code.

Inline the refactored code to the caller to prevent the potential overhead of indirect calls.

No functional changes or performance regressions are intended.

---
Changes from v8:
* Rebase on the AES-NI changes in the mainline -- mostly cleanup works.
* Introduce 'union x86_aes_ctx'. (Eric Biggers)
* Ensure 'inline' for wrapper functions.
* Tweak the changelog.

Changes from v7:
* Remove aesni_dec() as not referenced by the refactored helpers. But, keep the ASM symbol '__aesni_dec' to make it appears consistent with its counterpart '__aesni_enc'.
* Call out 'AES-XTS' in the subject.

Changes from v6:
* Inline the helper code to avoid the indirect call. (Eric Biggers)
* Rename the filename: aes-intel* -> aes-helper*. (Eric Biggers)
* Don't export symbols yet here. Instead, do it when needed later.
* Improve the coding style:
- Follow the symbol convention: '_' -> '__' (Eric Biggers)
- Fix a style issue -- 'dst = src = ...' catched by checkpatch.pl:
"CHECK: multiple assignments should be avoided"
* Cleanup: move some define back to AES-NI code as not used by AES-KL

Changes from v5:
* Clean up the staled function definition -- cbc_crypt_common().
* Ensure kernel_fpu_end() for the possible error return from xts_crypt_common()->crypt1_fn().

Changes from v4:
* Drop CBC mode changes. (Eric Biggers)

Changes from v3:
* Drop ECB and CTR mode changes. (Eric Biggers)
* Export symbols. (Eric Biggers)

Changes from RFC v2:
* Massage the changelog. (Dan Williams)

Changes from RFC v1:
* Added as a new patch. (Ard Biesheuvel)

---
 arch/x86/crypto/aes-helper_asm.S   |  22 +++
 arch/x86/crypto/aes-helper_glue.h  | 167 ++++++++++++++++++++++
 arch/x86/crypto/aesni-intel_asm.S  |  47 +++----
 arch/x86/crypto/aesni-intel_glue.c | 213 ++++++++---------------------
 4 files changed, 261 insertions(+), 188 deletions(-)
 create mode 100644 arch/x86/crypto/aes-helper_asm.S
 create mode 100644 arch/x86/crypto/aes-helper_glue.h
arch/x86/crypto/aes-helper_asm.S new file +22 -0
new file mode 100644
index 000000000000..b31abcdf63cb
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+/*
+ * Constant values shared between AES implementations:
+ */
+
+.pushsection .rodata
+.align 16
+.Lcts_permute_table:
+	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+	.byte		0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07
+	.byte		0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f
+	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+.popsection
+
+.section	.rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16
+.align 16
+.Lgf128mul_x_ble_mask:
+	.octa 0x00000000000000010000000000000087
+.previous
arch/x86/crypto/aes-helper_glue.h new file +167 -0

167 lines not on this page · raw message

arch/x86/crypto/aesni-intel_asm.S +17 -30

116 lines not on this page · raw message

arch/x86/crypto/aesni-intel_glue.c +55 -158

321 lines not on this page · raw message

Signature · 1 line
2.34.1
#144 lore raw

Re: [PATCH v9 13/14] crypto: x86/aes - Prepare for new AES-XTS implementation

This patch is no longer needed, as the rework leads more destructive code changes in the AES-NI side.

Thanks,
Chang

#145 lore raw

PATCH v9 14/14 crypto: x86/aes-kl - Implement the AES-XTS algorithm

In reply to Chang S. Bae

Key Locker is a CPU feature to reduce key exfiltration opportunities. It converts the AES key into an encoded form, called 'key handle', to reduce the exposure of private key material in memory.

This key conversion as well as all subsequent data transformations are provided by new AES instructions ('AES-KL'). AES-KL is analogous to that of AES-NI as maintains a similar programming interface.

Support the XTS mode as the primary use case is dm-crypt. The support has some details worth mentioning, which differentiate itself from AES-NI, that users may need to be aware of:

== Key Handle Restriction ==

The AES-KL instruction set supports selecting key usage restrictions at key handle creation time. Restrict all key handles created by the kernel to kernel mode use only.

The AES-KL instructions themselves are executable in userspace. This restriction enforces the mode consistency in its operation.

If the key handle is created in userspace but referenced in the kernel, then encrypt() and decrypt() functions will return -EINVAL.

=== AES-NI Dependency for AES Compliance ===

Key Locker is not AES compliant as it lacks 192-bit key support. However, per the expectations of Linux crypto-cipher implementations, the software cipher implementation must support all the AES-compliant key sizes.

The AES-KL cipher implementation achieves this constraint by logging a warning and falling back to AES-NI. In other words, the 192-bit key-size limitation for what can be converted into a key handle is only documented, not enforced.

This then creates a rather strong dependency on AES-NI. If this driver supports a module build, the exported AES-NI functions cannot be inlined. More importantly, indirect calls will impact the performance.

To simplify, disallow a module build for AES-KL and always select AES-NI. This restriction can be relaxed if strong use cases arise against it.

== Wrapping Key Restore Failure Handling ==

In the event of hardware failure, the wrapping key is lost from deep sleep states. Then, the wrapping key turns to zero which is an unusable state.

The x86 core provides valid_keylocker() to indicate the failure. Subsequent setkey() as well as encode()/decode() can check it and return -ENODEV if failed. In this way, an error code can be returned, instead of facing abrupt exceptions.

== Userspace Exposition ==

The Keylocker implementations so far have measurable performance penalties. So, keep AES-NI as the default.

However, with a slow storage device, storage bandwidth is the bottleneck, even if disk encryption is enabled by AES-KL. Thus, it is an end-user consideration for selecting AES-KL. Users may pick it according to the name 'xts-aes-aeskl' shown in /proc/crypto.

== 64-bit Only ==

Support 64-bit only, as the 32-bit kernel is being deprecated.

---
Changes from v8:
* Rebase on the upstream changes.
* Combine the XTS enc/dec assembly code in a macro. (Eric Biggers)
* Define setkey() as void instead of returning 'int'. (Eric Biggers)
* Rearrange the assembly code to reduce jumps especially for success cases. (Eric Biggers)
* Update the changelog for clarification. (Eric Biggers)
* Exclude module build.

Changes from v7:
* Update the changelog -- remove 'API Limitation'. (Eric Biggers)
* Update the comment for valid_keylocker(). (Eric Biggers)
* Improve the code:
- Remove the key-length check and simplify the code. (Eric Biggers)
- Remove aeskl_dec() and __aeskl_dec() as not needed.
- Simplify the register-function return handling. (Eric Biggers)
- Rename setkey functions for coherent naming:

    aeskl_setkey() -> __aeskl_setkey(),
    aeskl_setkey_common() -> aeskl_setkey(),
    aeskl_xts_setkey() -> xts_setkey()

- Revert an unnecessary comment.

Changes from v6:
* Merge all the AES-KL patches. (Eric Biggers)
* Make the driver for the 64-bit mode only. (Eric Biggers)
* Rework the key-size check code:
- Trim unnecessary checks. (Eric Biggers)
- Document the reason
- Make sure both XTS keys with the same size
* Adjust the Kconfig change:
- Move the location. (Robert Elliott)
- Trim the description to follow others such as AES-NI.
* Update the changelog:
- Explain the priority value for the common name under 'User Exposition' (renamed from 'Performance'). (Eric Biggers)
- Trim the introduction
- Switch to more imperative mood for those explaining the code change
- Add a new section '64-bit Only'
* Adjust the ASM code to return a proper error code. (Eric Biggers)
* Update assembly code macros:
- Remove unused one.
- Document the reason for the duplicated ones.

Changes from v5:
* Replace the ret instruction with RET as rebased on the upstream -- commit f94909ceb1ed ("x86: Prepare asm files for straight-line-speculation").

Changes from v3:
* Exclude non-AES-KL objects. (Eric Biggers)
* Simplify the assembler dependency check. (Peter Zijlstra)
* Trim the Kconfig help text. (Dan Williams)
* Fix a defined-but-not-used warning.

Changes from RFC v2:
* Move out each mode support in new patches.
* Update the changelog to describe the limitation and the tradeoff clearly. (Andy Lutomirski)

Changes from RFC v1:
* Rebased on the refactored code. (Ard Biesheuvel)
* Dropped exporting the single block interface. (Ard Biesheuvel)
* Fixed the fallback and error handling paths. (Ard Biesheuvel)
* Revised the module description. (Dave Hansen and Peter Zijlstra)
* Made the build depend on the binutils version to support new instructions. (Borislav Petkov and Peter Zijlstra)
* Updated the changelog accordingly.

---
 arch/x86/Kconfig.assembler         |   5 +
 arch/x86/crypto/Kconfig            |  17 ++
 arch/x86/crypto/Makefile           |   3 +
 arch/x86/crypto/aes-helper_glue.h  |   7 +-
 arch/x86/crypto/aeskl-intel_asm.S  | 412 +++++++++++++++++++++++++++++
 arch/x86/crypto/aeskl-intel_glue.c | 187 +++++++++++++
 arch/x86/crypto/aeskl-intel_glue.h |  35 +++
 arch/x86/crypto/aesni-intel_glue.c |  30 +--
 arch/x86/crypto/aesni-intel_glue.h |  40 +++
 9 files changed, 704 insertions(+), 32 deletions(-)
 create mode 100644 arch/x86/crypto/aeskl-intel_asm.S
 create mode 100644 arch/x86/crypto/aeskl-intel_glue.c
 create mode 100644 arch/x86/crypto/aeskl-intel_glue.h
 create mode 100644 arch/x86/crypto/aesni-intel_glue.h
arch/x86/Kconfig.assembler +5 -0
index 8ad41da301e5..0e58f2b61dd3 100644
@@ -25,6 +25,11 @@ config AS_GFNI
 	help
 	  Supported by binutils >= 2.30 and LLVM integrated assembler
 
+config AS_HAS_KEYLOCKER
+	def_bool $(as-instr,encodekey256 %eax$(comma)%eax)
+	help
+	  Supported by binutils >= 2.36 and LLVM integrated assembler >= V12
+
 config AS_WRUSS
 	def_bool $(as-instr,wrussq %rax$(comma)(%rbx))
 	help
arch/x86/crypto/Kconfig +17 -0
index c9e59589a1ce..067bb149998b 100644
@@ -29,6 +29,23 @@ config CRYPTO_AES_NI_INTEL
 	  Architecture: x86 (32-bit and 64-bit) using:
 	  - AES-NI (AES new instructions)
 
+config CRYPTO_AES_KL
+	bool "Ciphers: AES, modes: XTS (AES-KL)"
+	depends on X86 && 64BIT
+	depends on AS_HAS_KEYLOCKER
+	select CRYPTO_AES_NI_INTEL
+	select X86_KEYLOCKER
+
+	help
+	  Block cipher: AES cipher algorithms
+	  Length-preserving ciphers: AES with XTS
+
+	  Architecture: x86 (64-bit) using:
+	  - AES-KL (AES Key Locker)
+	  - AES-NI for a 192-bit key
+
+	  See Documentation/arch/x86/keylocker.rst for more details.
+
 config CRYPTO_BLOWFISH_X86_64
 	tristate "Ciphers: Blowfish, modes: ECB, CBC"
 	depends on X86 && 64BIT
arch/x86/crypto/Makefile +3 -0
index 9aa46093c91b..ae2aa7abd151 100644
@@ -50,6 +50,9 @@ obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
 aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
 
+obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-intel.o
+aeskl-intel-y := aeskl-intel_asm.o aeskl-intel_glue.o
+
 obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
 sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
 sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
arch/x86/crypto/aes-helper_glue.h +4 -3
index 52ba1fe5cf71..262c1cec0011 100644
@@ -19,16 +19,17 @@
 #include <crypto/internal/aead.h>
 #include <crypto/internal/simd.h>
 
+#include "aeskl-intel_glue.h"
+
 #define AES_ALIGN		16
 #define AES_ALIGN_ATTR		__attribute__((__aligned__(AES_ALIGN)))
 #define AES_ALIGN_EXTRA		((AES_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1))
 #define XTS_AES_CTX_SIZE	(sizeof(struct aes_xts_ctx) + AES_ALIGN_EXTRA)
 
-/*
- * Preserve data types for various AES implementations available in x86
- */
+/* Data types for the two AES implementations available in x86 */
 union x86_aes_ctx {
 	struct crypto_aes_ctx aesni;
+	struct aeskl_ctx aeskl;
 };
 
 struct aes_xts_ctx {
arch/x86/crypto/aeskl-intel_asm.S new file +412 -0

412 lines not on this page · raw message

arch/x86/crypto/aeskl-intel_glue.c new file +187 -0

187 lines not on this page · raw message

arch/x86/crypto/aeskl-intel_glue.h new file +35 -0
new file mode 100644
index 000000000000..57cfd6c55a4f
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _AESKL_INTEL_GLUE_H
+#define _AESKL_INTEL_GLUE_H
+
+#include <crypto/aes.h>
+#include <linux/types.h>
+
+#define AESKL_AAD_SIZE		16
+#define AESKL_TAG_SIZE		16
+#define AESKL_CIPHERTEXT_MAX	AES_KEYSIZE_256
+
+/* The Key Locker handle is an encoded form of the AES key. */
+struct aeskl_handle {
+	u8 additional_authdata[AESKL_AAD_SIZE];
+	u8 integrity_tag[AESKL_TAG_SIZE];
+	u8 ciphre_text[AESKL_CIPHERTEXT_MAX];
+};
+
+/*
+ * Key Locker does not support 192-bit key size. The driver needs to
+ * retrieve the key size in the first place. The offset of the
+ * 'key_length' field here should be compatible with struct
+ * crypto_aes_ctx.
+ */
+#define AESKL_CTX_RESERVED (sizeof(struct crypto_aes_ctx) - sizeof(struct aeskl_handle) \
+			    - sizeof(u32))
+
+struct aeskl_ctx {
+	struct aeskl_handle handle;
+	u8 reserved[AESKL_CTX_RESERVED];
+	u32 key_length;
+};
+
+#endif /* _AESKL_INTEL_GLUE_H */
arch/x86/crypto/aesni-intel_glue.c +1 -29
index 4ac7b9a28967..d9c4aa055383 100644
@@ -37,6 +37,7 @@
 #include <linux/static_call.h>
 
 #include "aes-helper_glue.h"
+#include "aesni-intel_glue.h"
 
 #define RFC4106_HASH_SUBKEY_SIZE 16
 #define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1))
@@ -72,9 +73,6 @@ struct gcm_context_data {
 	u8 hash_keys[GCM_BLOCK_LEN * 16];
 };
 
-asmlinkage void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
-			      unsigned int key_len);
-asmlinkage void __aesni_enc(const void *ctx, u8 *out, const u8 *in);
 asmlinkage void __aesni_dec(const void *ctx, u8 *out, const u8 *in);
 asmlinkage void aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out,
 			      const u8 *in, unsigned int len);
@@ -89,21 +87,9 @@ asmlinkage void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out,
 asmlinkage void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out,
 				  const u8 *in, unsigned int len, u8 *iv);
 
-static inline int aesni_enc(const void *ctx, u8 *out, const u8 *in)
-{
-	__aesni_enc(ctx, out, in);
-	return 0;
-}
-
 #define AVX_GEN2_OPTSIZE 640
 #define AVX_GEN4_OPTSIZE 4096
 
-asmlinkage void __aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out,
-				    const u8 *in, unsigned int len, u8 *iv);
-
-asmlinkage void __aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out,
-				    const u8 *in, unsigned int len, u8 *iv);
-
 #ifdef CONFIG_X86_64
 
 asmlinkage void aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out,
@@ -271,20 +257,6 @@ static inline int aesni_xts_setkey(union x86_aes_ctx *ctx,
 	return aes_set_key_common(&ctx->aesni, in_key, key_len);
 }
 
-static inline int aesni_xts_encrypt(const union x86_aes_ctx *ctx, u8 *out, const u8 *in,
-				    unsigned int len, u8 *iv)
-{
-	__aesni_xts_encrypt(&ctx->aesni, out, in, len, iv);
-	return 0;
-}
-
-static inline int aesni_xts_decrypt(const union x86_aes_ctx *ctx, u8 *out, const u8 *in,
-				    unsigned int len, u8 *iv)
-{
-	__aesni_xts_decrypt(&ctx->aesni, out, in, len, iv);
-	return 0;
-}
-
 static int aesni_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
 			         unsigned int len)
 {
arch/x86/crypto/aesni-intel_glue.h new file +40 -0
new file mode 100644
index 000000000000..999f81f5bcde
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * These are AES-NI functions that are used by the AES-KL code as a
+ * fallback when it is given a 192-bit key. Key Locker does not support
+ * 192-bit keys.
+ */
+
+#ifndef _AESNI_INTEL_GLUE_H
+#define _AESNI_INTEL_GLUE_H
+
+asmlinkage void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
+			      unsigned int key_len);
+asmlinkage void __aesni_enc(const void *ctx, u8 *out, const u8 *in);
+asmlinkage void __aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out,
+				    const u8 *in, unsigned int len, u8 *iv);
+asmlinkage void __aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out,
+				    const u8 *in, unsigned int len, u8 *iv);
+
+static inline int aesni_enc(const void *ctx, u8 *out, const u8 *in)
+{
+	__aesni_enc(ctx, out, in);
+	return 0;
+}
+
+static inline int aesni_xts_encrypt(const union x86_aes_ctx *ctx, u8 *out, const u8 *in,
+				    unsigned int len, u8 *iv)
+{
+	__aesni_xts_encrypt(&ctx->aesni, out, in, len, iv);
+	return 0;
+}
+
+static inline int aesni_xts_decrypt(const union x86_aes_ctx *ctx, u8 *out, const u8 *in,
+				    unsigned int len, u8 *iv)
+{
+	__aesni_xts_decrypt(&ctx->aesni, out, in, len, iv);
+	return 0;
+}
+
+#endif /* _AESNI_INTEL_GLUE_H */
Signature · 1 line
2.34.1
#146 lore raw

PATCH v9 14/14 crypto: x86/aes-kl - Implement the AES-XTS algorithm

Key Locker is a CPU feature to reduce key exfiltration opportunities. It converts the AES key into an encoded form, called 'key handle', to reduce the exposure of private key material in memory.

This key conversion along with all subsequent data transformations, is provided by new AES instructions ('AES-KL'). AES-KL is analogous to that of AES-NI as maintains a similar programming interface.

Support the XTS mode as the primary use case is dm-crypt. The support has some details worth mentioning, which differentiate itself from AES-NI, that users may need to be aware of:

== Key Handle Restriction ==

The AES-KL instruction set supports selecting key usage restrictions at key handle creation time. Restrict all key handles created by the kernel to kernel mode use only.

Although the AES-KL instructions themselves are executable in userspace, this restriction enforces the mode consistency in its operation.

If the key handle is created in userspace but referenced in the kernel, then encrypt() and decrypt() functions will return -EINVAL.

=== AES-NI Dependency for AES Compliance ===

Key Locker is not AES compliant as it lacks 192-bit key support. However, per the expectations of Linux crypto-cipher implementations, the software cipher implementation must support all the AES-compliant key sizes.

The AES-KL cipher implementation achieves this constraint by logging a warning and falling back to VAES. In other words, the 192-bit key-size limitation is documented but not enforced.

== Wrapping Key Restore Failure Handling ==

In the event of hardware failure, the wrapping key is lost from deep sleep states. Then, the wrapping key turns to zero which is an unusable state.

The x86 core provides valid_keylocker() to indicate the failure. Subsequent setkey() as well as encode()/decode() can check it and return -ENODEV if failed. This allows an error code to be returned instead of encountering abrupt exceptions.

== Userspace Exposition ==

Keylocker implementations have measurable performance penalties. Therefore, keep the current default remains the same.

However, with a slow storage device, storage bandwidth is the bottleneck, even if disk encryption is enabled by AES-KL. Thus, it is up to the end user to decide whether to use AES-KL. User can select it by the name 'xts-aes-aeskl' shown in /proc/crypto.

== 64-bit Only ==

Support 64-bit only, as the 32-bit kernel is being deprecated.

Cc Eric Biggers <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Herbert Xu <[email protected]>

---
I've reworked this patch based on feedback,

    https://lore.kernel.org/lkml/[email protected]/

and rebased to upstream v6.10 Linus merge tree on May 13th: commit 84c7d76b5ab6 ("Merge tag 'v6.10-p1' of
git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6")

According to the dm-crypt benchmark, using VEX-encoded instructions for tweak processing enhances performance by approximately 2-3%. The PCLMULDQD instruction did not yield a measurable impact, so I dropped it to simplify the implementation.

In contrast to other AES instructions, AES-KL does not permit tweak processing between rounds. In XTS mode, a single instruction covers all rounds of 8 blocks without interleaving instructions. Maybe this is one of the reasons for the limited performance gain.

Moving forward, I would like to address any further feedback on this AES-KL driver code first before the next revision of the whole series.

Changes from v9:
* Duplicate the new XTS glue code, instead of sharing (Eric).
* Use VEX-coded instructions for non-AES parts of the code (Eric).
* Adjust ASM code to stylistically follow the new VAES support (Eric).
* Export and reference the high-level AES-NI XTS functions (Eric). Then, support a module build, along with rearranging build dependencies.
* Reorganize the glue code and improve ASM code readability.
* Revoke the review tag due to major changes.

---
 arch/x86/Kconfig.assembler         |   5 +
 arch/x86/crypto/Kconfig            |  18 ++
 arch/x86/crypto/Makefile           |   3 +
 arch/x86/crypto/aeskl-xts-x86_64.S | 358 +++++++++++++++++++++++++++
 arch/x86/crypto/aeskl_glue.c       | 376 +++++++++++++++++++++++++++++
 arch/x86/crypto/aesni-intel_glue.c |  13 +-
 arch/x86/crypto/aesni-xts.h        |  15 ++
 7 files changed, 783 insertions(+), 5 deletions(-)
 create mode 100644 arch/x86/crypto/aeskl-xts-x86_64.S
 create mode 100644 arch/x86/crypto/aeskl_glue.c
 create mode 100644 arch/x86/crypto/aesni-xts.h
arch/x86/Kconfig.assembler +5 -0
index 59aedf32c4ea..89e326c9dbfe 100644
@@ -35,6 +35,11 @@ config AS_VPCLMULQDQ
 	help
 	  Supported by binutils >= 2.30 and LLVM integrated assembler
 
+config AS_HAS_KEYLOCKER
+	def_bool $(as-instr,encodekey256 %eax$(comma)%eax)
+	help
+	  Supported by binutils >= 2.36 and LLVM integrated assembler >= V12
+
 config AS_WRUSS
 	def_bool $(as-instr,wrussq %rax$(comma)(%rbx))
 	help
arch/x86/crypto/Kconfig +18 -0
index c9e59589a1ce..d55704fc9a8f 100644
@@ -29,6 +29,24 @@ config CRYPTO_AES_NI_INTEL
 	  Architecture: x86 (32-bit and 64-bit) using:
 	  - AES-NI (AES new instructions)
 
+config CRYPTO_AES_KL
+	tristate "Ciphers: AES, modes: XTS (AES-KL)"
+	depends on X86 && 64BIT
+	depends on AS_HAS_KEYLOCKER
+	select CRYPTO_AES_NI_INTEL
+	select CRYPTO_SIMD
+	select X86_KEYLOCKER
+
+	help
+	  Block cipher: AES cipher algorithms
+	  Length-preserving ciphers: AES with XTS
+
+	  Architecture: x86 (64-bit) using:
+	  - AES-KL (AES Key Locker)
+	  - AES-NI for a 192-bit key
+
+	  See Documentation/arch/x86/keylocker.rst for more details.
+
 config CRYPTO_BLOWFISH_X86_64
 	tristate "Ciphers: Blowfish, modes: ECB, CBC"
 	depends on X86 && 64BIT
arch/x86/crypto/Makefile +3 -0
index 9c5ce5613738..c46fd2d9dd16 100644
@@ -51,6 +51,9 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
 aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o \
 	aes_ctrby8_avx-x86_64.o aes-xts-avx-x86_64.o
 
+obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-x86_64.o
+aeskl-x86_64-y := aeskl-xts-x86_64.o aeskl_glue.o
+
 obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
 sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
 sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
arch/x86/crypto/aeskl-xts-x86_64.S new file +358 -0

358 lines not on this page · raw message

arch/x86/crypto/aeskl_glue.c new file +376 -0

376 lines not on this page · raw message

arch/x86/crypto/aesni-intel_glue.c +8 -5
index 5b25d2a58aeb..61456f0a99fa 100644
@@ -35,7 +35,7 @@
 #include <linux/workqueue.h>
 #include <linux/spinlock.h>
 #include <linux/static_call.h>
-
+#include "aesni-xts.h"
 
 #define AESNI_ALIGN	16
 #define AESNI_ALIGN_ATTR __attribute__ ((__aligned__(AESNI_ALIGN)))
@@ -864,8 +864,8 @@ static int helper_rfc4106_decrypt(struct aead_request *req)
 }
 #endif
 
-static int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key,
-			    unsigned int keylen)
+int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key,
+		     unsigned int keylen)
 {
 	struct aesni_xts_ctx *ctx = aes_xts_ctx(tfm);
 	int err;
@@ -884,6 +884,7 @@ static int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key,
 	/* second half of xts-key is for tweak */
 	return aes_set_key_common(&ctx->tweak_ctx, key + keylen, keylen);
 }
+EXPORT_SYMBOL_GPL(xts_setkey_aesni);
 
 typedef void (*xts_encrypt_iv_func)(const struct crypto_aes_ctx *tweak_key,
 				    u8 iv[AES_BLOCK_SIZE]);
@@ -1020,15 +1021,17 @@ static void aesni_xts_decrypt(const struct crypto_aes_ctx *key,
 	aesni_xts_dec(key, dst, src, len, tweak);
 }
 
-static int xts_encrypt_aesni(struct skcipher_request *req)
+int xts_encrypt_aesni(struct skcipher_request *req)
 {
 	return xts_crypt(req, aesni_xts_encrypt_iv, aesni_xts_encrypt);
 }
+EXPORT_SYMBOL_GPL(xts_encrypt_aesni);
 
-static int xts_decrypt_aesni(struct skcipher_request *req)
+int xts_decrypt_aesni(struct skcipher_request *req)
 {
 	return xts_crypt(req, aesni_xts_encrypt_iv, aesni_xts_decrypt);
 }
+EXPORT_SYMBOL_GPL(xts_decrypt_aesni);
 
 static struct crypto_alg aesni_cipher_alg = {
 	.cra_name		= "aes",
arch/x86/crypto/aesni-xts.h new file +15 -0
new file mode 100644
index 000000000000..9833da2bd9d2
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _AESNI_XTS_H
+#define _AESNI_XTS_H
+
+/*
+ * These AES-NI functions are used by the AES-KL code as a fallback when
+ * a 192-bit key is provided. Key Locker does not support 192-bit keys.
+ */
+
+int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen);
+int xts_encrypt_aesni(struct skcipher_request *req);
+int xts_decrypt_aesni(struct skcipher_request *req);
+
+#endif /* _AESNI_XTS_H */
Signature · 1 line
2.34.1
#147 lore raw

Re: [PATCH v9a 14/14] crypto: x86/aes-kl - Implement the AES-XTS algorithm

Quoted 19 lines · Chang S. Bae

I've reworked this patch based on feedback,

    https://lore.kernel.org/lkml/[email protected]/

and rebased to upstream v6.10 Linus merge tree on May 13th: commit 84c7d76b5ab6 ("Merge tag 'v6.10-p1' of
git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6")

According to the dm-crypt benchmark, using VEX-encoded instructions for tweak processing enhances performance by approximately 2-3%. The PCLMULDQD instruction did not yield a measurable impact, so I dropped it to simplify the implementation.

In contrast to other AES instructions, AES-KL does not permit tweak processing between rounds. In XTS mode, a single instruction covers all rounds of 8 blocks without interleaving instructions. Maybe this is one of the reasons for the limited performance gain.

Moving forward, I would like to address any further feedback on this AES-KL driver code first before the next revision of the whole series.

Quoted 9 lines

Changes from v9:
* Duplicate the new XTS glue code, instead of sharing (Eric).
* Use VEX-coded instructions for non-AES parts of the code (Eric).
* Adjust ASM code to stylistically follow the new VAES support (Eric).
* Export and reference the high-level AES-NI XTS functions (Eric). Then, support a module build, along with rearranging build dependencies.
* Reorganize the glue code and improve ASM code readability.
* Revoke the review tag due to major changes.
---

Thanks for the updated patch!

arch/x86/Kconfig.assembler config AS_VPCLMULQDQ · line 41

Quoted 12 lines
arch/x86/Kconfig.assembler +4 -0
index 59aedf32c4ea..89e326c9dbfe 100644
@@ -35,6 +35,11 @@ config AS_VPCLMULQDQ
 	help
 	  Supported by binutils >= 2.30 and LLVM integrated assembler
 
+config AS_HAS_KEYLOCKER
+	def_bool $(as-instr,encodekey256 %eax$(comma)%eax)
+	help
+	  Supported by binutils >= 2.36 and LLVM integrated assembler >= V12

Adding AS_HAS_KEYLOCKER should be its own patch.

Quoted 52 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +46 -0
new file mode 100644
index 000000000000..6ff8b5feebfc
@@ -0,0 +1,358 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Implement AES algorithm using AES Key Locker instructions.
+ *
+ * Most code is primarily derived from aesni-intel_asm.S and
+ * stylistically aligned with aes-xts-avx-x86_64.S.
+ */
+
+#include <linux/linkage.h>
+#include <linux/cfi_types.h>
+#include <asm/errno.h>
+#include <asm/inst.h>
+#include <asm/frame.h>
+
+/* Constant values shared between AES implementations: */
+
+.section .rodata
+.p2align 4
+.Lgf_poly:
+	/*
+	 * Represents the polynomial x^7 + x^2 + x + 1, where the low 64
+	 * bits are XOR'd into the tweak's low 64 bits when a carry
+	 * occurs from the high 64 bits.
+	 */
+	.quad	0x87, 1
+
+	/*
+	 * Table of constants for variable byte shifts and blending
+	 * during ciphertext stealing operations.
+	 */
+.Lcts_permute_table:
+	.byte	0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+	.byte	0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+	.byte	0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07
+	.byte	0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f
+	.byte	0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+	.byte	0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+
+.text
+
+.set	V0,		%xmm0
+.set	V1,		%xmm1
+.set	V2,		%xmm2
+.set	V3,		%xmm3
+.set	V4,		%xmm4
+.set	V5,		%xmm5

arch/x86/crypto/aeskl-xts-x86_64.S line 56

Quoted 10 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +10 -0
+.set	V6,		%xmm6
+.set	V7,		%xmm7
+.set	V8,		%xmm8
+.set	V9,		%xmm9
+.set	V10,		%xmm10
+.set	V11,		%xmm11
+.set	V12,		%xmm12
+.set	V13,		%xmm13
+.set	V14,		%xmm14
+.set	V15,		%xmm15

The point of the V[0-15] aliases in aes-xts-avx-x86_64.S are to support both ymm and zmm registers. Here all registers are xmm, so there's no need for the layer of indirection and you should just use xmm[0-15] directly.

arch/x86/crypto/aeskl-xts-x86_64.S line 68

Quoted 11 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +11 -0
+.set	TWEAK_XMM1,	V8
+.set	TWEAK_XMM2,	V9
+.set	TWEAK_XMM3,	V10
+.set	TWEAK_XMM4,	V11
+.set	TWEAK_XMM5,	V12
+.set	TWEAK_XMM6,	V13
+.set	TWEAK_XMM7,	V14
+.set	GF_POLY_XMM,	V15
+.set	TWEAK_TMP,	TWEAK_XMM1
+.set	TWEAK_XMM,	TWEAK_XMM2
+.set	TMP,		%r10

Similarly, the _XMM suffixes are not really helpful since ymm and zmm registers are not in play here.

arch/x86/crypto/aeskl-xts-x86_64.S line 77

Quoted 8 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +8 -0
+/* Function parameters */
+.set	HANDLEP,	%rdi	/* Pointer to struct aeskl_ctx */
+.set	DST,		%rsi	/* Pointer to next destination data */
+.set	UKEYP,		DST	/* Pointer to the original key */
+.set	KLEN,		%r9d	/* AES key length in bytes */
+.set	SRC,		%rdx	/* Pointer to next source data */
+.set	LEN,		%rcx	/* Remaining length in bytes */
+.set	TWEAK,		%r8	/* Pointer to next tweak */

Please don't put parameters for different functions in the same list like this. There should be a separate list at the beginning of each function. (Yes, it doesn't work perfectly because '.set' is global and doesn't go out of scope once the function ends. But at least this would make it clear what the intent is.)

Also LEN needs to be %ecx, not %rcx, because it is unsigned int.

Quoted 13 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +13 -0
+SYM_FUNC_START(__aeskl_setkey)
+	FRAME_BEGIN
+	movl		%edx, 480(HANDLEP)
+	vmovdqu		(UKEYP), V0
+	mov		$1, %eax
+	cmp		$16, %dl
+	je		.Lsetkey_128
+
+	vmovdqu		0x10(UKEYP), V1
+	encodekey256	%eax, %eax
+	vmovdqu		V3, 0x30(HANDLEP)
+	jmp		.Lsetkey_end
+.Lsetkey_128:

arch/x86/crypto/aeskl-xts-x86_64.S line 105

Quoted 10 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +10 -0
+	encodekey128	%eax, %eax
+
+.Lsetkey_end:
+	vmovdqu		V0, 0x00(HANDLEP)
+	vmovdqu		V1, 0x10(HANDLEP)
+	vmovdqu		V2, 0x20(HANDLEP)
+
+	FRAME_END
+	RET
+SYM_FUNC_END(__aeskl_setkey)

These are all leaf functions, so they don't need FRAME_BEGIN and FRAME_END.

Quoted 23 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +23 -0
+.macro _aeskl		width, operation
+	cmp		$16, KLEN
+	je		.Laeskl128\@
+.ifc \width, wide
+ .ifc \operation, dec
+	aesdecwide256kl	(HANDLEP)
+ .else
+	aesencwide256kl	(HANDLEP)
+ .endif
+.else
+ .ifc \operation, dec
+	aesdec256kl	(HANDLEP), V0
+ .else
+	aesenc256kl	(HANDLEP), V0
+ .endif
+.endif
+	jmp		.Laesklend\@
+.Laeskl128\@:
+.ifc \width, wide
+ .ifc \operation, dec
+	aesdecwide128kl	(HANDLEP)
+ .else
+	aesencwide128kl	(HANDLEP)

arch/x86/crypto/aeskl-xts-x86_64.S line 139

Quoted 10 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +10 -0
+ .endif
+.else
+ .ifc \operation, dec
+	aesdec128kl	(HANDLEP), V0
+ .else
+	aesenc128kl	(HANDLEP), V0
+ .endif
+.endif
+.Laesklend\@:
+.endm

I think it would be easier to read if this was split into two macros, one for 1-block and one for 8-block.

Quoted 4 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +4 -0
+/* int __aeskl_enc(const void *handlep, u8 *dst, const u8 *src) */
+SYM_FUNC_START(__aeskl_enc)
+	FRAME_BEGIN
+	vmovdqu		(SRC), V0

arch/x86/crypto/aeskl-xts-x86_64.S line 154

Quoted 10 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +10 -0
+	movl		480(HANDLEP), KLEN
+
+	_aeskl		oneblock, enc
+	jz		.Lerror
+	xor		%rax, %rax
+	vmovdqu		V0, (DST)
+	FRAME_END
+	RET
+.Lerror:
+	mov		$(-EINVAL), %rax

For returning an int, %eax should be used, not %rax.

(Note that instructions that operate on %eax also tend to be slightly shorter.)

arch/x86/crypto/aeskl-xts-x86_64.S line 354

Quoted 12 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +12 -0
+/*
+ * int __aeskl_xts_encrypt(const struct aeskl_ctx *handlep, u8 *dst,
+ *			   const u8 *src, unsigned int klen, le128 *tweak)
+ */
+SYM_FUNC_START(__aeskl_xts_encrypt)
+	_aeskl_xts_crypt	enc
+SYM_FUNC_END(__aeskl_xts_encrypt)
+
+/*
+ * int __aeskl_xts_decrypt(const struct crypto_aes_ctx *handlep, u8 *dst,
+ *			   const u8 *src, unsigned int klen, le128 *twek)
+ */

Please make sure the function prototypes, including the parameter names, match the ones used in the .c file and also the lists of register aliases.

Quoted 33 lines
arch/x86/crypto/aeskl_glue.c new file +27 -0
new file mode 100644
index 000000000000..6dc4d380be54
@@ -0,0 +1,376 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Support for AES Key Locker instructions. This file contains glue
+ * code and the real AES implementation is in aeskl-intel_asm.S.
+ *
+ * Most code is based on aesni-intel_glue.c
+ */
+
+#include <linux/types.h>
+#include <linux/module.h>
+#include <linux/err.h>
+#include <crypto/algapi.h>
+#include <crypto/aes.h>
+#include <crypto/xts.h>
+#include <crypto/scatterwalk.h>
+#include <crypto/internal/skcipher.h>
+#include <crypto/internal/simd.h>
+#include <asm/simd.h>
+#include <asm/cpu_device_id.h>
+#include <asm/fpu/api.h>
+#include <asm/keylocker.h>
+#include "aesni-xts.h"
+
+#define AESKL_ALIGN		16
+#define AESKL_ALIGN_ATTR	__attribute__ ((__aligned__(AESKL_ALIGN)))
+#define AESKL_ALIGN_EXTRA	((AESKL_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1))
+

arch/x86/crypto/aeskl_glue.c line 37

Quoted 10 lines
arch/x86/crypto/aeskl_glue.c new file +10 -0
+#define AESKL_AAD_SIZE		16
+#define AESKL_TAG_SIZE		16
+#define AESKL_CIPHERTEXT_MAX	AES_KEYSIZE_256
+
+/* The Key Locker handle is an encoded form of the AES key. */
+struct aeskl_handle {
+	u8 additional_authdata[AESKL_AAD_SIZE];
+	u8 integrity_tag[AESKL_TAG_SIZE];
+	u8 ciphre_text[AESKL_CIPHERTEXT_MAX];
+};

ciphre_text => ciphertext

arch/x86/crypto/aeskl_glue.c line 42

Quoted 4 lines
arch/x86/crypto/aeskl_glue.c new file +4 -0
+/*
+ * Key Locker does not support 192-bit key size. The driver needs to
+ * retrieve the key size in the first place. The offset of the
+ * 'key_length' field here should be compatible with struct

should => must

Quoted 5 lines
arch/x86/crypto/aeskl_glue.c new file +5 -0
+ * crypto_aes_ctx.
+ */
+#define AESKL_CTX_RESERVED (sizeof(struct crypto_aes_ctx) - sizeof(struct aeskl_handle) \
+			    - sizeof(u32))
+

arch/x86/crypto/aeskl_glue.c line 57

Quoted 10 lines
arch/x86/crypto/aeskl_glue.c new file +10 -0
+struct aeskl_ctx {
+	struct aeskl_handle handle;
+	u8 reserved[AESKL_CTX_RESERVED];
+	u32 key_length;
+};
+
+struct aeskl_xts_ctx {
+	struct aeskl_ctx tweak_ctx AESKL_ALIGN_ATTR;
+	struct aeskl_ctx crypt_ctx AESKL_ALIGN_ATTR;
+};

So there's a union between aeskl_ctx and crypto_aes_ctx going on here, but it's not made explicit through a C union. How about doing that?

Also, there should be a BUILD_BUG_ON() that enforces that the key_length is really at the same offset in both.

arch/x86/crypto/aeskl_glue.c line 152

Quoted 12 lines
arch/x86/crypto/aeskl_glue.c new file +12 -0
+/*
+ * The glue code in xts_crypt() and xts_crypt_slowpath() follows
+ * aesni-intel_glue.c. While this code is shareable, the key
+ * material format difference can cause more destructive code changes in
+ * the AES-NI side.
+ */
+
+typedef int (*xts_encrypt_iv_func)(const struct aeskl_ctx *tweak_key,
+				   u8 iv[AES_BLOCK_SIZE]);
+typedef int (*xts_crypt_func)(const struct aeskl_ctx *key,
+			      const u8 *src, u8 *dst, unsigned int len,
+			      u8 tweak[AES_BLOCK_SIZE]);

Since there are so few functions in play here (one xts_encrypt_iv_func and two xts_crypt_func) I think you should just use direct calls instead of function pointers. A simple parameter 'bool enc' would take care of selecting the encryption function vs. the decryption one.

One of the issues with indirect calls, even when inlined with the intention that they be optimized out, is that there's no guarantee that the compiler will actually optimize them out. That has the consequence that CFI stubs are still needed in the assembly.

Direct calls avoid this issue.

(BTW, in my AES-GCM patchset I'm using direct calls:
https://lore.kernel.org/linux-crypto/[email protected]/.
I'm thinking I should have used that approach with AES-XTS too.)

arch/x86/crypto/aeskl_glue.c line 320

Quoted 6 lines
arch/x86/crypto/aeskl_glue.c new file +6 -0
+static struct skcipher_alg aeskl_skciphers[] = {
+	{
+		.base = {
+			.cra_name		= "__xts(aes)",
+			.cra_driver_name	= "__xts-aes-aeskl",
+			.cra_priority		= 200,

Maybe add a comment here that explains that this is intentionally made lower priority than xts-aes-aesni.

Quoted 8 lines
arch/x86/crypto/aeskl_glue.c new file +8 -0
+static int __init aeskl_init(void)
+{
+	u32 eax, ebx, ecx, edx;
+
+	if (!valid_keylocker())
+		return -ENODEV;
+
+	cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx);

arch/x86/crypto/aeskl_glue.c line 355

Quoted 10 lines
arch/x86/crypto/aeskl_glue.c new file +10 -0
+	if (!(ebx & KEYLOCKER_CPUID_EBX_WIDE))
+		return -ENODEV;
+
+	/*
+	 * AES-KL itself does not rely on AES-NI. But, AES-KL does not
+	 * support 192-bit keys. To ensure AES compliance, AES-KL falls
+	 * back to AES-NI.
+	 */
+	if (!cpu_feature_enabled(X86_FEATURE_AES))
+		return -ENODEV;

Everywhere else in arch/x86/crypto/ uses boot_cpu_has(), not cpu_feature_enabled().

arch/x86/crypto/aeskl_glue.c line 359

Quoted 4 lines
arch/x86/crypto/aeskl_glue.c new file +4 -0
+
+	/* The tweak processing is optimized using AVX instructions. */
+	if (!cpu_feature_enabled(X86_FEATURE_AVX))
+		return -ENODEV;

The whole implementation uses VEX-coded instructions now, not just the tweak processing. So either fix or delete the above comment.

- Eric

#148 lore raw

PATCH v9 14/14 crypto: x86/aes-kl - Implement the AES-XTS algorithm

Hi Eric,

I really appreciate your review. Keeping track of about 800 lines of code, including assembly lines, in a single patch is demanding. I hope this version meets your expectations.

The overall diff can be found here:
https://github.com/intel-staging/keylocker/compare/f8420d4e27fc..57472d9b3f8e

Thanks,
Chang

---
Key Locker is a CPU feature to reduce key exfiltration opportunities. It converts the AES key into an encoded form, called 'key handle', to reduce the exposure of private key material in memory.

This key conversion along with all subsequent data transformations, is provided by new AES instructions ('AES-KL'). AES-KL is analogous to that of AES-NI as maintains a similar programming interface.

Support the XTS mode as the primary use case is dm-crypt. The support has some details worth mentioning, which differentiate itself from AES-NI, that users may need to be aware of:

== Key Handle Restriction ==

The AES-KL instruction set supports selecting key usage restrictions at key handle creation time. Restrict all key handles created by the kernel to kernel mode use only.

Although the AES-KL instructions themselves are executable in userspace, this restriction enforces the mode consistency in its operation.

If the key handle is created in userspace but referenced in the kernel, then encrypt() and decrypt() functions will return -EINVAL.

=== AES-NI Dependency for AES Compliance ===

Key Locker is not AES compliant as it lacks 192-bit key support. However, per the expectations of Linux crypto-cipher implementations, the software cipher implementation must support all the AES-compliant key sizes.

The AES-KL cipher implementation achieves this constraint by logging a warning and falling back to AES-NI. In other words, the 192-bit key-size limitation is documented but not enforced.

== Wrapping Key Restore Failure Handling ==

In the event of hardware failure, the wrapping key is lost from deep sleep states. Then, the wrapping key turns to zero which is an unusable state.

The x86 core provides valid_keylocker() to indicate the failure. Subsequent setkey() as well as encode()/decode() can check it and return -ENODEV if failed. This allows an error code to be returned instead of encountering abrupt exceptions.

== Userspace Exposition ==

Keylocker implementations have measurable performance penalties. Therefore, keep the current default remains the same.

However, with a slow storage device, storage bandwidth is the bottleneck, even if disk encryption is enabled by AES-KL. Thus, it is up to the end user to decide whether to use AES-KL. User can select it by the name 'xts-aes-aeskl' shown in /proc/crypto.

== 64-bit Only ==

Support 64-bit only, as the 32-bit kernel is being deprecated.

---
Changes from the previous posting (Eric):
* Assembly Code
- Improve function argument descriptions.
- Simplify the AES processing macro.
- Rename some symbols.
* Glue Code:
- Do not use function pointer variables; direct calls.
- Define a union for the two 'ctx' structures.
- Clarify a few code spots with comments.
- Adjust some variable and struct names.
* Kconfig
- Separate out the Kconfig.assembler change.

---
 arch/x86/crypto/Kconfig            |  18 ++
 arch/x86/crypto/Makefile           |   3 +
 arch/x86/crypto/aeskl-xts-x86_64.S | 337 ++++++++++++++++++++++++
 arch/x86/crypto/aeskl_glue.c       | 409 +++++++++++++++++++++++++++++
 arch/x86/crypto/aesni-intel_glue.c |  13 +-
 arch/x86/crypto/aesni-xts.h        |  15 ++
 6 files changed, 790 insertions(+), 5 deletions(-)
 create mode 100644 arch/x86/crypto/aeskl-xts-x86_64.S
 create mode 100644 arch/x86/crypto/aeskl_glue.c
 create mode 100644 arch/x86/crypto/aesni-xts.h
arch/x86/crypto/Kconfig +18 -0
index c9e59589a1ce..c45d8f48f24e 100644
@@ -29,6 +29,24 @@ config CRYPTO_AES_NI_INTEL
 	  Architecture: x86 (32-bit and 64-bit) using:
 	  - AES-NI (AES new instructions)
 
+config CRYPTO_AES_KL
+	tristate "Ciphers: AES, modes: XTS (AES-KL)"
+	depends on X86 && 64BIT
+	depends on AS_KEYLOCKER
+	select CRYPTO_AES_NI_INTEL
+	select CRYPTO_SIMD
+	select X86_KEYLOCKER
+
+	help
+	  Block cipher: AES cipher algorithms
+	  Length-preserving ciphers: AES with XTS
+
+	  Architecture: x86 (64-bit) using:
+	  - AES-KL (AES Key Locker)
+	  - AES-NI for a 192-bit key
+
+	  See Documentation/arch/x86/keylocker.rst for more details.
+
 config CRYPTO_BLOWFISH_X86_64
 	tristate "Ciphers: Blowfish, modes: ECB, CBC"
 	depends on X86 && 64BIT
arch/x86/crypto/Makefile +3 -0
index 9c5ce5613738..c46fd2d9dd16 100644
@@ -51,6 +51,9 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
 aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o \
 	aes_ctrby8_avx-x86_64.o aes-xts-avx-x86_64.o
 
+obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-x86_64.o
+aeskl-x86_64-y := aeskl-xts-x86_64.o aeskl_glue.o
+
 obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
 sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
 sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
arch/x86/crypto/aeskl-xts-x86_64.S new file +337 -0

337 lines not on this page · raw message

arch/x86/crypto/aeskl_glue.c new file +409 -0

409 lines not on this page · raw message

arch/x86/crypto/aesni-intel_glue.c +8 -5
index ef031655b2d3..49fb56efac56 100644
@@ -35,7 +35,7 @@
 #include <linux/workqueue.h>
 #include <linux/spinlock.h>
 #include <linux/static_call.h>
-
+#include "aesni-xts.h"
 
 #define AESNI_ALIGN	16
 #define AESNI_ALIGN_ATTR __attribute__ ((__aligned__(AESNI_ALIGN)))
@@ -864,8 +864,8 @@ static int helper_rfc4106_decrypt(struct aead_request *req)
 }
 #endif
 
-static int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key,
-			    unsigned int keylen)
+int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key,
+		     unsigned int keylen)
 {
 	struct aesni_xts_ctx *ctx = aes_xts_ctx(tfm);
 	int err;
@@ -884,6 +884,7 @@ static int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key,
 	/* second half of xts-key is for tweak */
 	return aes_set_key_common(&ctx->tweak_ctx, key + keylen, keylen);
 }
+EXPORT_SYMBOL_GPL(xts_setkey_aesni);
 
 typedef void (*xts_encrypt_iv_func)(const struct crypto_aes_ctx *tweak_key,
 				    u8 iv[AES_BLOCK_SIZE]);
@@ -1020,15 +1021,17 @@ static void aesni_xts_decrypt(const struct crypto_aes_ctx *key,
 	aesni_xts_dec(key, dst, src, len, tweak);
 }
 
-static int xts_encrypt_aesni(struct skcipher_request *req)
+int xts_encrypt_aesni(struct skcipher_request *req)
 {
 	return xts_crypt(req, aesni_xts_encrypt_iv, aesni_xts_encrypt);
 }
+EXPORT_SYMBOL_GPL(xts_encrypt_aesni);
 
-static int xts_decrypt_aesni(struct skcipher_request *req)
+int xts_decrypt_aesni(struct skcipher_request *req)
 {
 	return xts_crypt(req, aesni_xts_encrypt_iv, aesni_xts_decrypt);
 }
+EXPORT_SYMBOL_GPL(xts_decrypt_aesni);
 
 static struct crypto_alg aesni_cipher_alg = {
 	.cra_name		= "aes",
arch/x86/crypto/aesni-xts.h new file +15 -0
new file mode 100644
index 000000000000..9833da2bd9d2
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _AESNI_XTS_H
+#define _AESNI_XTS_H
+
+/*
+ * These AES-NI functions are used by the AES-KL code as a fallback when
+ * a 192-bit key is provided. Key Locker does not support 192-bit keys.
+ */
+
+int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen);
+int xts_encrypt_aesni(struct skcipher_request *req);
+int xts_decrypt_aesni(struct skcipher_request *req);
+
+#endif /* _AESNI_XTS_H */
Signature · 1 line
2.34.1
#149 lore raw
In reply to Chang S. Bae
Quoted 9 lines · Chang S. Bae

Then, the following is a summary of changes per patch since v8 [6]:

PATCH7-8:
* Invoke the setup code via arch_initcall() due to upstream changes delaying the FPU setup.

PATCH9-11:
* Add new patches for security and hotplug support clarification

I've recently made updates to a few patches, primarily related to the mitigation parts. While the series is still under review, Eric's VAES patches have been merged into the crypto tree and are currently being sorted out. Once things settle down, I will make a few adjustments on the crypto side. Then, another revision will be necessary thereafter.

Thanks,
Chang

#150 lore raw

Hi,

Quoted 11 lines · Chang S. Bae
Quoted 9 lines · Chang S. Bae

Then, the following is a summary of changes per patch since v8 [6]:

PATCH7-8:
* Invoke the setup code via arch_initcall() due to upstream changes delaying the FPU setup.

PATCH9-11:
* Add new patches for security and hotplug support clarification

Quoted 8 lines

I've recently made updates to a few patches, primarily related to the mitigation parts. While the series is still under review, Eric's VAES patches have been merged into the crypto tree and are currently being sorted out. Once things settle down, I will make a few adjustments on the crypto side. Then, another revision will be necessary thereafter.

Thanks,
Chang

Thanks for the updated patchset!

Do you have a plan for how this will be merged? Which trees will the patches go through? I think that the actual AES-XTS implementation could still use a bit more polishing; see my comments below. However, patches 1-12 don't need to wait for that. Perhaps the x86 maintainers would like to take patches 1-12 for v6.10? Then the AES-XTS support can go through the crypto tree afterwards.

As you noticed, this cycle I've been optimizing AES-XTS for x86_64 by adding new VAES and AES-NI + AVX implementations. I have some ideas for the Key Locker based implementation of AES-XTS:

First, surely it's the case that in practice, all CPUs that support Key Locker also support AVX? If so, then there's no need for the Key Locker assembly to use legacy SSE instructions. It should instead target AVX and use VEX-coded instructions. This would save some instructions and improve performance.

Since the Key Locker assembly only supports 64-bit mode, it should also feel free to use registers xmm8-xmm15 for purposes such as caching the XTS tweaks. This would improve performance.

Since the Key Locker assembly advances a large number of XTS tweaks at a time (8), I'm also wondering if it would be faster to multiply by x^8 directly instead of multiplying by x sequentially eight times. This can be done using the pclmulqdq instruction; see aes-xts-avx-x86_64.S which implements this optimization. Probably all CPUs that support Key Locker also support PCLMULQDQ.

I'm also trying to think of the best way to organize the Key Locker AES-XTS glue code. I see that you're proposing to share the glue code with the existing AES-XTS implementations. Unfortunately I don't think this ends up working very well, due to the facts that the Key Locker code can return errors and uses a different key type. I think that for now, I'd prefer that you simply copied the XTS glue code into aeskl-intel_glue.c and modified it as needed. (But make sure to use the new version of the glue code, which is faster.)

For falling back to AES-NI, I think the cleanest solution is to call the top-level setkey, encrypt, and decrypt functions (the ones that are set in the xts-aes-aesni skcipher_alg), instead of calling lower-level functions as your current patchset does.

If you could keep the Key Locker assembly roughly stylistically consistent with the new aes-xts-avx-x86_64.S, that would be great too.

Do you happen to know if there's any way to test the Key Locker AES-XTS code without having access to a bare metal machine with a CPU that supports Key Locker? I tried a Sapphire Rapids based VM in Google Compute Engine, but it doesn't enumerate Key Locker. I also don't see anything in QEMU related to Key Locker. So I don't currently have an easy way to test this patchset.

Finally, a high level question. Key Locker has been reported to be substantially slower than AES-NI. At the same time, VAES has recently doubled performance over AES-NI. I'd guess this leaves Key Locker even further behind. Given that, how useful is this patchset? I'm a bit concerned that this might be something that sounds good in theory but won't be used in practice. Are performance improvements for Key Locker on the horizon? (Well, there are the improvements I suggested above, which should help, but it sounds like main issue is the Key Locker instructions themselves which are just fundamentally slower.)

- Eric

#151 lore raw

Hi Eric,

Sorry for my delay. I just found your message in my cluttered junk folder today after returning from my week off.

Quoted 6 lines · Eric Biggers

Do you have a plan for how this will be merged? Which trees will the patches go through? I think that the actual AES-XTS implementation could still use a bit more polishing; see my comments below. However, patches 1-12 don't need to wait for that. Perhaps the x86 maintainers would like to take patches 1-12 for v6.10? Then the AES-XTS support can go through the crypto tree afterwards.

Yeah, this series spans both x86 code and the crypto driver. I believe the decision should be made by maintainers. But, I suspect they'll want to see a well-established use case code before moving forward.

Quoted 3 lines

As you noticed, this cycle I've been optimizing AES-XTS for x86_64 by adding new VAES and AES-NI + AVX implementations. I have some ideas for the Key Locker based implementation of AES-XTS:

Thanks for the effort! I agree that the code could be beneficial for daily disk encryption needs.

Quoted 4 lines

First, surely it's the case that in practice, all CPUs that support Key Locker also support AVX? If so, then there's no need for the Key Locker assembly to use legacy SSE instructions. It should instead target AVX and use VEX-coded instructions. This would save some instructions and improve performance.

Unfortunately, the Key Locker instructions using the AVX states were never implemented.

Quoted 9 lines

Since the Key Locker assembly only supports 64-bit mode, it should also feel free to use registers xmm8-xmm15 for purposes such as caching the XTS tweaks. This would improve performance.

Since the Key Locker assembly advances a large number of XTS tweaks at a time (8), I'm also wondering if it would be faster to multiply by x^8 directly instead of multiplying by x sequentially eight times. This can be done using the pclmulqdq instruction; see aes-xts-avx-x86_64.S which implements this optimization. Probably all CPUs that support Key Locker also support PCLMULQDQ.

I'll revisit the assembly code to incorporate your suggestions.

Quoted 7 lines

I'm also trying to think of the best way to organize the Key Locker AES-XTS glue code. I see that you're proposing to share the glue code with the existing AES-XTS implementations. Unfortunately I don't think this ends up working very well, due to the facts that the Key Locker code can return errors and uses a different key type. I think that for now, I'd prefer that you simply copied the XTS glue code into aeskl-intel_glue.c and modified it as needed. (But make sure to use the new version of the glue code, which is faster.)

I agree; the proposed glue code looks messy due to the different return errors and key types. Ard made a point earlier about establishing a shared common code as they're logically quite similar. But I suppose it is more practical to pursue a separate glue code at this point.

Quoted 4 lines

For falling back to AES-NI, I think the cleanest solution is to call the top-level setkey, encrypt, and decrypt functions (the ones that are set in the xts-aes-aesni skcipher_alg), instead of calling lower-level functions as your current patchset does.

Yes, falling back is indeed one of the ugly parts of this series. Let me retry this as you suggested.

Quoted 2 lines

If you could keep the Key Locker assembly roughly stylistically consistent with the new aes-xts-avx-x86_64.S, that would be great too.

Okay.

Quoted 5 lines

Do you happen to know if there's any way to test the Key Locker AES-XTS code without having access to a bare metal machine with a CPU that supports Key Locker? I tried a Sapphire Rapids based VM in Google Compute Engine, but it doesn't enumerate Key Locker. I also don't see anything in QEMU related to Key Locker. So I don't currently have an easy way to test this patchset.

No, there isn't currently an emulation option available to the public that I'm aware of. this feature has been available on client systems since the Tiger Lake generation.

Quoted 8 lines

Finally, a high level question. Key Locker has been reported to be substantially slower than AES-NI. At the same time, VAES has recently doubled performance over AES-NI. I'd guess this leaves Key Locker even further behind. Given that, how useful is this patchset? I'm a bit concerned that this might be something that sounds good in theory but won't be used in practice. Are performance improvements for Key Locker on the horizon? (Well, there are the improvements I suggested above, which should help, but it sounds like main issue is the Key Locker instructions themselves which are just fundamentally slower.)

On our latest implementations, we've observed the Key Locker performance on cryptsetup seems to be roughly the same as what we posted earlier [2]. Yes, this sounds like a fair analogy to me, especially given your vAES code.

Thanks,
Chang

[1] https://lore.kernel.org/lkml/CAMj1kXGa4f21eH0mdxd1pQsZMUjUr1Btq+Dgw-gC=O-yYft7xw@mail.gmail.com/
[2] https://lore.kernel.org/lkml/[email protected]/

#152 lore raw
Quoted 7 lines · Chang S. Bae
Quoted 4 lines

First, surely it's the case that in practice, all CPUs that support Key Locker also support AVX? If so, then there's no need for the Key Locker assembly to use legacy SSE instructions. It should instead target AVX and use VEX-coded instructions. This would save some instructions and improve performance.

Unfortunately, the Key Locker instructions using the AVX states were never implemented.

Sure, you could still use VEX-coded 128-bit instructions for everything other than the actual AES (for example, the XTS tweak computation) though, right? They're a bit more convenient to work with since they are non-destructive.

- Eric

#153 lore raw
Quoted 4 lines · Eric Biggers

Sure, you could still use VEX-coded 128-bit instructions for everything other than the actual AES (for example, the XTS tweak computation) though, right? They're a bit more convenient to work with since they are non-destructive.

Right.

Thanks,
Chang