x86: Support Key Locker
Subsystems (7)
PATCH v9 12/14 x86/Kconfig: Add symbols for Key Locker
In reply to Chang S. BaeAdd CONFIG_X86_KEYLOCKER to control whether Key Locker is initialized at boot. Additionally, add the AS_KEYLOCKER config symbol to indicate whether the assembler supports Key Locker.
The former will be selected, and the latter will be referenced by the Key Locker cipher module CRYPTO_AES_KL, to be added in a later patch.
---
Changes from v9:
* Include AS_KEYLOCKER symbol (Eric Biggers).
* Revoke the earlier tag.
--- arch/x86/Kconfig | 3 +++ arch/x86/Kconfig.assembler | 5 +++++ 2 files changed, 8 insertions(+)
arch/x86/Kconfig +3 -0
@@ -1881,6 +1881,9 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS If unsure, say y. +config X86_KEYLOCKER + bool + choice prompt "TSX enable mode" depends on CPU_SUP_INTEL
arch/x86/Kconfig.assembler +5 -0
@@ -35,6 +35,11 @@ config AS_VPCLMULQDQ help Supported by binutils >= 2.30 and LLVM integrated assembler +config AS_KEYLOCKER + def_bool $(as-instr,encodekey256 %eax$(comma)%eax) + help + Supported by binutils >= 2.36 and LLVM integrated assembler >= V12 + config AS_WRUSS def_bool $(as-instr,wrussq %rax$(comma)(%rbx)) help
Signature · 1 line
2.34.1
PATCH v9 13/14 crypto: x86/aes - Prepare for new AES-XTS implementation
In reply to Chang S. BaeThe key Locker's AES instruction set ('AES-KL') shares a similar programming interface with AES-NI. The internal ABI in the assembly code will have the same prototype as AES-NI, and the glue code will also be identical.
The upcoming AES code will exclusively support the XTS mode as disk encryption is the only intended use case.
Refactor the XTS-related code to eliminate code duplication and relocate certain constant values to make them shareable. Also, introduce wrappers for data transformation functions to return an error code, as AES-KL may populate it.
Introduce union x86_aes_ctx as AES-KL will reference an encoded form instead of an expanded AES key. This allows different AES context formats in the shared code.
Inline the refactored code to the caller to prevent the potential overhead of indirect calls.
No functional changes or performance regressions are intended.
---
Changes from v8:
* Rebase on the AES-NI changes in the mainline -- mostly cleanup works.
* Introduce 'union x86_aes_ctx'. (Eric Biggers)
* Ensure 'inline' for wrapper functions.
* Tweak the changelog.
Changes from v7:
* Remove aesni_dec() as not referenced by the refactored helpers. But, keep the ASM symbol '__aesni_dec' to make it appears consistent with its counterpart '__aesni_enc'.
* Call out 'AES-XTS' in the subject.
Changes from v6:
* Inline the helper code to avoid the indirect call. (Eric Biggers)
* Rename the filename: aes-intel* -> aes-helper*. (Eric Biggers)
* Don't export symbols yet here. Instead, do it when needed later.
* Improve the coding style:
- Follow the symbol convention: '_' -> '__' (Eric Biggers)
- Fix a style issue -- 'dst = src = ...' catched by checkpatch.pl:
"CHECK: multiple assignments should be avoided"
* Cleanup: move some define back to AES-NI code as not used by AES-KL
Changes from v5:
* Clean up the staled function definition -- cbc_crypt_common().
* Ensure kernel_fpu_end() for the possible error return from xts_crypt_common()->crypt1_fn().
Changes from v4:
* Drop CBC mode changes. (Eric Biggers)
Changes from v3:
* Drop ECB and CTR mode changes. (Eric Biggers)
* Export symbols. (Eric Biggers)
Changes from RFC v2:
* Massage the changelog. (Dan Williams)
Changes from RFC v1:
* Added as a new patch. (Ard Biesheuvel)
--- arch/x86/crypto/aes-helper_asm.S | 22 +++ arch/x86/crypto/aes-helper_glue.h | 167 ++++++++++++++++++++++ arch/x86/crypto/aesni-intel_asm.S | 47 +++---- arch/x86/crypto/aesni-intel_glue.c | 213 ++++++++--------------------- 4 files changed, 261 insertions(+), 188 deletions(-) create mode 100644 arch/x86/crypto/aes-helper_asm.S create mode 100644 arch/x86/crypto/aes-helper_glue.h
arch/x86/crypto/aes-helper_asm.S new file +22 -0
@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +/* + * Constant values shared between AES implementations: + */ + +.pushsection .rodata +.align 16 +.Lcts_permute_table: + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 +.popsection + +.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 +.align 16 +.Lgf128mul_x_ble_mask: + .octa 0x00000000000000010000000000000087 +.previous
arch/x86/crypto/aes-helper_glue.h new file +167 -0
167 lines not on this page · raw message
arch/x86/crypto/aesni-intel_asm.S +17 -30
116 lines not on this page · raw message
arch/x86/crypto/aesni-intel_glue.c +55 -158
321 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v9 13/14] crypto: x86/aes - Prepare for new AES-XTS implementation
This patch is no longer needed, as the rework leads more destructive code changes in the AES-NI side.
Thanks,
Chang
PATCH v9 14/14 crypto: x86/aes-kl - Implement the AES-XTS algorithm
In reply to Chang S. BaeKey Locker is a CPU feature to reduce key exfiltration opportunities. It converts the AES key into an encoded form, called 'key handle', to reduce the exposure of private key material in memory.
This key conversion as well as all subsequent data transformations are provided by new AES instructions ('AES-KL'). AES-KL is analogous to that of AES-NI as maintains a similar programming interface.
Support the XTS mode as the primary use case is dm-crypt. The support has some details worth mentioning, which differentiate itself from AES-NI, that users may need to be aware of:
== Key Handle Restriction ==
The AES-KL instruction set supports selecting key usage restrictions at key handle creation time. Restrict all key handles created by the kernel to kernel mode use only.
The AES-KL instructions themselves are executable in userspace. This restriction enforces the mode consistency in its operation.
If the key handle is created in userspace but referenced in the kernel, then encrypt() and decrypt() functions will return -EINVAL.
=== AES-NI Dependency for AES Compliance ===
Key Locker is not AES compliant as it lacks 192-bit key support. However, per the expectations of Linux crypto-cipher implementations, the software cipher implementation must support all the AES-compliant key sizes.
The AES-KL cipher implementation achieves this constraint by logging a warning and falling back to AES-NI. In other words, the 192-bit key-size limitation for what can be converted into a key handle is only documented, not enforced.
This then creates a rather strong dependency on AES-NI. If this driver supports a module build, the exported AES-NI functions cannot be inlined. More importantly, indirect calls will impact the performance.
To simplify, disallow a module build for AES-KL and always select AES-NI. This restriction can be relaxed if strong use cases arise against it.
== Wrapping Key Restore Failure Handling ==
In the event of hardware failure, the wrapping key is lost from deep sleep states. Then, the wrapping key turns to zero which is an unusable state.
The x86 core provides valid_keylocker() to indicate the failure. Subsequent setkey() as well as encode()/decode() can check it and return -ENODEV if failed. In this way, an error code can be returned, instead of facing abrupt exceptions.
== Userspace Exposition ==
The Keylocker implementations so far have measurable performance penalties. So, keep AES-NI as the default.
However, with a slow storage device, storage bandwidth is the bottleneck, even if disk encryption is enabled by AES-KL. Thus, it is an end-user consideration for selecting AES-KL. Users may pick it according to the name 'xts-aes-aeskl' shown in /proc/crypto.
== 64-bit Only ==
Support 64-bit only, as the 32-bit kernel is being deprecated.
---
Changes from v8:
* Rebase on the upstream changes.
* Combine the XTS enc/dec assembly code in a macro. (Eric Biggers)
* Define setkey() as void instead of returning 'int'. (Eric Biggers)
* Rearrange the assembly code to reduce jumps especially for success cases. (Eric Biggers)
* Update the changelog for clarification. (Eric Biggers)
* Exclude module build.
Changes from v7:
* Update the changelog -- remove 'API Limitation'. (Eric Biggers)
* Update the comment for valid_keylocker(). (Eric Biggers)
* Improve the code:
- Remove the key-length check and simplify the code. (Eric Biggers)
- Remove aeskl_dec() and __aeskl_dec() as not needed.
- Simplify the register-function return handling. (Eric Biggers)
- Rename setkey functions for coherent naming:
aeskl_setkey() -> __aeskl_setkey(),
aeskl_setkey_common() -> aeskl_setkey(),
aeskl_xts_setkey() -> xts_setkey()- Revert an unnecessary comment.
Changes from v6:
* Merge all the AES-KL patches. (Eric Biggers)
* Make the driver for the 64-bit mode only. (Eric Biggers)
* Rework the key-size check code:
- Trim unnecessary checks. (Eric Biggers)
- Document the reason
- Make sure both XTS keys with the same size
* Adjust the Kconfig change:
- Move the location. (Robert Elliott)
- Trim the description to follow others such as AES-NI.
* Update the changelog:
- Explain the priority value for the common name under 'User Exposition' (renamed from 'Performance'). (Eric Biggers)
- Trim the introduction
- Switch to more imperative mood for those explaining the code change
- Add a new section '64-bit Only'
* Adjust the ASM code to return a proper error code. (Eric Biggers)
* Update assembly code macros:
- Remove unused one.
- Document the reason for the duplicated ones.
Changes from v5:
* Replace the ret instruction with RET as rebased on the upstream -- commit f94909ceb1ed ("x86: Prepare asm files for straight-line-speculation").
Changes from v3:
* Exclude non-AES-KL objects. (Eric Biggers)
* Simplify the assembler dependency check. (Peter Zijlstra)
* Trim the Kconfig help text. (Dan Williams)
* Fix a defined-but-not-used warning.
Changes from RFC v2:
* Move out each mode support in new patches.
* Update the changelog to describe the limitation and the tradeoff clearly. (Andy Lutomirski)
Changes from RFC v1:
* Rebased on the refactored code. (Ard Biesheuvel)
* Dropped exporting the single block interface. (Ard Biesheuvel)
* Fixed the fallback and error handling paths. (Ard Biesheuvel)
* Revised the module description. (Dave Hansen and Peter Zijlstra)
* Made the build depend on the binutils version to support new instructions. (Borislav Petkov and Peter Zijlstra)
* Updated the changelog accordingly.
--- arch/x86/Kconfig.assembler | 5 + arch/x86/crypto/Kconfig | 17 ++ arch/x86/crypto/Makefile | 3 + arch/x86/crypto/aes-helper_glue.h | 7 +- arch/x86/crypto/aeskl-intel_asm.S | 412 +++++++++++++++++++++++++++++ arch/x86/crypto/aeskl-intel_glue.c | 187 +++++++++++++ arch/x86/crypto/aeskl-intel_glue.h | 35 +++ arch/x86/crypto/aesni-intel_glue.c | 30 +-- arch/x86/crypto/aesni-intel_glue.h | 40 +++ 9 files changed, 704 insertions(+), 32 deletions(-) create mode 100644 arch/x86/crypto/aeskl-intel_asm.S create mode 100644 arch/x86/crypto/aeskl-intel_glue.c create mode 100644 arch/x86/crypto/aeskl-intel_glue.h create mode 100644 arch/x86/crypto/aesni-intel_glue.h
arch/x86/Kconfig.assembler +5 -0
@@ -25,6 +25,11 @@ config AS_GFNI help Supported by binutils >= 2.30 and LLVM integrated assembler +config AS_HAS_KEYLOCKER + def_bool $(as-instr,encodekey256 %eax$(comma)%eax) + help + Supported by binutils >= 2.36 and LLVM integrated assembler >= V12 + config AS_WRUSS def_bool $(as-instr,wrussq %rax$(comma)(%rbx)) help
arch/x86/crypto/Kconfig +17 -0
@@ -29,6 +29,23 @@ config CRYPTO_AES_NI_INTEL Architecture: x86 (32-bit and 64-bit) using: - AES-NI (AES new instructions) +config CRYPTO_AES_KL + bool "Ciphers: AES, modes: XTS (AES-KL)" + depends on X86 && 64BIT + depends on AS_HAS_KEYLOCKER + select CRYPTO_AES_NI_INTEL + select X86_KEYLOCKER + + help + Block cipher: AES cipher algorithms + Length-preserving ciphers: AES with XTS + + Architecture: x86 (64-bit) using: + - AES-KL (AES Key Locker) + - AES-NI for a 192-bit key + + See Documentation/arch/x86/keylocker.rst for more details. + config CRYPTO_BLOWFISH_X86_64 tristate "Ciphers: Blowfish, modes: ECB, CBC" depends on X86 && 64BIT
arch/x86/crypto/Makefile +3 -0
@@ -50,6 +50,9 @@ obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o +obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-intel.o +aeskl-intel-y := aeskl-intel_asm.o aeskl-intel_glue.o + obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
arch/x86/crypto/aes-helper_glue.h +4 -3
@@ -19,16 +19,17 @@#include <crypto/internal/aead.h> #include <crypto/internal/simd.h> +#include "aeskl-intel_glue.h" +#define AES_ALIGN 16#define AES_ALIGN_ATTR __attribute__((__aligned__(AES_ALIGN)))#define AES_ALIGN_EXTRA ((AES_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1))#define XTS_AES_CTX_SIZE (sizeof(struct aes_xts_ctx) + AES_ALIGN_EXTRA) -/* -* Preserve data types for various AES implementations available in x86 -*/ +/* Data types for the two AES implementations available in x86 */ union x86_aes_ctx {struct crypto_aes_ctx aesni; +struct aeskl_ctx aeskl; };struct aes_xts_ctx {
arch/x86/crypto/aeskl-intel_asm.S new file +412 -0
412 lines not on this page · raw message
arch/x86/crypto/aeskl-intel_glue.c new file +187 -0
187 lines not on this page · raw message
arch/x86/crypto/aeskl-intel_glue.h new file +35 -0
@@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _AESKL_INTEL_GLUE_H +#define _AESKL_INTEL_GLUE_H + +#include <crypto/aes.h> +#include <linux/types.h> + +#define AESKL_AAD_SIZE 16 +#define AESKL_TAG_SIZE 16 +#define AESKL_CIPHERTEXT_MAX AES_KEYSIZE_256 + +/* The Key Locker handle is an encoded form of the AES key. */ +struct aeskl_handle { +u8 additional_authdata[AESKL_AAD_SIZE ]; +u8 integrity_tag[AESKL_TAG_SIZE ]; +u8 ciphre_text[AESKL_CIPHERTEXT_MAX ]; +}; + +/* +* Key Locker does not support 192-bit key size. The driver needs to +* retrieve the key size in the first place. The offset of the +* 'key_length' field here should be compatible with struct +* crypto_aes_ctx. +*/ +#define AESKL_CTX_RESERVED (sizeof(struct crypto_aes_ctx) - sizeof(struct aeskl_handle) \ + - sizeof(u32)) + +struct aeskl_ctx { +struct aeskl_handle handle; +u8 reserved[AESKL_CTX_RESERVED ]; +u32 key_length; +}; + +#endif /* _AESKL_INTEL_GLUE_H */
arch/x86/crypto/aesni-intel_glue.c +1 -29
@@ -37,6 +37,7 @@#include <linux/static_call.h> #include "aes-helper_glue.h" +#include "aesni-intel_glue.h" #define RFC4106_HASH_SUBKEY_SIZE 16#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1))
@@ -72,9 +73,6 @@ struct gcm_context_data {u8 hash_keys[GCM_BLOCK_LEN *16 ]; }; -asmlinkage void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, -unsigned int key_len); -asmlinkage void __aesni_enc(const void *ctx, u8 *out, const u8 *in); asmlinkage void __aesni_dec(const void *ctx, u8 *out, const u8 *in); asmlinkage void aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out,const u8 *in,unsigned int len);
@@ -89,21 +87,9 @@ asmlinkage void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, asmlinkage void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out,const u8 *in,unsigned int len,u8 *iv); -static inline int aesni_enc (const void *ctx,u8 *out,const u8 *in) -{ -__aesni_enc (ctx, out, in); -return 0 ; -} -#define AVX_GEN2_OPTSIZE 640#define AVX_GEN4_OPTSIZE 4096 -asmlinkage void __aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, -const u8 *in,unsigned int len,u8 *iv); - -asmlinkage void __aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, -const u8 *in,unsigned int len,u8 *iv); -#ifdef CONFIG_X86_64 asmlinkage void aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out,
@@ -271,20 +257,6 @@ static inline int aesni_xts_setkey(union x86_aes_ctx *ctx,return aes_set_key_common (&ctx->aesni, in_key, key_len); } -static inline int aesni_xts_encrypt (const union x86_aes_ctx *ctx,u8 *out,const u8 *in, -unsigned int len,u8 *iv) -{ -__aesni_xts_encrypt (&ctx->aesni, out, in, len, iv); -return 0 ; -} - -static inline int aesni_xts_decrypt (const union x86_aes_ctx *ctx,u8 *out,const u8 *in, -unsigned int len,u8 *iv) -{ -__aesni_xts_decrypt (&ctx->aesni, out, in, len, iv); -return 0 ; -} -static int aesni_skcipher_setkey (struct crypto_skcipher *tfm,const u8 *key,unsigned int len) {
arch/x86/crypto/aesni-intel_glue.h new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* +* These are AES-NI functions that are used by the AES-KL code as a +* fallback when it is given a 192-bit key. Key Locker does not support +* 192-bit keys. +*/ + +#ifndef _AESNI_INTEL_GLUE_H +#define _AESNI_INTEL_GLUE_H + +asmlinkage void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, +unsigned int key_len); +asmlinkage void __aesni_enc(const void *ctx, u8 *out, const u8 *in); +asmlinkage void __aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, +const u8 *in,unsigned int len,u8 *iv); +asmlinkage void __aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, +const u8 *in,unsigned int len,u8 *iv); + +static inline int aesni_enc (const void *ctx,u8 *out,const u8 *in) +{ +__aesni_enc (ctx, out, in); +return 0 ; +} + +static inline int aesni_xts_encrypt (const union x86_aes_ctx *ctx,u8 *out,const u8 *in, +unsigned int len,u8 *iv) +{ +__aesni_xts_encrypt (&ctx->aesni, out, in, len, iv); +return 0 ; +} + +static inline int aesni_xts_decrypt (const union x86_aes_ctx *ctx,u8 *out,const u8 *in, +unsigned int len,u8 *iv) +{ +__aesni_xts_decrypt (&ctx->aesni, out, in, len, iv); +return 0 ; +} + +#endif /* _AESNI_INTEL_GLUE_H */
Signature · 1 line
2.34.1
PATCH v9 14/14 crypto: x86/aes-kl - Implement the AES-XTS algorithm
Key Locker is a CPU feature to reduce key exfiltration opportunities. It converts the AES key into an encoded form, called 'key handle', to reduce the exposure of private key material in memory.
This key conversion along with all subsequent data transformations, is provided by new AES instructions ('AES-KL'). AES-KL is analogous to that of AES-NI as maintains a similar programming interface.
Support the XTS mode as the primary use case is dm-crypt. The support has some details worth mentioning, which differentiate itself from AES-NI, that users may need to be aware of:
== Key Handle Restriction ==
The AES-KL instruction set supports selecting key usage restrictions at key handle creation time. Restrict all key handles created by the kernel to kernel mode use only.
Although the AES-KL instructions themselves are executable in userspace, this restriction enforces the mode consistency in its operation.
If the key handle is created in userspace but referenced in the kernel, then encrypt() and decrypt() functions will return -EINVAL.
=== AES-NI Dependency for AES Compliance ===
Key Locker is not AES compliant as it lacks 192-bit key support. However, per the expectations of Linux crypto-cipher implementations, the software cipher implementation must support all the AES-compliant key sizes.
The AES-KL cipher implementation achieves this constraint by logging a warning and falling back to VAES. In other words, the 192-bit key-size limitation is documented but not enforced.
== Wrapping Key Restore Failure Handling ==
In the event of hardware failure, the wrapping key is lost from deep sleep states. Then, the wrapping key turns to zero which is an unusable state.
The x86 core provides valid_keylocker() to indicate the failure. Subsequent setkey() as well as encode()/decode() can check it and return -ENODEV if failed. This allows an error code to be returned instead of encountering abrupt exceptions.
== Userspace Exposition ==
Keylocker implementations have measurable performance penalties. Therefore, keep the current default remains the same.
However, with a slow storage device, storage bandwidth is the bottleneck, even if disk encryption is enabled by AES-KL. Thus, it is up to the end user to decide whether to use AES-KL. User can select it by the name 'xts-aes-aeskl' shown in /proc/crypto.
== 64-bit Only ==
Support 64-bit only, as the 32-bit kernel is being deprecated.
---
I've reworked this patch based on feedback,
https://lore.kernel.org/lkml/[email protected]/
and rebased to upstream v6.10 Linus merge tree on May 13th: commit 84c7d76b5ab6 ("Merge tag 'v6.10-p1' ofgit://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6")
According to the dm-crypt benchmark, using VEX-encoded instructions for tweak processing enhances performance by approximately 2-3%. The PCLMULDQD instruction did not yield a measurable impact, so I dropped it to simplify the implementation.
In contrast to other AES instructions, AES-KL does not permit tweak processing between rounds. In XTS mode, a single instruction covers all rounds of 8 blocks without interleaving instructions. Maybe this is one of the reasons for the limited performance gain.
Moving forward, I would like to address any further feedback on this AES-KL driver code first before the next revision of the whole series.
Changes from v9:
* Duplicate the new XTS glue code, instead of sharing (Eric).
* Use VEX-coded instructions for non-AES parts of the code (Eric).
* Adjust ASM code to stylistically follow the new VAES support (Eric).
* Export and reference the high-level AES-NI XTS functions (Eric). Then, support a module build, along with rearranging build dependencies.
* Reorganize the glue code and improve ASM code readability.
* Revoke the review tag due to major changes.
--- arch/x86/Kconfig.assembler | 5 + arch/x86/crypto/Kconfig | 18 ++ arch/x86/crypto/Makefile | 3 + arch/x86/crypto/aeskl-xts-x86_64.S | 358 +++++++++++++++++++++++++++ arch/x86/crypto/aeskl_glue.c | 376 +++++++++++++++++++++++++++++ arch/x86/crypto/aesni-intel_glue.c | 13 +- arch/x86/crypto/aesni-xts.h | 15 ++ 7 files changed, 783 insertions(+), 5 deletions(-) create mode 100644 arch/x86/crypto/aeskl-xts-x86_64.S create mode 100644 arch/x86/crypto/aeskl_glue.c create mode 100644 arch/x86/crypto/aesni-xts.h
arch/x86/Kconfig.assembler +5 -0
@@ -35,6 +35,11 @@ config AS_VPCLMULQDQ help Supported by binutils >= 2.30 and LLVM integrated assembler +config AS_HAS_KEYLOCKER + def_bool $(as-instr,encodekey256 %eax$(comma)%eax) + help + Supported by binutils >= 2.36 and LLVM integrated assembler >= V12 + config AS_WRUSS def_bool $(as-instr,wrussq %rax$(comma)(%rbx)) help
arch/x86/crypto/Kconfig +18 -0
@@ -29,6 +29,24 @@ config CRYPTO_AES_NI_INTEL Architecture: x86 (32-bit and 64-bit) using: - AES-NI (AES new instructions) +config CRYPTO_AES_KL + tristate "Ciphers: AES, modes: XTS (AES-KL)" + depends on X86 && 64BIT + depends on AS_HAS_KEYLOCKER + select CRYPTO_AES_NI_INTEL + select CRYPTO_SIMD + select X86_KEYLOCKER + + help + Block cipher: AES cipher algorithms + Length-preserving ciphers: AES with XTS + + Architecture: x86 (64-bit) using: + - AES-KL (AES Key Locker) + - AES-NI for a 192-bit key + + See Documentation/arch/x86/keylocker.rst for more details. + config CRYPTO_BLOWFISH_X86_64 tristate "Ciphers: Blowfish, modes: ECB, CBC" depends on X86 && 64BIT
arch/x86/crypto/Makefile +3 -0
@@ -51,6 +51,9 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o \ aes_ctrby8_avx-x86_64.o aes-xts-avx-x86_64.o +obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-x86_64.o +aeskl-x86_64-y := aeskl-xts-x86_64.o aeskl_glue.o + obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
arch/x86/crypto/aeskl-xts-x86_64.S new file +358 -0
358 lines not on this page · raw message
arch/x86/crypto/aeskl_glue.c new file +376 -0
376 lines not on this page · raw message
arch/x86/crypto/aesni-intel_glue.c +8 -5
@@ -35,7 +35,7 @@#include <linux/workqueue.h> #include <linux/spinlock.h> #include <linux/static_call.h> - +#include "aesni-xts.h" #define AESNI_ALIGN 16#define AESNI_ALIGN_ATTR __attribute__ ((__aligned__(AESNI_ALIGN)))
@@ -864,8 +864,8 @@ static int helper_rfc4106_decrypt(struct aead_request *req) }#endif -static int xts_setkey_aesni (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keylen) +int xts_setkey_aesni (struct crypto_skcipher *tfm,const u8 *key, +unsigned int keylen) {struct aesni_xts_ctx *ctx =aes_xts_ctx (tfm);int err;
@@ -884,6 +884,7 @@ static int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key,/* second half of xts-key is for tweak */ return aes_set_key_common (&ctx->tweak_ctx, key + keylen, keylen); } +EXPORT_SYMBOL_GPL (xts_setkey_aesni);typedef void (*xts_encrypt_iv_func )(const struct crypto_aes_ctx *tweak_key,u8 iv[AES_BLOCK_SIZE ]);
@@ -1020,15 +1021,17 @@ static void aesni_xts_decrypt(const struct crypto_aes_ctx *key,aesni_xts_dec (key, dst, src, len, tweak); } -static int xts_encrypt_aesni (struct skcipher_request *req) +int xts_encrypt_aesni (struct skcipher_request *req) {return xts_crypt (req, aesni_xts_encrypt_iv, aesni_xts_encrypt); } +EXPORT_SYMBOL_GPL (xts_encrypt_aesni); -static int xts_decrypt_aesni (struct skcipher_request *req) +int xts_decrypt_aesni (struct skcipher_request *req) {return xts_crypt (req, aesni_xts_encrypt_iv, aesni_xts_decrypt); } +EXPORT_SYMBOL_GPL (xts_decrypt_aesni);static struct crypto_alg aesni_cipher_alg = { .cra_name ="aes" ,
arch/x86/crypto/aesni-xts.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _AESNI_XTS_H +#define _AESNI_XTS_H + +/* +* These AES-NI functions are used by the AES-KL code as a fallback when +* a 192-bit key is provided. Key Locker does not support 192-bit keys. +*/ + +int xts_setkey_aesni (struct crypto_skcipher *tfm,const u8 *key,unsigned int keylen); +int xts_encrypt_aesni (struct skcipher_request *req); +int xts_decrypt_aesni (struct skcipher_request *req); + +#endif /* _AESNI_XTS_H */
Signature · 1 line
2.34.1
Re: [PATCH v9a 14/14] crypto: x86/aes-kl - Implement the AES-XTS algorithm
Quoted 19 lines · Chang S. Bae
I've reworked this patch based on feedback,
https://lore.kernel.org/lkml/[email protected]/and rebased to upstream v6.10 Linus merge tree on May 13th: commit
84c7d76b5ab6("Merge tag 'v6.10-p1' ofgit://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6")According to the dm-crypt benchmark, using VEX-encoded instructions for tweak processing enhances performance by approximately 2-3%. The PCLMULDQD instruction did not yield a measurable impact, so I dropped it to simplify the implementation.
In contrast to other AES instructions, AES-KL does not permit tweak processing between rounds. In XTS mode, a single instruction covers all rounds of 8 blocks without interleaving instructions. Maybe this is one of the reasons for the limited performance gain.
Moving forward, I would like to address any further feedback on this AES-KL driver code first before the next revision of the whole series.
Quoted 9 lines
Changes from v9:
* Duplicate the new XTS glue code, instead of sharing (Eric).
* Use VEX-coded instructions for non-AES parts of the code (Eric).
* Adjust ASM code to stylistically follow the new VAES support (Eric).
* Export and reference the high-level AES-NI XTS functions (Eric). Then, support a module build, along with rearranging build dependencies.
* Reorganize the glue code and improve ASM code readability.
* Revoke the review tag due to major changes.
---
Thanks for the updated patch!
arch/x86/Kconfig.assembler config AS_VPCLMULQDQ · line 41
Quoted 12 lines
arch/x86/Kconfig.assembler +4
-0@@ -35,6 +35,11 @@ config AS_VPCLMULQDQ help Supported by binutils >= 2.30 and LLVM integrated assembler +config AS_HAS_KEYLOCKER + def_bool $(as-instr,encodekey256 %eax$(comma)%eax) + help + Supported by binutils >= 2.36 and LLVM integrated assembler >= V12
Adding AS_HAS_KEYLOCKER should be its own patch.
Quoted 52 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +46
-0@@ -0,0 +1,358 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Implement AES algorithm using AES Key Locker instructions. + * + * Most code is primarily derived from aesni-intel_asm.S and + * stylistically aligned with aes-xts-avx-x86_64.S. + */ + +#include <linux/linkage.h> +#include <linux/cfi_types.h> +#include <asm/errno.h> +#include <asm/inst.h> +#include <asm/frame.h> + +/* Constant values shared between AES implementations: */ + +.section .rodata +.p2align 4 +.Lgf_poly: + /* + * Represents the polynomial x^7 + x^2 + x + 1, where the low 64 + * bits are XOR'd into the tweak's low 64 bits when a carry + * occurs from the high 64 bits. + */ + .quad 0x87, 1 + + /* + * Table of constants for variable byte shifts and blending + * during ciphertext stealing operations. + */ +.Lcts_permute_table: + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + +.text + +.set V0, %xmm0 +.set V1, %xmm1 +.set V2, %xmm2 +.set V3, %xmm3 +.set V4, %xmm4 +.set V5, %xmm5
arch/x86/crypto/aeskl-xts-x86_64.S line 56
Quoted 10 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +10
-0+.set V6, %xmm6 +.set V7, %xmm7 +.set V8, %xmm8 +.set V9, %xmm9 +.set V10, %xmm10 +.set V11, %xmm11 +.set V12, %xmm12 +.set V13, %xmm13 +.set V14, %xmm14 +.set V15, %xmm15
The point of the V[0-15] aliases in aes-xts-avx-x86_64.S are to support both ymm and zmm registers. Here all registers are xmm, so there's no need for the layer of indirection and you should just use xmm[0-15] directly.
arch/x86/crypto/aeskl-xts-x86_64.S line 68
Quoted 11 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +11
-0+.set TWEAK_XMM1, V8 +.set TWEAK_XMM2, V9 +.set TWEAK_XMM3, V10 +.set TWEAK_XMM4, V11 +.set TWEAK_XMM5, V12 +.set TWEAK_XMM6, V13 +.set TWEAK_XMM7, V14 +.set GF_POLY_XMM, V15 +.set TWEAK_TMP, TWEAK_XMM1 +.set TWEAK_XMM, TWEAK_XMM2 +.set TMP, %r10
Similarly, the _XMM suffixes are not really helpful since ymm and zmm registers are not in play here.
arch/x86/crypto/aeskl-xts-x86_64.S line 77
Quoted 8 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +8
-0+/* Function parameters */ +.set HANDLEP, %rdi /* Pointer to struct aeskl_ctx */ +.set DST, %rsi /* Pointer to next destination data */ +.set UKEYP, DST /* Pointer to the original key */ +.set KLEN, %r9d /* AES key length in bytes */ +.set SRC, %rdx /* Pointer to next source data */ +.set LEN, %rcx /* Remaining length in bytes */ +.set TWEAK, %r8 /* Pointer to next tweak */
Please don't put parameters for different functions in the same list like this. There should be a separate list at the beginning of each function. (Yes, it doesn't work perfectly because '.set' is global and doesn't go out of scope once the function ends. But at least this would make it clear what the intent is.)
Also LEN needs to be %ecx, not %rcx, because it is unsigned int.
Quoted 13 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +13
-0+SYM_FUNC_START(__aeskl_setkey) + FRAME_BEGIN + movl %edx, 480(HANDLEP) + vmovdqu (UKEYP), V0 + mov $1, %eax + cmp $16, %dl + je .Lsetkey_128 + + vmovdqu 0x10(UKEYP), V1 + encodekey256 %eax, %eax + vmovdqu V3, 0x30(HANDLEP) + jmp .Lsetkey_end +.Lsetkey_128:
arch/x86/crypto/aeskl-xts-x86_64.S line 105
Quoted 10 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +10
-0+ encodekey128 %eax, %eax + +.Lsetkey_end: + vmovdqu V0, 0x00(HANDLEP) + vmovdqu V1, 0x10(HANDLEP) + vmovdqu V2, 0x20(HANDLEP) + + FRAME_END + RET +SYM_FUNC_END(__aeskl_setkey)
These are all leaf functions, so they don't need FRAME_BEGIN and FRAME_END.
Quoted 23 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +23
-0+.macro _aeskl width, operation + cmp $16, KLEN + je .Laeskl128\@ +.ifc \width, wide + .ifc \operation, dec + aesdecwide256kl (HANDLEP) + .else + aesencwide256kl (HANDLEP) + .endif +.else + .ifc \operation, dec + aesdec256kl (HANDLEP), V0 + .else + aesenc256kl (HANDLEP), V0 + .endif +.endif + jmp .Laesklend\@ +.Laeskl128\@: +.ifc \width, wide + .ifc \operation, dec + aesdecwide128kl (HANDLEP) + .else + aesencwide128kl (HANDLEP)
arch/x86/crypto/aeskl-xts-x86_64.S line 139
Quoted 10 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +10
-0+ .endif +.else + .ifc \operation, dec + aesdec128kl (HANDLEP), V0 + .else + aesenc128kl (HANDLEP), V0 + .endif +.endif +.Laesklend\@: +.endm
I think it would be easier to read if this was split into two macros, one for 1-block and one for 8-block.
Quoted 4 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +4
-0+/* int __aeskl_enc(const void *handlep, u8 *dst, const u8 *src) */ +SYM_FUNC_START(__aeskl_enc) + FRAME_BEGIN + vmovdqu (SRC), V0
arch/x86/crypto/aeskl-xts-x86_64.S line 154
Quoted 10 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +10
-0+ movl 480(HANDLEP), KLEN + + _aeskl oneblock, enc + jz .Lerror + xor %rax, %rax + vmovdqu V0, (DST) + FRAME_END + RET +.Lerror: + mov $(-EINVAL), %rax
For returning an int, %eax should be used, not %rax.
(Note that instructions that operate on %eax also tend to be slightly shorter.)
arch/x86/crypto/aeskl-xts-x86_64.S line 354
Quoted 12 lines
arch/x86/crypto/aeskl-xts-x86_64.S new file +12
-0+/* + * int __aeskl_xts_encrypt(const struct aeskl_ctx *handlep, u8 *dst, + * const u8 *src, unsigned int klen, le128 *tweak) + */ +SYM_FUNC_START(__aeskl_xts_encrypt) + _aeskl_xts_crypt enc +SYM_FUNC_END(__aeskl_xts_encrypt) + +/* + * int __aeskl_xts_decrypt(const struct crypto_aes_ctx *handlep, u8 *dst, + * const u8 *src, unsigned int klen, le128 *twek) + */
Please make sure the function prototypes, including the parameter names, match the ones used in the .c file and also the lists of register aliases.
Quoted 33 lines
arch/x86/crypto/aeskl_glue.c new file +27
-0@@ -0,0 +1,376 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* Support for AES Key Locker instructions. This file contains glue +* code and the real AES implementation is in aeskl-intel_asm.S. +* +* Most code is based on aesni-intel_glue.c +*/ + +#include <linux/types.h> +#include <linux/module.h> +#include <linux/err.h> +#include <crypto/algapi.h> +#include <crypto/aes.h> +#include <crypto/xts.h> +#include <crypto/scatterwalk.h> +#include <crypto/internal/skcipher.h> +#include <crypto/internal/simd.h> +#include <asm/simd.h> +#include <asm/cpu_device_id.h> +#include <asm/fpu/api.h> +#include <asm/keylocker.h> +#include "aesni-xts.h" + +#define AESKL_ALIGN 16 +#define AESKL_ALIGN_ATTR __attribute__ ((__aligned__(AESKL_ALIGN))) +#define AESKL_ALIGN_EXTRA ((AESKL_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1)) +
arch/x86/crypto/aeskl_glue.c line 37
Quoted 10 lines
arch/x86/crypto/aeskl_glue.c new file +10
-0+#define AESKL_AAD_SIZE 16 +#define AESKL_TAG_SIZE 16 +#define AESKL_CIPHERTEXT_MAX AES_KEYSIZE_256 + +/* The Key Locker handle is an encoded form of the AES key. */ +struct aeskl_handle { +u8 additional_authdata[AESKL_AAD_SIZE ]; +u8 integrity_tag[AESKL_TAG_SIZE ]; +u8 ciphre_text[AESKL_CIPHERTEXT_MAX ]; +};
ciphre_text => ciphertext
arch/x86/crypto/aeskl_glue.c line 42
Quoted 4 lines
arch/x86/crypto/aeskl_glue.c new file +4
-0+/* +* Key Locker does not support 192-bit key size. The driver needs to +* retrieve the key size in the first place. The offset of the +* 'key_length' field here should be compatible with struct
should => must
Quoted 5 lines
arch/x86/crypto/aeskl_glue.c new file +5
-0+* crypto_aes_ctx. +*/ +#define AESKL_CTX_RESERVED (sizeof(struct crypto_aes_ctx) - sizeof(struct aeskl_handle) \ + - sizeof(u32)) +
arch/x86/crypto/aeskl_glue.c line 57
Quoted 10 lines
arch/x86/crypto/aeskl_glue.c new file +10
-0+struct aeskl_ctx { +struct aeskl_handle handle; +u8 reserved[AESKL_CTX_RESERVED ]; +u32 key_length; +}; + +struct aeskl_xts_ctx { + struct aeskl_ctx tweak_ctx AESKL_ALIGN_ATTR; + struct aeskl_ctx crypt_ctx AESKL_ALIGN_ATTR; +};
So there's a union between aeskl_ctx and crypto_aes_ctx going on here, but it's not made explicit through a C union. How about doing that?
Also, there should be a BUILD_BUG_ON() that enforces that the key_length is really at the same offset in both.
arch/x86/crypto/aeskl_glue.c line 152
Quoted 12 lines
arch/x86/crypto/aeskl_glue.c new file +12
-0+/* +* The glue code in xts_crypt() and xts_crypt_slowpath() follows +* aesni-intel_glue.c. While this code is shareable, the key +* material format difference can cause more destructive code changes in +* the AES-NI side. +*/ + +typedef int (*xts_encrypt_iv_func )(const struct aeskl_ctx *tweak_key, +u8 iv[AES_BLOCK_SIZE ]); +typedef int (*xts_crypt_func )(const struct aeskl_ctx *key, +const u8 *src,u8 *dst,unsigned int len, +u8 tweak[AES_BLOCK_SIZE ]);
Since there are so few functions in play here (one xts_encrypt_iv_func and two xts_crypt_func) I think you should just use direct calls instead of function pointers. A simple parameter 'bool enc' would take care of selecting the encryption function vs. the decryption one.
One of the issues with indirect calls, even when inlined with the intention that they be optimized out, is that there's no guarantee that the compiler will actually optimize them out. That has the consequence that CFI stubs are still needed in the assembly.
Direct calls avoid this issue.
(BTW, in my AES-GCM patchset I'm using direct calls:
https://lore.kernel.org/linux-crypto/[email protected]/.
I'm thinking I should have used that approach with AES-XTS too.)
arch/x86/crypto/aeskl_glue.c line 320
Quoted 6 lines
arch/x86/crypto/aeskl_glue.c new file +6
-0+static struct skcipher_alg aeskl_skciphers[] = { + { + .base = { + .cra_name ="__xts(aes)" , + .cra_driver_name ="__xts-aes-aeskl" , + .cra_priority =200 ,
Maybe add a comment here that explains that this is intentionally made lower priority than xts-aes-aesni.
Quoted 8 lines
arch/x86/crypto/aeskl_glue.c new file +8
-0+static int __init aeskl_init (void ) +{ +u32 eax, ebx, ecx, edx; + +if (!valid_keylocker ()) +return -ENODEV ; + +cpuid_count (KEYLOCKER_CPUID ,0 , &eax, &ebx, &ecx, &edx);
arch/x86/crypto/aeskl_glue.c line 355
Quoted 10 lines
arch/x86/crypto/aeskl_glue.c new file +10
-0+if (!(ebx &KEYLOCKER_CPUID_EBX_WIDE )) +return -ENODEV ; + +/* +* AES-KL itself does not rely on AES-NI. But, AES-KL does not +* support 192-bit keys. To ensure AES compliance, AES-KL falls +* back to AES-NI. +*/ +if (!cpu_feature_enabled (X86_FEATURE_AES )) +return -ENODEV ;
Everywhere else in arch/x86/crypto/ uses boot_cpu_has(), not cpu_feature_enabled().
arch/x86/crypto/aeskl_glue.c line 359
Quoted 4 lines
arch/x86/crypto/aeskl_glue.c new file +4
-0+ +/* The tweak processing is optimized using AVX instructions. */ +if (!cpu_feature_enabled (X86_FEATURE_AVX )) +return -ENODEV ;
The whole implementation uses VEX-coded instructions now, not just the tweak processing. So either fix or delete the above comment.
- Eric
PATCH v9 14/14 crypto: x86/aes-kl - Implement the AES-XTS algorithm
Hi Eric,
I really appreciate your review. Keeping track of about 800 lines of code, including assembly lines, in a single patch is demanding. I hope this version meets your expectations.
The overall diff can be found here:
https://github.com/intel-staging/keylocker/compare/f8420d4e27fc..57472d9b3f8e
Thanks,
Chang
---
Key Locker is a CPU feature to reduce key exfiltration opportunities. It converts the AES key into an encoded form, called 'key handle', to reduce the exposure of private key material in memory.
This key conversion along with all subsequent data transformations, is provided by new AES instructions ('AES-KL'). AES-KL is analogous to that of AES-NI as maintains a similar programming interface.
Support the XTS mode as the primary use case is dm-crypt. The support has some details worth mentioning, which differentiate itself from AES-NI, that users may need to be aware of:
== Key Handle Restriction ==
The AES-KL instruction set supports selecting key usage restrictions at key handle creation time. Restrict all key handles created by the kernel to kernel mode use only.
Although the AES-KL instructions themselves are executable in userspace, this restriction enforces the mode consistency in its operation.
If the key handle is created in userspace but referenced in the kernel, then encrypt() and decrypt() functions will return -EINVAL.
=== AES-NI Dependency for AES Compliance ===
Key Locker is not AES compliant as it lacks 192-bit key support. However, per the expectations of Linux crypto-cipher implementations, the software cipher implementation must support all the AES-compliant key sizes.
The AES-KL cipher implementation achieves this constraint by logging a warning and falling back to AES-NI. In other words, the 192-bit key-size limitation is documented but not enforced.
== Wrapping Key Restore Failure Handling ==
In the event of hardware failure, the wrapping key is lost from deep sleep states. Then, the wrapping key turns to zero which is an unusable state.
The x86 core provides valid_keylocker() to indicate the failure. Subsequent setkey() as well as encode()/decode() can check it and return -ENODEV if failed. This allows an error code to be returned instead of encountering abrupt exceptions.
== Userspace Exposition ==
Keylocker implementations have measurable performance penalties. Therefore, keep the current default remains the same.
However, with a slow storage device, storage bandwidth is the bottleneck, even if disk encryption is enabled by AES-KL. Thus, it is up to the end user to decide whether to use AES-KL. User can select it by the name 'xts-aes-aeskl' shown in /proc/crypto.
== 64-bit Only ==
Support 64-bit only, as the 32-bit kernel is being deprecated.
---
Changes from the previous posting (Eric):
* Assembly Code
- Improve function argument descriptions.
- Simplify the AES processing macro.
- Rename some symbols.
* Glue Code:
- Do not use function pointer variables; direct calls.
- Define a union for the two 'ctx' structures.
- Clarify a few code spots with comments.
- Adjust some variable and struct names.
* Kconfig
- Separate out the Kconfig.assembler change.
--- arch/x86/crypto/Kconfig | 18 ++ arch/x86/crypto/Makefile | 3 + arch/x86/crypto/aeskl-xts-x86_64.S | 337 ++++++++++++++++++++++++ arch/x86/crypto/aeskl_glue.c | 409 +++++++++++++++++++++++++++++ arch/x86/crypto/aesni-intel_glue.c | 13 +- arch/x86/crypto/aesni-xts.h | 15 ++ 6 files changed, 790 insertions(+), 5 deletions(-) create mode 100644 arch/x86/crypto/aeskl-xts-x86_64.S create mode 100644 arch/x86/crypto/aeskl_glue.c create mode 100644 arch/x86/crypto/aesni-xts.h
arch/x86/crypto/Kconfig +18 -0
@@ -29,6 +29,24 @@ config CRYPTO_AES_NI_INTEL Architecture: x86 (32-bit and 64-bit) using: - AES-NI (AES new instructions) +config CRYPTO_AES_KL + tristate "Ciphers: AES, modes: XTS (AES-KL)" + depends on X86 && 64BIT + depends on AS_KEYLOCKER + select CRYPTO_AES_NI_INTEL + select CRYPTO_SIMD + select X86_KEYLOCKER + + help + Block cipher: AES cipher algorithms + Length-preserving ciphers: AES with XTS + + Architecture: x86 (64-bit) using: + - AES-KL (AES Key Locker) + - AES-NI for a 192-bit key + + See Documentation/arch/x86/keylocker.rst for more details. + config CRYPTO_BLOWFISH_X86_64 tristate "Ciphers: Blowfish, modes: ECB, CBC" depends on X86 && 64BIT
arch/x86/crypto/Makefile +3 -0
@@ -51,6 +51,9 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o \ aes_ctrby8_avx-x86_64.o aes-xts-avx-x86_64.o +obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-x86_64.o +aeskl-x86_64-y := aeskl-xts-x86_64.o aeskl_glue.o + obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
arch/x86/crypto/aeskl-xts-x86_64.S new file +337 -0
337 lines not on this page · raw message
arch/x86/crypto/aeskl_glue.c new file +409 -0
409 lines not on this page · raw message
arch/x86/crypto/aesni-intel_glue.c +8 -5
@@ -35,7 +35,7 @@#include <linux/workqueue.h> #include <linux/spinlock.h> #include <linux/static_call.h> - +#include "aesni-xts.h" #define AESNI_ALIGN 16#define AESNI_ALIGN_ATTR __attribute__ ((__aligned__(AESNI_ALIGN)))
@@ -864,8 +864,8 @@ static int helper_rfc4106_decrypt(struct aead_request *req) }#endif -static int xts_setkey_aesni (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keylen) +int xts_setkey_aesni (struct crypto_skcipher *tfm,const u8 *key, +unsigned int keylen) {struct aesni_xts_ctx *ctx =aes_xts_ctx (tfm);int err;
@@ -884,6 +884,7 @@ static int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key,/* second half of xts-key is for tweak */ return aes_set_key_common (&ctx->tweak_ctx, key + keylen, keylen); } +EXPORT_SYMBOL_GPL (xts_setkey_aesni);typedef void (*xts_encrypt_iv_func )(const struct crypto_aes_ctx *tweak_key,u8 iv[AES_BLOCK_SIZE ]);
@@ -1020,15 +1021,17 @@ static void aesni_xts_decrypt(const struct crypto_aes_ctx *key,aesni_xts_dec (key, dst, src, len, tweak); } -static int xts_encrypt_aesni (struct skcipher_request *req) +int xts_encrypt_aesni (struct skcipher_request *req) {return xts_crypt (req, aesni_xts_encrypt_iv, aesni_xts_encrypt); } +EXPORT_SYMBOL_GPL (xts_encrypt_aesni); -static int xts_decrypt_aesni (struct skcipher_request *req) +int xts_decrypt_aesni (struct skcipher_request *req) {return xts_crypt (req, aesni_xts_encrypt_iv, aesni_xts_decrypt); } +EXPORT_SYMBOL_GPL (xts_decrypt_aesni);static struct crypto_alg aesni_cipher_alg = { .cra_name ="aes" ,
arch/x86/crypto/aesni-xts.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _AESNI_XTS_H +#define _AESNI_XTS_H + +/* +* These AES-NI functions are used by the AES-KL code as a fallback when +* a 192-bit key is provided. Key Locker does not support 192-bit keys. +*/ + +int xts_setkey_aesni (struct crypto_skcipher *tfm,const u8 *key,unsigned int keylen); +int xts_encrypt_aesni (struct skcipher_request *req); +int xts_decrypt_aesni (struct skcipher_request *req); + +#endif /* _AESNI_XTS_H */
Signature · 1 line
2.34.1
Quoted 9 lines · Chang S. Bae
Then, the following is a summary of changes per patch since v8 [6]:
PATCH7-8:
* Invoke the setup code viaarch_initcall()due to upstream changes delaying the FPU setup.PATCH9-11:
* Add new patches for security and hotplug support clarification
I've recently made updates to a few patches, primarily related to the mitigation parts. While the series is still under review, Eric's VAES patches have been merged into the crypto tree and are currently being sorted out. Once things settle down, I will make a few adjustments on the crypto side. Then, another revision will be necessary thereafter.
Thanks,
Chang
Hi,
Quoted 11 lines · Chang S. Bae
Quoted 9 lines · Chang S. Bae
Then, the following is a summary of changes per patch since v8 [6]:
PATCH7-8:
* Invoke the setup code viaarch_initcall()due to upstream changes delaying the FPU setup.PATCH9-11:
* Add new patches for security and hotplug support clarification
Quoted 8 lines
I've recently made updates to a few patches, primarily related to the mitigation parts. While the series is still under review, Eric's VAES patches have been merged into the crypto tree and are currently being sorted out. Once things settle down, I will make a few adjustments on the crypto side. Then, another revision will be necessary thereafter.
Thanks,
Chang
Thanks for the updated patchset!
Do you have a plan for how this will be merged? Which trees will the patches go through? I think that the actual AES-XTS implementation could still use a bit more polishing; see my comments below. However, patches 1-12 don't need to wait for that. Perhaps the x86 maintainers would like to take patches 1-12 for v6.10? Then the AES-XTS support can go through the crypto tree afterwards.
As you noticed, this cycle I've been optimizing AES-XTS for x86_64 by adding new VAES and AES-NI + AVX implementations. I have some ideas for the Key Locker based implementation of AES-XTS:
First, surely it's the case that in practice, all CPUs that support Key Locker also support AVX? If so, then there's no need for the Key Locker assembly to use legacy SSE instructions. It should instead target AVX and use VEX-coded instructions. This would save some instructions and improve performance.
Since the Key Locker assembly only supports 64-bit mode, it should also feel free to use registers xmm8-xmm15 for purposes such as caching the XTS tweaks. This would improve performance.
Since the Key Locker assembly advances a large number of XTS tweaks at a time (8), I'm also wondering if it would be faster to multiply by x^8 directly instead of multiplying by x sequentially eight times. This can be done using the pclmulqdq instruction; see aes-xts-avx-x86_64.S which implements this optimization. Probably all CPUs that support Key Locker also support PCLMULQDQ.
I'm also trying to think of the best way to organize the Key Locker AES-XTS glue code. I see that you're proposing to share the glue code with the existing AES-XTS implementations. Unfortunately I don't think this ends up working very well, due to the facts that the Key Locker code can return errors and uses a different key type. I think that for now, I'd prefer that you simply copied the XTS glue code into aeskl-intel_glue.c and modified it as needed. (But make sure to use the new version of the glue code, which is faster.)
For falling back to AES-NI, I think the cleanest solution is to call the top-level setkey, encrypt, and decrypt functions (the ones that are set in the xts-aes-aesni skcipher_alg), instead of calling lower-level functions as your current patchset does.
If you could keep the Key Locker assembly roughly stylistically consistent with the new aes-xts-avx-x86_64.S, that would be great too.
Do you happen to know if there's any way to test the Key Locker AES-XTS code without having access to a bare metal machine with a CPU that supports Key Locker? I tried a Sapphire Rapids based VM in Google Compute Engine, but it doesn't enumerate Key Locker. I also don't see anything in QEMU related to Key Locker. So I don't currently have an easy way to test this patchset.
Finally, a high level question. Key Locker has been reported to be substantially slower than AES-NI. At the same time, VAES has recently doubled performance over AES-NI. I'd guess this leaves Key Locker even further behind. Given that, how useful is this patchset? I'm a bit concerned that this might be something that sounds good in theory but won't be used in practice. Are performance improvements for Key Locker on the horizon? (Well, there are the improvements I suggested above, which should help, but it sounds like main issue is the Key Locker instructions themselves which are just fundamentally slower.)
- Eric
Hi Eric,
Sorry for my delay. I just found your message in my cluttered junk folder today after returning from my week off.
Quoted 6 lines · Eric Biggers
Do you have a plan for how this will be merged? Which trees will the patches go through? I think that the actual AES-XTS implementation could still use a bit more polishing; see my comments below. However, patches 1-12 don't need to wait for that. Perhaps the x86 maintainers would like to take patches 1-12 for v6.10? Then the AES-XTS support can go through the crypto tree afterwards.
Yeah, this series spans both x86 code and the crypto driver. I believe the decision should be made by maintainers. But, I suspect they'll want to see a well-established use case code before moving forward.
Quoted 3 lines
As you noticed, this cycle I've been optimizing AES-XTS for
x86_64by adding new VAES and AES-NI + AVX implementations. I have some ideas for the Key Locker based implementation of AES-XTS:
Thanks for the effort! I agree that the code could be beneficial for daily disk encryption needs.
Quoted 4 lines
First, surely it's the case that in practice, all CPUs that support Key Locker also support AVX? If so, then there's no need for the Key Locker assembly to use legacy SSE instructions. It should instead target AVX and use VEX-coded instructions. This would save some instructions and improve performance.
Unfortunately, the Key Locker instructions using the AVX states were never implemented.
Quoted 9 lines
Since the Key Locker assembly only supports 64-bit mode, it should also feel free to use registers xmm8-xmm15 for purposes such as caching the XTS tweaks. This would improve performance.
Since the Key Locker assembly advances a large number of XTS tweaks at a time (8), I'm also wondering if it would be faster to multiply by x^8 directly instead of multiplying by x sequentially eight times. This can be done using the pclmulqdq instruction; see
aes-xts-avx-x86_64.Swhich implements this optimization. Probably all CPUs that support Key Locker also support PCLMULQDQ.
I'll revisit the assembly code to incorporate your suggestions.
Quoted 7 lines
I'm also trying to think of the best way to organize the Key Locker AES-XTS glue code. I see that you're proposing to share the glue code with the existing AES-XTS implementations. Unfortunately I don't think this ends up working very well, due to the facts that the Key Locker code can return errors and uses a different key type. I think that for now, I'd prefer that you simply copied the XTS glue code into
aeskl-intel_glue.cand modified it as needed. (But make sure to use the new version of the glue code, which is faster.)
I agree; the proposed glue code looks messy due to the different return errors and key types. Ard made a point earlier about establishing a shared common code as they're logically quite similar. But I suppose it is more practical to pursue a separate glue code at this point.
Quoted 4 lines
For falling back to AES-NI, I think the cleanest solution is to call the top-level setkey, encrypt, and decrypt functions (the ones that are set in the xts-aes-aesni
skcipher_alg), instead of calling lower-level functions as your current patchset does.
Yes, falling back is indeed one of the ugly parts of this series. Let me retry this as you suggested.
Quoted 2 lines
If you could keep the Key Locker assembly roughly stylistically consistent with the new
aes-xts-avx-x86_64.S, that would be great too.
Okay.
Quoted 5 lines
Do you happen to know if there's any way to test the Key Locker AES-XTS code without having access to a bare metal machine with a CPU that supports Key Locker? I tried a Sapphire Rapids based VM in Google Compute Engine, but it doesn't enumerate Key Locker. I also don't see anything in QEMU related to Key Locker. So I don't currently have an easy way to test this patchset.
No, there isn't currently an emulation option available to the public that I'm aware of. this feature has been available on client systems since the Tiger Lake generation.
Quoted 8 lines
Finally, a high level question. Key Locker has been reported to be substantially slower than AES-NI. At the same time, VAES has recently doubled performance over AES-NI. I'd guess this leaves Key Locker even further behind. Given that, how useful is this patchset? I'm a bit concerned that this might be something that sounds good in theory but won't be used in practice. Are performance improvements for Key Locker on the horizon? (Well, there are the improvements I suggested above, which should help, but it sounds like main issue is the Key Locker instructions themselves which are just fundamentally slower.)
On our latest implementations, we've observed the Key Locker performance on cryptsetup seems to be roughly the same as what we posted earlier [2]. Yes, this sounds like a fair analogy to me, especially given your vAES code.
Thanks,
Chang
[1] https://lore.kernel.org/lkml/CAMj1kXGa4f21eH0mdxd1pQsZMUjUr1Btq+Dgw-gC=O-yYft7xw@mail.gmail.com/
[2] https://lore.kernel.org/lkml/[email protected]/
Quoted 7 lines · Chang S. Bae
Quoted 4 lines
First, surely it's the case that in practice, all CPUs that support Key Locker also support AVX? If so, then there's no need for the Key Locker assembly to use legacy SSE instructions. It should instead target AVX and use VEX-coded instructions. This would save some instructions and improve performance.
Unfortunately, the Key Locker instructions using the AVX states were never implemented.
Sure, you could still use VEX-coded 128-bit instructions for everything other than the actual AES (for example, the XTS tweak computation) though, right? They're a bit more convenient to work with since they are non-destructive.
- Eric
Quoted 4 lines · Eric Biggers
Sure, you could still use VEX-coded 128-bit instructions for everything other than the actual AES (for example, the XTS tweak computation) though, right? They're a bit more convenient to work with since they are non-destructive.
Right.
Thanks,
Chang