Skip to contentSkip to conversation

x86: Support Key Locker

PATCH Superseded Chang S. Bae 5 participants 30 messages Nov 24, 2021 – Dec 6, 2021

Subsystems (4)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover x86: Support Key Locker

Recall that AES-NI [1] is a set of CPU instructions to perform encryption operations. Like all other software encryption implementations it relies on OS memory protections to keep clear-text key material from being exfiltrated across security domains. However, there are demonstrated methods for exfiltrating data across security domains.

Key Locker [2] is a CPU feature to reduce key exfiltration opportunities while maintaining a programming interface similar to AES-NI. It converts the AES key into an encoded form, called the 'key handle' [3]. The key handle is a wrapped version of the clear-text key where the wrapping key has limited exposure. Once converted, all subsequent data encryption using new AES instructions (AES-KL) uses this key handle, reducing the exposure of private key material in memory.

As mentioned, Key Locker introduces a CPU-internal wrapping key. This key is loaded in a CPU state that software can not access, and then it is used to convert the AES keys. At every boot, a new randomized key is generated and loaded. Thus, the key handle is revoked upon system reboot (including kexec-reboot).

== Threat Model and Mitigation Description ==

A targeted attack is with brief physical access [4] to the victim device in unlock state or with the screen locked, but with keys in memory. For example, the attacker might use a malicious USB peripheral to exploit a kernel bug or perform a cold boot attack [4], and then extract the disk encryption master key. Using this master key, the attacker will be able to extract the contents of the disk. This includes data yet-to-be-written when acquiring the device from the victim user at a later point in time and performing forensic analysis.

Key Locker reduces opportunities for long-lived keys to be exfiltrated from system RAM. Once the key is converted to a key handle, the clear text key is no longer available to a class of data exfiltration techniques.

== Disk Encryption Use Case ==

Disk encryption uses Key Locker to mitigate key exfiltration as follows:

1. Configuration for Key Locker: AES-KL shows up in /proc/crypto as a distinct cipher option. From there, tools like cryptsetup [5] can select AES-KL vs AES-NI. For example,

   $ cryptsetup luksFormat --cipher="capi:xts-aes-aeskl-plain" <device>

Note: AES-KL has a performance tradeoff. See details in 'Performance' below.

2. Disk encryption flow with key protection:

* The cryptsetup utility is responsible for loading the volume key into the kernel's keyring and passing a reference of the key. Once dm-crypt [6] has set up the volume, user space is responsible for invalidating the key material so that only the key handle remains in memory. Cryptsetup does this, e.g. via crypt_free_volume_key() and crypt_safe_free().

* The AES-KL code in the kernel's crypto library uses the key handle instead of the actual clear text key.

== Non Use Cases ==

Bare metal disk encryption is the only use case intended by these patches. Userspace usage is not supported because there is no ABI provided to communicate and coordinate wrapping-key restore failures to userspace. For now, the key restore failures are only coordinated with kernel users. For this reason a "keylocker" indicator is not published in /proc/cpuinfo. At the same time, the kernel can not prevent userspace from using the AES-KL instructions when Key Locker support has been enabled, so the lack of userspace support is only documented, not actively enforced. Key Locker support is not enumerated to VM guests.

== Performance ==

This feature comes with some performance penalty vs AESNI. The cryptsetup utility [5] has been used to measure the Key Locker performance. Below results have been measured [8] on an Intel 11th-gen Core Processor, code named Tigerlake mobile part [9].

Below is the average encryption and decryption rates with key size of 256b.

Commands:
cryptsetup version – 2.3.4

$ cryptsetup benchmark-c aes-cbc -s 256
$ cryptsetup benchmark-c aes-xts -s 256

Tests are approximate using memory only (no storage IO).

+-----------+---------------+---------------+
| Cipher    |   Encryption  | Decryption    |
| (AES-NI)  |    (MiB/s)    | (MiB/s)       |
+-----------+---------------+---------------+
| AES-CBC   |     1242.6    |   4446.5      |
| AES-XTS   |     4233.3    |   4359.7      |
+-----------+-------------------------------+
+-----------+---------------+---------------+
| Cipher    |   Encryption  | Decryption    |
| (AES-KL)  |    (MiB/s)    | (MiB/s)       |
+-----------+---------------+---------------+
| AES-CBC   |     505.3     |   2097.8      |
| AES-XTS   |     1130      |   696.4       |
+-----------+-------------------------------+

The cryptsetup benchmark indicates Key Locker raw throughput can be ~5x slower than AES-NI. For disk encryption, storage bandwidth may be the bottleneck before encryption bandwidth, but the potential performance difference is why AES-KL is advertised as a distinct cipher in /proc/crypto rather than the kernel transparently replacing AES-NI usage with AES-KL.

== Patch Series ==

The series touches two areas -- the x86 core and the x86 crypto library:

* PATCH01-09: Implement Key Locker support in the x86 core. A new internal wrapping key is loaded at boot time and then it is restored from deep sleep states. The implication is that, e.g., a dm-crypt user needs to re-enter the private key at every power-on, per typical expectations, but it does not expect the user to re-enter the key over suspend events. The AES-KL code in the kernel's crypto library depends on this key support. Build up this support via helpers in the feature-dedicated .c file. Include documentation.

* PATCH10-15: For the x86 crypto library, it first prepares the AES-NI code to accommodate the new AES implementation. Then incrementally add base functions and various modes support -- ECB, CBC, CTR, and XTS. The code was found to pass the crypto test.

Changes from RFC v2 [7]:
* Clarify the usage case -- bare metal disk encryption. (Andy Lutomirski)
* Change the backup key failure handling. (Dan Williams) (PATCH8)
* Do not publish 'keylocker' indicator to userspace via /proc/cpuinfo. (PATCH2)
* Make CONFIG_X86_KEYLOCKER selected by CONFIG_CRYPTO_AES_KL. (Dan Williams) (PATCH9)
* Clarify AES-KL limitation and tradeoff. (Andy Lutomirski) (PATCH11)
* Polish the changelog, and refactor patches.
* Drop the self-test as recommending no userspace use. (Dan Williams)
* Drop the hardware randomization option.
* Limit to support bare metal only. (PATCH7)
* Rename MSRs. (Dan Williams) (PATCH5)
* Clean up code. (Dan Williams) (PATCH7)
* Add documentation. (PATCH1)

Thanks to Dan Williams, Charishma Gairuboyina, and Kumar Dwarakanath for help with the cover letter.

== Reference ==

[1] Intel Advanced Encryption Standard Instructions (AES-NI):
https://www.intel.com/content/www/us/en/developer/articles/technical/advanced-encryption-standard-instructions-aes-ni.html
[2] Intel Key Locker Specification:
https://software.intel.com/content/dam/develop/external/us/en/documents/343965-intel-key-locker-specification.pdf
[3] This encoded form contains ciphertext of AES key, Additional Authentication Data, and integrity tag information. Section 1.4 Handle Format [2] describes the format.
[4] Key Locker cannot protect the user data in the event of a full system compromise, or against the scenarios where the attack can observe the creation of the key handle from the original key.
[5] cryptsetup: https://gitlab.com/cryptsetup/cryptsetup
[6] DM-crypt:
https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-crypt.html
[7] RFC V2: https://lore.kernel.org/lkml/[email protected]/
[8] Intel publishes information about product performance at www.Intel.com/PerformanceIndex.
[9] Tigerlake:
https://www.intel.com/content/www/us/en/products/docs/processors/embedded/11th-gen-product-brief.html

Chang S. Bae (15):
Documentation/x86: Document Key Locker
x86/cpufeature: Enumerate Key Locker feature
x86/insn: Add Key Locker instructions to the opcode map x86/asm: Add a wrapper function for the LOADIWKEY instruction x86/msr-index: Add MSRs for Key Locker internal wrapping key x86/keylocker: Define Key Locker CPUID leaf x86/cpu/keylocker: Load an internal wrapping key at boot-time x86/power/keylocker: Restore internal wrapping key from the ACPI S3/4
sleep states
x86/cpu: Add a configuration and command line option for Key Locker
crypto: x86/aes - Prepare for a new AES implementation
crypto: x86/aes-kl - Support AES algorithm using Key Locker
instructions
crypto: x86/aes-kl - Support ECB mode
crypto: x86/aes-kl - Support CBC mode
crypto: x86/aes-kl - Support CTR mode
crypto: x86/aes-kl - Support XTS mode

 .../admin-guide/kernel-parameters.txt         |    2 +
 Documentation/x86/index.rst                   |    1 +
 Documentation/x86/keylocker.rst               |   98 ++
 arch/x86/Kconfig                              |    3 +
 arch/x86/crypto/Makefile                      |    5 +-
 arch/x86/crypto/aes-intel_asm.S               |   26 +
 arch/x86/crypto/aes-intel_glue.c              |  219 +++
 arch/x86/crypto/aes-intel_glue.h              |   61 +
 arch/x86/crypto/aeskl-intel_asm.S             | 1186 +++++++++++++++++
 arch/x86/crypto/aeskl-intel_glue.c            |  401 ++++++
 arch/x86/crypto/aesni-intel_asm.S             |   90 +-
 arch/x86/crypto/aesni-intel_glue.c            |  313 +----
 arch/x86/crypto/aesni-intel_glue.h            |   88 ++
 arch/x86/include/asm/cpufeatures.h            |    1 +
 arch/x86/include/asm/disabled-features.h      |    8 +-
 arch/x86/include/asm/keylocker.h              |   45 +
 arch/x86/include/asm/msr-index.h              |    6 +
 arch/x86/include/asm/special_insns.h          |   33 +
 arch/x86/include/uapi/asm/processor-flags.h   |    2 +
 arch/x86/kernel/Makefile                      |    1 +
 arch/x86/kernel/cpu/common.c                  |   21 +-
 arch/x86/kernel/cpu/cpuid-deps.c              |    1 +
 arch/x86/kernel/keylocker.c                   |  199 +++
 arch/x86/kernel/smpboot.c                     |    2 +
 arch/x86/lib/x86-opcode-map.txt               |   11 +-
 arch/x86/power/cpu.c                          |    2 +
 crypto/Kconfig                                |   44 +
 tools/arch/x86/lib/x86-opcode-map.txt         |   11 +-
 28 files changed, 2534 insertions(+), 346 deletions(-)
 create mode 100644 Documentation/x86/keylocker.rst
 create mode 100644 arch/x86/crypto/aes-intel_asm.S
 create mode 100644 arch/x86/crypto/aes-intel_glue.c
 create mode 100644 arch/x86/crypto/aes-intel_glue.h
 create mode 100644 arch/x86/crypto/aeskl-intel_asm.S
 create mode 100644 arch/x86/crypto/aeskl-intel_glue.c
 create mode 100644 arch/x86/crypto/aesni-intel_glue.h
 create mode 100644 arch/x86/include/asm/keylocker.h
 create mode 100644 arch/x86/kernel/keylocker.c

base-commit: 7284bd9822f33a3be80ac6d92b4540d6dcfb5219 --
2.17.1

#1 lore raw

PATCH v3 1/15 Documentation/x86: Document Key Locker

Document the overview of the feature along with relevant consideration when provisioning dm-crypt volumes with AES-KL instead of AES-NI.

---
Changes from RFC v2:
* Add as a new patch.

---
 Documentation/x86/index.rst     |  1 +
 Documentation/x86/keylocker.rst | 98 +++++++++++++++++++++++++++++++++
 2 files changed, 99 insertions(+)
 create mode 100644 Documentation/x86/keylocker.rst
Documentation/x86/index.rst +1 -0
index f498f1d36cd3..bbea47ea10f6 100644
@@ -38,3 +38,4 @@ x86-specific Documentation
    features
    elf_auxvec
    xstate
+   keylocker
Documentation/x86/keylocker.rst new file +98 -0

98 lines not on this page · raw message

Signature · 1 line
2.17.1
#2 lore raw

PATCH v3 2/15 x86/cpufeature: Enumerate Key Locker feature

In reply to Chang S. Bae

Key Locker is a CPU feature to minimize exposure of clear-text key material. An encoded form, called 'key handle', is referenced for data encryption or decryption instead of accessing the clear text key.

A wrapping key loaded in the CPU's software-inaccessible state is used to transform a user key into a key handle.

It supports Advanced Encryption Standard (AES) cipher algorithm with new SIMD instruction set like its predecessor (AES-NI). So a new AES implementation will follow in the kernel's crypto library.

Here add it to enumerate the hardware capability, but it will not be shown in /proc/cpuinfo as userspace usage is not supported.

Make the feature depend on XMM2 as it comes with AES SIMD instructions.

Add X86_FEATURE_KEYLOCKER to the disabled features mask. It will be enabled under a new config option.

---
Changes from RFC v2:
* Do not publish
* Update the changelog.

Changes from RFC v1:
* Updated the changelog.

---
 arch/x86/include/asm/cpufeatures.h          | 1 +
 arch/x86/include/asm/disabled-features.h    | 8 +++++++-
 arch/x86/include/uapi/asm/processor-flags.h | 2 ++
 arch/x86/kernel/cpu/cpuid-deps.c            | 1 +
 4 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeatures.h +1 -0
index d5b5f2ab87a0..e1964446bbe5 100644
@@ -361,6 +361,7 @@
 #define X86_FEATURE_AVX512_VPOPCNTDQ	(16*32+14) /* POPCNT for vectors of DW/QW */
 #define X86_FEATURE_LA57		(16*32+16) /* 5-level page tables */
 #define X86_FEATURE_RDPID		(16*32+22) /* RDPID instruction */
+#define X86_FEATURE_KEYLOCKER		(16*32+23) /* "" Key Locker */
 #define X86_FEATURE_BUS_LOCK_DETECT	(16*32+24) /* Bus Lock detect */
 #define X86_FEATURE_CLDEMOTE		(16*32+25) /* CLDEMOTE instruction */
 #define X86_FEATURE_MOVDIRI		(16*32+27) /* MOVDIRI instruction */
arch/x86/include/asm/disabled-features.h +7 -1
index 8f28fafa98b3..75e1e87640d4 100644
@@ -44,6 +44,12 @@
 # define DISABLE_OSPKE		(1<<(X86_FEATURE_OSPKE & 31))
 #endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */
 
+#ifdef CONFIG_X86_KEYLOCKER
+# define DISABLE_KEYLOCKER	0
+#else
+# define DISABLE_KEYLOCKER	(1<<(X86_FEATURE_KEYLOCKER & 31))
+#endif /* CONFIG_X86_KEYLOCKER */
+
 #ifdef CONFIG_X86_5LEVEL
 # define DISABLE_LA57	0
 #else
@@ -85,7 +91,7 @@
 #define DISABLED_MASK14	0
 #define DISABLED_MASK15	0
 #define DISABLED_MASK16	(DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \
-			 DISABLE_ENQCMD)
+			 DISABLE_ENQCMD|DISABLE_KEYLOCKER)
 #define DISABLED_MASK17	0
 #define DISABLED_MASK18	0
 #define DISABLED_MASK19	0
arch/x86/include/uapi/asm/processor-flags.h +2 -0
index bcba3c643e63..b958a95a0908 100644
@@ -124,6 +124,8 @@
 #define X86_CR4_PCIDE		_BITUL(X86_CR4_PCIDE_BIT)
 #define X86_CR4_OSXSAVE_BIT	18 /* enable xsave and xrestore */
 #define X86_CR4_OSXSAVE		_BITUL(X86_CR4_OSXSAVE_BIT)
+#define X86_CR4_KEYLOCKER_BIT	19 /* enable Key Locker */
+#define X86_CR4_KEYLOCKER	_BITUL(X86_CR4_KEYLOCKER_BIT)
 #define X86_CR4_SMEP_BIT	20 /* enable SMEP support */
 #define X86_CR4_SMEP		_BITUL(X86_CR4_SMEP_BIT)
 #define X86_CR4_SMAP_BIT	21 /* enable SMAP support */
arch/x86/kernel/cpu/cpuid-deps.c +1 -0
index c881bcafba7d..abe7e04b27d9 100644
@@ -78,6 +78,7 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_XFD,			X86_FEATURE_XSAVES    },
 	{ X86_FEATURE_XFD,			X86_FEATURE_XGETBV1   },
 	{ X86_FEATURE_AMX_TILE,			X86_FEATURE_XFD       },
+	{ X86_FEATURE_KEYLOCKER,		X86_FEATURE_XMM2      },
 	{}
 };
 
Signature · 1 line
2.17.1
#3 lore raw

PATCH v3 3/15 x86/insn: Add Key Locker instructions to the opcode map

In reply to Chang S. Bae

Add the following Key Locker instructions to the opcode map:

LOADIWKEY:
Load an CPU-internal wrapping key.

ENCODEKEY128:
Wrap a 128-bit AES key to a key handle.

ENCODEKEY256:
Wrap a 256-bit AES key to a key handle.

AESENC128KL:
Encrypt a 128-bit block of data using a 128-bit AES key indicated by a key handle.

AESENC256KL:
Encrypt a 128-bit block of data using a 256-bit AES key indicated by a key handle.

AESDEC128KL:
Decrypt a 128-bit block of data using a 128-bit AES key indicated by a key handle.

AESDEC256KL:
Decrypt a 128-bit block of data using a 256-bit AES key indicated by a key handle.

AESENCWIDE128KL:
Encrypt 8 128-bit blocks of data using a 128-bit AES key indicated by a key handle.

AESENCWIDE256KL:
Encrypt 8 128-bit blocks of data using a 256-bit AES key indicated by a key handle.

AESDECWIDE128KL:
Decrypt 8 128-bit blocks of data using a 128-bit AES key indicated by a key handle.

AESDECWIDE256KL:
Decrypt 8 128-bit blocks of data using a 256-bit AES key indicated by a key handle.

Details of these instructions can be found in Intel Software Developer's Manual.

---
Changes from RFC v1:
* Separated out the LOADIWKEY addition in a new patch.
* Included AES instructions to avoid warning messages when the AES Key Locker module is built.

---
 arch/x86/lib/x86-opcode-map.txt       | 11 +++++++----
 tools/arch/x86/lib/x86-opcode-map.txt | 11 +++++++----
 2 files changed, 14 insertions(+), 8 deletions(-)
arch/x86/lib/x86-opcode-map.txt +7 -4
index ec31f5b60323..eb702fc6572e 100644
@@ -794,11 +794,12 @@ cb: sha256rnds2 Vdq,Wdq | vrcp28ss/d Vx,Hx,Wx (66),(ev)
 cc: sha256msg1 Vdq,Wdq | vrsqrt28ps/d Vx,Wx (66),(ev)
 cd: sha256msg2 Vdq,Wdq | vrsqrt28ss/d Vx,Hx,Wx (66),(ev)
 cf: vgf2p8mulb Vx,Wx (66)
+d8: AESENCWIDE128KL Qpi (F3),(000),(00B) | AESENCWIDE256KL Qpi (F3),(000),(10B) | AESDECWIDE128KL Qpi (F3),(000),(01B) | AESDECWIDE256KL Qpi (F3),(000),(11B)
 db: VAESIMC Vdq,Wdq (66),(v1)
-dc: vaesenc Vx,Hx,Wx (66)
-dd: vaesenclast Vx,Hx,Wx (66)
-de: vaesdec Vx,Hx,Wx (66)
-df: vaesdeclast Vx,Hx,Wx (66)
+dc: vaesenc Vx,Hx,Wx (66) | LOADIWKEY Vx,Hx (F3) | AESENC128KL Vpd,Qpi (F3)
+dd: vaesenclast Vx,Hx,Wx (66) | AESDEC128KL Vpd,Qpi (F3)
+de: vaesdec Vx,Hx,Wx (66) | AESENC256KL Vpd,Qpi (F3)
+df: vaesdeclast Vx,Hx,Wx (66) | AESDEC256KL Vpd,Qpi (F3)
 f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2)
 f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2)
 f2: ANDN Gy,By,Ey (v)
@@ -808,6 +809,8 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSSD/Q My,
 f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v)
 f8: MOVDIR64B Gv,Mdqq (66) | ENQCMD Gv,Mdqq (F2) | ENQCMDS Gv,Mdqq (F3)
 f9: MOVDIRI My,Gy
+fa: ENCODEKEY128 Ew,Ew (F3)
+fb: ENCODEKEY256 Ew,Ew (F3)
 EndTable
 
 Table: 3-byte opcode 2 (0x0f 0x3a)
tools/arch/x86/lib/x86-opcode-map.txt +7 -4
index ec31f5b60323..eb702fc6572e 100644
@@ -794,11 +794,12 @@ cb: sha256rnds2 Vdq,Wdq | vrcp28ss/d Vx,Hx,Wx (66),(ev)
 cc: sha256msg1 Vdq,Wdq | vrsqrt28ps/d Vx,Wx (66),(ev)
 cd: sha256msg2 Vdq,Wdq | vrsqrt28ss/d Vx,Hx,Wx (66),(ev)
 cf: vgf2p8mulb Vx,Wx (66)
+d8: AESENCWIDE128KL Qpi (F3),(000),(00B) | AESENCWIDE256KL Qpi (F3),(000),(10B) | AESDECWIDE128KL Qpi (F3),(000),(01B) | AESDECWIDE256KL Qpi (F3),(000),(11B)
 db: VAESIMC Vdq,Wdq (66),(v1)
-dc: vaesenc Vx,Hx,Wx (66)
-dd: vaesenclast Vx,Hx,Wx (66)
-de: vaesdec Vx,Hx,Wx (66)
-df: vaesdeclast Vx,Hx,Wx (66)
+dc: vaesenc Vx,Hx,Wx (66) | LOADIWKEY Vx,Hx (F3) | AESENC128KL Vpd,Qpi (F3)
+dd: vaesenclast Vx,Hx,Wx (66) | AESDEC128KL Vpd,Qpi (F3)
+de: vaesdec Vx,Hx,Wx (66) | AESENC256KL Vpd,Qpi (F3)
+df: vaesdeclast Vx,Hx,Wx (66) | AESDEC256KL Vpd,Qpi (F3)
 f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2)
 f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2)
 f2: ANDN Gy,By,Ey (v)
@@ -808,6 +809,8 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSSD/Q My,
 f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v)
 f8: MOVDIR64B Gv,Mdqq (66) | ENQCMD Gv,Mdqq (F2) | ENQCMDS Gv,Mdqq (F3)
 f9: MOVDIRI My,Gy
+fa: ENCODEKEY128 Ew,Ew (F3)
+fb: ENCODEKEY256 Ew,Ew (F3)
 EndTable
 
 Table: 3-byte opcode 2 (0x0f 0x3a)
Signature · 1 line
2.17.1
#4 lore raw

PATCH v3 4/15 x86/asm: Add a wrapper function for the LOADIWKEY instruction

In reply to Chang S. Bae

Key Locker introduces a CPU-internal wrapping key to encode a user key to a key handle. Then a key handle is referenced instead of the plain text key.

The new instruction loads an internal wrapping key in the software-inaccessible CPU state. It operates only in kernel mode.

Define struct iwkey to pass the key value.

The kernel will use this function to load a new key at boot time when the feature is enabled.

---
Changes from RFC v2:
* Separate out the code as a new patch.
* Improve the usability with the new struct as an argument. (Dan Williams)

Note, Dan wondered if:
WARN_ON(!irq_fpu_usable());
would be appropriate in the load_xmm_iwkey() function.

---
 arch/x86/include/asm/keylocker.h     | 25 +++++++++++++++++++++
 arch/x86/include/asm/special_insns.h | 33 ++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
 create mode 100644 arch/x86/include/asm/keylocker.h
arch/x86/include/asm/keylocker.h new file +25 -0
new file mode 100644
index 000000000000..df84c83228a1
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef _ASM_KEYLOCKER_H
+#define _ASM_KEYLOCKER_H
+
+#ifndef __ASSEMBLY__
+
+#include <asm/fpu/types.h>
+
+/**
+ * struct iwkey - A temporary internal wrapping key storage.
+ * @integrity_key:	A 128-bit key to check that key handles have not
+ *			been tampered with.
+ * @encryption_key:	A 256-bit encryption key used in
+ *			wrapping/unwrapping a clear text key.
+ *
+ * This storage should be flushed immediately after loaded.
+ */
+struct iwkey {
+	struct reg_128_bit integrity_key;
+	struct reg_128_bit encryption_key[2];
+};
+
+#endif /*__ASSEMBLY__ */
+#endif /* _ASM_KEYLOCKER_H */
arch/x86/include/asm/special_insns.h +33 -0
index 68c257a3de0d..e6469b05facf 100644
@@ -9,6 +9,7 @@
 #include <asm/processor-flags.h>
 #include <linux/irqflags.h>
 #include <linux/jump_label.h>
+#include <asm/keylocker.h>
 
 /*
  * The compiler should not reorder volatile asm statements with respect to each
@@ -294,6 +295,38 @@ static inline int enqcmds(void __iomem *dst, const void *src)
 	return 0;
 }
 
+
+/**
+ * load_xmm_iwkey - Load a CPU-internal wrapping key
+ * @key:	A struct iwkey pointer.
+ *
+ * Load @key to XMMs then do LOADIWKEY. After this, flush XMM
+ * registers. Caller is responsible for kernel_cpu_begin().
+ */
+static inline void load_xmm_iwkey(struct iwkey *key)
+{
+	struct reg_128_bit zeros = { 0 };
+
+	asm volatile ("movdqu %0, %%xmm0; movdqu %1, %%xmm1; movdqu %2, %%xmm2;"
+		      :: "m"(key->integrity_key), "m"(key->encryption_key[0]),
+			 "m"(key->encryption_key[1]));
+
+	/*
+	 * LOADIWKEY %xmm1,%xmm2
+	 *
+	 * EAX and XMM0 are implicit operands. Load a key value
+	 * from XMM0-2 to a software-invisible CPU state. With zero
+	 * in EAX, CPU does not do hardware randomization and the key
+	 * backup is allowed.
+	 *
+	 * This instruction is supported by binutils >= 2.36.
+	 */
+	asm volatile (".byte 0xf3,0x0f,0x38,0xdc,0xd1" :: "a"(0));
+
+	asm volatile ("movdqu %0, %%xmm0; movdqu %0, %%xmm1; movdqu %0, %%xmm2;"
+		      :: "m"(zeros));
+}
+
 #endif /* __KERNEL__ */
 
 #endif /* _ASM_X86_SPECIAL_INSNS_H */
Signature · 1 line
2.17.1
#5 lore raw

PATCH v3 5/15 x86/msr-index: Add MSRs for Key Locker internal wrapping key

In reply to Chang S. Bae

The CPU state that contains the internal wrapping key is in the same power domain as the cache. So any sleep state that would invalidate the cache (like S3) also invalidates the state of the wrapping key.

But, since the state is inaccessible to software, it needs a special mechanism to save and restore the key during deep sleep.

A set of new MSRs are provided as an abstract interface to save and restore the wrapping key, and to check the key status.

The wrapping key is saved in a platform-scoped state of non-volatile media. The backup itself and its path from the CPU are encrypted and integrity protected.

But this storage's non-volatility is not architecturally guaranteed across off states, such as S5 and G3.

The MSRs will be used to back up the key for S3/4 sleep states. Then the kernel code writes one of them to restore the key in each CPU state.

---
Changes from RFC v2:
* Update the changelog. (Dan Williams)
* Rename the MSRs. (Dan Williams)

---
 arch/x86/include/asm/msr-index.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/msr-index.h +6 -0
index 01e2650b9585..7f11a3b3a75b 100644
@@ -941,4 +941,10 @@
 #define MSR_VM_IGNNE                    0xc0010115
 #define MSR_VM_HSAVE_PA                 0xc0010117
 
+/* MSRs for managing an internal wrapping key for Key Locker. */
+#define MSR_IA32_IWKEY_COPY_STATUS		0x00000990
+#define MSR_IA32_IWKEY_BACKUP_STATUS		0x00000991
+#define MSR_IA32_BACKUP_IWKEY_TO_PLATFORM	0x00000d91
+#define MSR_IA32_COPY_IWKEY_TO_LOCAL		0x00000d92
+
 #endif /* _ASM_X86_MSR_INDEX_H */
Signature · 1 line
2.17.1
#6 lore raw

PATCH v3 6/15 x86/keylocker: Define Key Locker CPUID leaf

In reply to Chang S. Bae

Define the feature-specific CPUID leaf and bits. Both Key Locker enabling code in the x86 core and AES Key Locker code in the crypto library will reference them.

---
Changes from RFC v2:
* Separate out the code as a new patch.

---
 arch/x86/include/asm/keylocker.h | 7 +++++++
 1 file changed, 7 insertions(+)
arch/x86/include/asm/keylocker.h +7 -0
index df84c83228a1..e85dfb6c1524 100644
@@ -5,6 +5,7 @@
 
 #ifndef __ASSEMBLY__
 
+#include <linux/bits.h>
 #include <asm/fpu/types.h>
 
 /**
@@ -21,5 +22,11 @@ struct iwkey {
 	struct reg_128_bit encryption_key[2];
 };
 
+#define KEYLOCKER_CPUID			0x019
+#define KEYLOCKER_CPUID_EAX_SUPERVISOR	BIT(0)
+#define KEYLOCKER_CPUID_EBX_AESKLE	BIT(0)
+#define KEYLOCKER_CPUID_EBX_WIDE	BIT(2)
+#define KEYLOCKER_CPUID_EBX_BACKUP	BIT(4)
+
 #endif /*__ASSEMBLY__ */
 #endif /* _ASM_KEYLOCKER_H */
Signature · 1 line
2.17.1
#7 lore raw

PATCH v3 7/15 x86/cpu/keylocker: Load an internal wrapping key at boot-time

In reply to Chang S. Bae

The Internal Wrapping Key (IWKey) is an entity of Key Locker to encode a clear text key into a key handle. This key is a pivot in protecting user keys. So the value has to be randomized before being loaded in the software-invisible CPU state.

IWKey needs to be established before the first user. Given that the only proposed Linux use case for Key Locker is dm-crypt, the feature could be lazily enabled when the first dm-crypt user arrives, but there is no precedent for late enabling of CPU features and it adds maintenance burden without demonstrative benefit outside of minimizing the visibility of Key Locker to userspace.

The kernel generates random bytes and load them at boot time. These bytes are flushed out immediately.

Setting the CR4.KL bit does not always enable the feature so ensure the dynamic CPU bit (CPUID.AESKLE) is set before loading the key.

Given that the Linux Key Locker support is only intended for bare metal dm-crypt consumption, and that switching IWKey per VM is untenable, explicitly skip Key Locker setup in the X86_FEATURE_HYPERVISOR case.

---
Changes from RFC v2:
* Make bare metal only.
* Clean up the code (e.g. dynamically allocate the key cache). (Dan Williams)
* Massage the changelog.
* Move out the LOADIWKEY wrapper and the Key Locker CPUID defines.

Note, Dan wonders that given that the only proposed Linux use case for Key Locker is dm-crypt, the feature could be lazily enabled when the first dm-crypt user arrives, but as Dave notes there is no precedent for late enabling of CPU features and it adds maintenance burden without demonstrative benefit outside of minimizing the visibility of Key Locker to userspace.

---
 arch/x86/include/asm/keylocker.h |  9 ++++
 arch/x86/kernel/Makefile         |  1 +
 arch/x86/kernel/cpu/common.c     |  5 +-
 arch/x86/kernel/keylocker.c      | 79 ++++++++++++++++++++++++++++++++
 arch/x86/kernel/smpboot.c        |  2 +
 5 files changed, 95 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/kernel/keylocker.c
arch/x86/include/asm/keylocker.h +9 -0
index e85dfb6c1524..820ac29c06d9 100644
@@ -5,6 +5,7 @@
 
 #ifndef __ASSEMBLY__
 
+#include <asm/processor.h>
 #include <linux/bits.h>
 #include <asm/fpu/types.h>
 
@@ -28,5 +29,13 @@ struct iwkey {
 #define KEYLOCKER_CPUID_EBX_WIDE	BIT(2)
 #define KEYLOCKER_CPUID_EBX_BACKUP	BIT(4)
 
+#ifdef CONFIG_X86_KEYLOCKER
+void setup_keylocker(struct cpuinfo_x86 *c);
+void destroy_keylocker_data(void);
+#else
+#define setup_keylocker(c) do { } while (0)
+#define destroy_keylocker_data() do { } while (0)
+#endif
+
 #endif /*__ASSEMBLY__ */
 #endif /* _ASM_KEYLOCKER_H */
arch/x86/kernel/Makefile +1 -0
index 2ff3e600f426..e15efa238497 100644
@@ -144,6 +144,7 @@ obj-$(CONFIG_PERF_EVENTS)		+= perf_regs.o
 obj-$(CONFIG_TRACING)			+= tracepoint.o
 obj-$(CONFIG_SCHED_MC_PRIO)		+= itmt.o
 obj-$(CONFIG_X86_UMIP)			+= umip.o
+obj-$(CONFIG_X86_KEYLOCKER)		+= keylocker.o
 
 obj-$(CONFIG_UNWINDER_ORC)		+= unwind_orc.o
 obj-$(CONFIG_UNWINDER_FRAME_POINTER)	+= unwind_frame.o
arch/x86/kernel/cpu/common.c +4 -1
index 0083464de5e3..23b4aa437c1e 100644
@@ -57,6 +57,8 @@
 #include <asm/microcode_intel.h>
 #include <asm/intel-family.h>
 #include <asm/cpu_device_id.h>
+#include <asm/keylocker.h>
+
 #include <asm/uv/uv.h>
 #include <asm/sigframe.h>
 
@@ -1595,10 +1597,11 @@ static void identify_cpu(struct cpuinfo_x86 *c)
 	/* Disable the PN if appropriate */
 	squash_the_stupid_serial_number(c);
 
-	/* Set up SMEP/SMAP/UMIP */
+	/* Setup various Intel-specific CPU security features */
 	setup_smep(c);
 	setup_smap(c);
 	setup_umip(c);
+	setup_keylocker(c);
 
 	/* Enable FSGSBASE instructions if available. */
 	if (cpu_has(c, X86_FEATURE_FSGSBASE)) {
arch/x86/kernel/keylocker.c new file +79 -0

79 lines not on this page · raw message

arch/x86/kernel/smpboot.c +2 -0
index ac2909f0cab3..3b81fd643784 100644
@@ -82,6 +82,7 @@
 #include <asm/spec-ctrl.h>
 #include <asm/hw_irq.h>
 #include <asm/stackprotector.h>
+#include <asm/keylocker.h>
 
 #ifdef CONFIG_ACPI_CPPC_LIB
 #include <acpi/cppc_acpi.h>
@@ -1475,6 +1476,7 @@ void __init native_smp_cpus_done(unsigned int max_cpus)
 	nmi_selftest();
 	impress_friends();
 	mtrr_aps_init();
+	destroy_keylocker_data();
 }
 
 static int __initdata setup_possible_cpus = -1;
Signature · 1 line
2.17.1
#8 lore raw

PATCH v3 8/15 x86/power/keylocker: Restore internal wrapping key from the ACPI S3/4 sleep states

In reply to Chang S. Bae

When the system state switches to these sleep states, the internal wrapping key gets reset in the CPU state.

The primary use case for the feature is bare metal dm-crypt. The key needs to be restored properly on wakeup, as dm-crypt does not prompt for the key on resume from suspend. Even the prompt it does perform for unlocking the volume where the hibernation image is stored, it still expects to reuse the key handles within the hibernation image once it is loaded. So it is motivated to meet dm-crypt's expectation that the key handles in the suspend-image remain valid after resume from an S-state.

Key Locker provides a mechanism to back up the internal wrapping key in non-volatile storage. The kernel requests a backup right after the key is loaded at boot time. It is copied back to each CPU upon wakeup.

While the backup may be maintained in NVM across S5 and G3 "off" states it is not architecturally guaranteed, nor is it expected by dm-crypt which expects to prompt for the key each time the volume is started.

The entirety of Key Locker needs to be disabled if the backup mechanism is not available unless CONFIG_SUSPEND=n, otherwise dm-crypt requires the backup to be available.

In the event of a key restore failure the kernel proceeds with an initialized IWKey state. This has the effect of invalidating any key handles that might be present in a suspend-image. When this happens dm-crypt will see I/O errors resulting from error returns from crypto_skcipher_{en,de}crypt(). While this will disrupt operations in the current boot, data is not at risk and access is restored at the next reboot to create new handles relative to the current IWKey.

Manage a feature-specific flag to communicate with the crypto implementation. This ensures to stop using the AES instructions upon the key restore failure while not turning off the feature.

---
Changes from RFC v2:
* Change the backup key failure handling. (Dan Williams)

Changes from RFC v1:
* Folded the warning message into the if condition check. (Rafael Wysocki)
* Rebased on the changes of the previous patches.
* Added error code for key restoration failures.
* Moved the restore helper.
* Added function descriptions.

---
 arch/x86/include/asm/keylocker.h |   4 +
 arch/x86/kernel/keylocker.c      | 124 ++++++++++++++++++++++++++++++-
 arch/x86/power/cpu.c             |   2 +
 3 files changed, 128 insertions(+), 2 deletions(-)
arch/x86/include/asm/keylocker.h +4 -0
index 820ac29c06d9..4000a5eed2c2 100644
@@ -32,9 +32,13 @@ struct iwkey {
 #ifdef CONFIG_X86_KEYLOCKER
 void setup_keylocker(struct cpuinfo_x86 *c);
 void destroy_keylocker_data(void);
+void restore_keylocker(void);
+extern bool valid_keylocker(void);
 #else
 #define setup_keylocker(c) do { } while (0)
 #define destroy_keylocker_data() do { } while (0)
+#define restore_keylocker() do { } while (0)
+static inline bool valid_keylocker { return false; }
 #endif
 
 #endif /*__ASSEMBLY__ */
arch/x86/kernel/keylocker.c +122 -2

164 lines not on this page · raw message

arch/x86/power/cpu.c +2 -0
index 9f2b251e83c5..1a290f529c73 100644
@@ -25,6 +25,7 @@
 #include <asm/cpu.h>
 #include <asm/mmu_context.h>
 #include <asm/cpu_device_id.h>
+#include <asm/keylocker.h>
 
 #ifdef CONFIG_X86_32
 __visible unsigned long saved_context_ebx;
@@ -262,6 +263,7 @@ static void notrace __restore_processor_state(struct saved_context *ctxt)
 	mtrr_bp_restore();
 	perf_restore_debug_store();
 	msr_restore_context(ctxt);
+	restore_keylocker();
 
 	c = &cpu_data(smp_processor_id());
 	if (cpu_has(c, X86_FEATURE_MSR_IA32_FEAT_CTL))
Signature · 1 line
2.17.1
#9 lore raw

Re: [PATCH v3 08/15] x86/power/keylocker: Restore internal wrapping key from the ACPI S3/4 sleep states

Quoted 6 lines · Chang S. Bae
arch/x86/include/asm/keylocker.h +0 -0
index 820ac29c06d9..4000a5eed2c2 100644
@@ -32,9 +32,13 @@ struct iwkey {
 #ifdef CONFIG_X86_KEYLOCKER

arch/x86/include/asm/keylocker.h struct iwkey { · line 42

Quoted 10 lines
arch/x86/include/asm/keylocker.h +4 -0
 void setup_keylocker(struct cpuinfo_x86 *c);
 void destroy_keylocker_data(void);
+void restore_keylocker(void);
+extern bool valid_keylocker(void);
 #else
 #define setup_keylocker(c) do { } while (0)
 #define destroy_keylocker_data() do { } while (0)
+#define restore_keylocker() do { } while (0)
+static inline bool valid_keylocker { return false; }
 #endif

This breaks the build when !CONFIG_X86_KEYLOCKER.

- Eric

#10 lore raw

PATCH v3 8/15 x86/power/keylocker: Restore internal wrapping key from the ACPI S3/4 sleep states

When the system state switches to these sleep states, the internal wrapping key gets reset in the CPU state.

The primary use case for the feature is bare metal dm-crypt. The key needs to be restored properly on wakeup, as dm-crypt does not prompt for the key on resume from suspend. Even the prompt it does perform for unlocking the volume where the hibernation image is stored, it still expects to reuse the key handles within the hibernation image once it is loaded. So it is motivated to meet dm-crypt's expectation that the key handles in the suspend-image remain valid after resume from an S-state.

Key Locker provides a mechanism to back up the internal wrapping key in non-volatile storage. The kernel requests a backup right after the key is loaded at boot time. It is copied back to each CPU upon wakeup.

While the backup may be maintained in NVM across S5 and G3 "off" states it is not architecturally guaranteed, nor is it expected by dm-crypt which expects to prompt for the key each time the volume is started.

The entirety of Key Locker needs to be disabled if the backup mechanism is not available unless CONFIG_SUSPEND=n, otherwise dm-crypt requires the backup to be available.

In the event of a key restore failure the kernel proceeds with an initialized IWKey state. This has the effect of invalidating any key handles that might be present in a suspend-image. When this happens dm-crypt will see I/O errors resulting from error returns from crypto_skcipher_{en,de}crypt(). While this will disrupt operations in the current boot, data is not at risk and access is restored at the next reboot to create new handles relative to the current IWKey.

Manage a feature-specific flag to communicate with the crypto implementation. This ensures to stop using the AES instructions upon the key restore failure while not turning off the feature.

---
Changes from v3:
* Fix the build error. (Eric Biggers)

Changes from RFC v2:
* Change the backup key failure handling. (Dan Williams)

Changes from RFC v1:
* Folded the warning message into the if condition check. (Rafael Wysocki)
* Rebased on the changes of the previous patches.
* Added error code for key restoration failures.
* Moved the restore helper.
* Added function descriptions.

---
 arch/x86/include/asm/keylocker.h |   4 +
 arch/x86/kernel/keylocker.c      | 124 ++++++++++++++++++++++++++++++-
 arch/x86/power/cpu.c             |   2 +
 3 files changed, 128 insertions(+), 2 deletions(-)
arch/x86/include/asm/keylocker.h +4 -0
index 820ac29c06d9..4000a5eed2c2 100644
@@ -32,9 +32,13 @@ struct iwkey {
 #ifdef CONFIG_X86_KEYLOCKER
 void setup_keylocker(struct cpuinfo_x86 *c);
 void destroy_keylocker_data(void);
+void restore_keylocker(void);
+extern bool valid_keylocker(void);
 #else
 #define setup_keylocker(c) do { } while (0)
 #define destroy_keylocker_data() do { } while (0)
+#define restore_keylocker() do { } while (0)
+static inline bool valid_keylocker() { return false; }
 #endif
 
 #endif /*__ASSEMBLY__ */
arch/x86/kernel/keylocker.c +122 -2

164 lines not on this page · raw message

arch/x86/power/cpu.c +2 -0
index 9f2b251e83c5..1a290f529c73 100644
@@ -25,6 +25,7 @@
 #include <asm/cpu.h>
 #include <asm/mmu_context.h>
 #include <asm/cpu_device_id.h>
+#include <asm/keylocker.h>
 
 #ifdef CONFIG_X86_32
 __visible unsigned long saved_context_ebx;
@@ -262,6 +263,7 @@ static void notrace __restore_processor_state(struct saved_context *ctxt)
 	mtrr_bp_restore();
 	perf_restore_debug_store();
 	msr_restore_context(ctxt);
+	restore_keylocker();
 
 	c = &cpu_data(smp_processor_id());
 	if (cpu_has(c, X86_FEATURE_MSR_IA32_FEAT_CTL))
Signature · 1 line
2.17.1
#11 lore raw

Re: [PATCH v3 08/15] x86/power/keylocker: Restore internal wrapping key from the ACPI S3/4 sleep states

In reply to Eric Biggers
Quoted 11 lines · Eric Biggers
Quoted 10 lines · Chang S. Bae
arch/x86/include/asm/keylocker.h +2 -0
index 820ac29c06d9..4000a5eed2c2 100644
@@ -32,9 +32,13 @@ struct iwkey {
 #ifdef CONFIG_X86_KEYLOCKER
 void setup_keylocker(struct cpuinfo_x86 *c);
 void destroy_keylocker_data(void);
+void restore_keylocker(void);
+extern bool valid_keylocker(void);
Quoted 8 lines

arch/x86/include/asm/keylocker.h struct iwkey { · line 42

Quoted 6 lines
arch/x86/include/asm/keylocker.h +2 -0
 #else
 #define setup_keylocker(c) do { } while (0)
 #define destroy_keylocker_data() do { } while (0)
+#define restore_keylocker() do { } while (0)
+static inline bool valid_keylocker { return false; }
 #endif

This breaks the build when !CONFIG_X86_KEYLOCKER.

Ah, sorry. My bad on the last-minute change.

Thanks,
Chang

#12 lore raw

PATCH v3 9/15 x86/cpu: Add a configuration and command line option for Key Locker

In reply to Chang S. Bae

Add CONFIG_X86_KEYLOCKER to gate whether Key Locker is initialized at boot. The option is selected by the Key Locker cipher module CRYPTO_AES_KL (to be added in a later patch).

Add a new command line option "nokeylocker" to optionally override the default CONFIG_X86_KEYLOCKER=y behavior.

---
Changes from RFC v2:
* Make the option selected by CRYPTO_AES_KL. (Dan Williams)
* Massage the changelog and the config option description.

---
 Documentation/admin-guide/kernel-parameters.txt |  2 ++
 arch/x86/Kconfig                                |  3 +++
 arch/x86/kernel/cpu/common.c                    | 16 ++++++++++++++++
 3 files changed, 21 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +2 -0
index 9725c546a0d4..336495722b12 100644
@@ -3399,6 +3399,8 @@
 
 	nohugevmalloc	[PPC] Disable kernel huge vmalloc mappings.
 
+	nokeylocker	[X86] Disable Key Locker hardware feature.
+
 	nosmt		[KNL,S390] Disable symmetric multithreading (SMT).
 			Equivalent to smt=1.
 
arch/x86/Kconfig +3 -0
index b9281fab4e3e..b0050e8cf723 100644
@@ -1864,6 +1864,9 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS
 
 	  If unsure, say y.
 
+config X86_KEYLOCKER
+	bool
+
 choice
 	prompt "TSX enable mode"
 	depends on CPU_SUP_INTEL
arch/x86/kernel/cpu/common.c +16 -0
index 23b4aa437c1e..db1fc9ff0fe3 100644
@@ -364,6 +364,22 @@ static __always_inline void setup_umip(struct cpuinfo_x86 *c)
 /* These bits should not change their value after CPU init is finished. */
 static const unsigned long cr4_pinned_mask =
 	X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP | X86_CR4_FSGSBASE;
+
+static __init int x86_nokeylocker_setup(char *arg)
+{
+	/* Expect an exact match without trailing characters. */
+	if (strlen(arg))
+		return 0;
+
+	if (!cpu_feature_enabled(X86_FEATURE_KEYLOCKER))
+		return 1;
+
+	setup_clear_cpu_cap(X86_FEATURE_KEYLOCKER);
+	pr_info("x86/keylocker: Disabled by kernel command line.\n");
+	return 1;
+}
+__setup("nokeylocker", x86_nokeylocker_setup);
+
 static DEFINE_STATIC_KEY_FALSE_RO(cr_pinning);
 static unsigned long cr4_pinned_bits __ro_after_init;
 
Signature · 1 line
2.17.1
#13 lore raw

PATCH v3 10/15 crypto: x86/aes - Prepare for a new AES implementation

In reply to Chang S. Bae

Key Locker's AES instruction set ('AES-KL') has a similar programming interface to AES-NI. The internal ABI in the assembly code will have the same prototype as AES-NI. Then, the glue code will be the same as AES-NI's.

Refactor the common C code to avoid code duplication. The AES-NI code uses it with a function pointer argument to call back the AES-NI assembly code. So will the AES-KL code.

Introduce wrappers for data transformation functions to return an error value. AES-KL may populate an error during data transformation.

Also refactor some constant values in the AES-NI's assembly code.

No functional change intended.

---
Changes from RFC v2:
* Massage the changelog. (Dan Williams)

Changes from RFC v1:
* Added as a new patch. (Ard Biesheuvel)

---
 arch/x86/crypto/Makefile           |   2 +-
 arch/x86/crypto/aes-intel_asm.S    |  26 +++
 arch/x86/crypto/aes-intel_glue.c   | 219 ++++++++++++++++++++
 arch/x86/crypto/aes-intel_glue.h   |  61 ++++++
 arch/x86/crypto/aesni-intel_asm.S  |  90 ++++-----
 arch/x86/crypto/aesni-intel_glue.c | 313 +++--------------------------
 arch/x86/crypto/aesni-intel_glue.h |  88 ++++++++
 7 files changed, 463 insertions(+), 336 deletions(-)
 create mode 100644 arch/x86/crypto/aes-intel_asm.S
 create mode 100644 arch/x86/crypto/aes-intel_glue.c
 create mode 100644 arch/x86/crypto/aes-intel_glue.h
 create mode 100644 arch/x86/crypto/aesni-intel_glue.h
arch/x86/crypto/Makefile +1 -1
index f307c93fc90a..ef6c0b9f69c6 100644
@@ -47,7 +47,7 @@ chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha_glue.o
 chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o
 
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
-aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
+aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o
 aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
 
 obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
arch/x86/crypto/aes-intel_asm.S new file +26 -0
new file mode 100644
index 000000000000..98abf875af79
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+/*
+ * Constant values shared between AES implementations:
+ */
+
+.pushsection .rodata
+.align 16
+.Lcts_permute_table:
+	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+	.byte		0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07
+	.byte		0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f
+	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+	.byte		0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+#ifdef __x86_64__
+.Lbswap_mask:
+	.byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0
+#endif
+.popsection
+
+.section	.rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16
+.align 16
+.Lgf128mul_x_ble_mask:
+	.octa 0x00000000000000010000000000000087
+.previous
arch/x86/crypto/aes-intel_glue.c new file +219 -0

219 lines not on this page · raw message

arch/x86/crypto/aes-intel_glue.h new file +61 -0

61 lines not on this page · raw message

arch/x86/crypto/aesni-intel_asm.S +35 -55

203 lines not on this page · raw message

arch/x86/crypto/aesni-intel_glue.c +33 -280

465 lines not on this page · raw message

arch/x86/crypto/aesni-intel_glue.h new file +88 -0

88 lines not on this page · raw message

Signature · 1 line
2.17.1
#14 lore raw

PATCH v3 11/15 crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions

In reply to Chang S. Bae

Key Locker is a CPU feature to reduce key exfiltration opportunities while maintaining a programming interface similar to AES-NI. It converts the AES key into an encoded form, called the 'key handle'.

The key handle is a wrapped version of the clear-text key where the wrapping key has limited exposure. Once converted via setkey(), all subsequent data encryption using new AES instructions ('AES-KL') uses this key handle, reducing the exposure of private key material in memory.

AES-KL is analogous to that of AES-NI. Most assembly code is translated from the AES-NI code. They are operational in both 32-bit and 64-bit modes like AES-NI. However, users need to be aware of the following differences:

== Key Handle ==

AES-KL may fail with an invalid key handle. It could be corrupted or fail with handle restriction. A key handle may be encoded with some restrictions. The implementation restricts every handle only available in kernel mode via setkey().

=== AES Compliance ===

Key Locker is not AES compliant in that it lacks support for 192-bit keys. However, per the expectations of Linux crypto-cipher implementations the software cipher implementation must support all the AES compliant key-sizes. The AES-KL cipher implementation achieves this constraint by logging a warning and falling back to AES-NI. In other words the 192-bit key-size limitation for what can be converted into a Key Locker key-handle is only documented, not enforced. This along with the below performance and failure mode implications is an end-user consideration for selecting AES-KL vs AES-NI.

== API Limitation ==

The setkey() function transforms an AES key to a handle. An extended key is a usual outcome of setkey() in other AES cipher implementations. For this reason, a setkey() failure does not fall back to the other. So, AES-KL will be exposed via synchronous interfaces only.

== Wrapping-key Restore Failure ==

The setkey() failure is also possible with the wrapping-key restore failure. In an event of hardware failure, the wrapping key is lost from deep sleep states. Then, setkey() will fail with the ENODEV error. And the suspended data transforming task may resume but will fail due to the wrapping key change.

== Performance ==

This feature comes with some performance penalties vs AES-NI. The cryptsetup benchmark indicates Key Locker raw throughput can be ~5x slower than AES-NI. For disk encryption, storage bandwidth may be the bottleneck before encryption bandwidth, but the potential performance difference is why AES-KL is advertised as a distinct cipher in /proc/crypto rather than the kernel transparently replacing AES-NI usage with AES-KL.

---
Changes from RFC v2:
* Move out each mode support in new patches.
* Update the changelog to describe the limitation and the tradeoff clearly. (Andy Lutomirski)

Changes from RFC v1:
* Rebased on the refactored code. (Ard Biesheuvel)
* Dropped exporting the single block interface. (Ard Biesheuvel)
* Fixed the fallback and error handling paths. (Ard Biesheuvel)
* Revised the module description. (Dave Hansen and Peter Zijlsta)
* Made the build depend on the binutils version to support new instructions. (Borislav Petkov and Peter Zijlstra)
* Updated the changelog accordingly.

---
 arch/x86/crypto/Makefile           |   3 +
 arch/x86/crypto/aeskl-intel_asm.S  | 184 +++++++++++++++++++++++++++++
 arch/x86/crypto/aeskl-intel_glue.c | 115 ++++++++++++++++++
 crypto/Kconfig                     |  44 +++++++
 4 files changed, 346 insertions(+)
 create mode 100644 arch/x86/crypto/aeskl-intel_asm.S
 create mode 100644 arch/x86/crypto/aeskl-intel_glue.c
arch/x86/crypto/Makefile +3 -0
index ef6c0b9f69c6..f696b037faa5 100644
@@ -50,6 +50,9 @@ obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o
 aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
 
+obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-intel.o
+aeskl-intel-y := aeskl-intel_asm.o aesni-intel_asm.o aeskl-intel_glue.o aes-intel_glue.o
+
 obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
 sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
 sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
arch/x86/crypto/aeskl-intel_asm.S new file +184 -0

184 lines not on this page · raw message

arch/x86/crypto/aeskl-intel_glue.c new file +115 -0

115 lines not on this page · raw message

crypto/Kconfig +44 -0

50 lines not on this page · raw message

Signature · 1 line
2.17.1
#15 lore raw

Re: [PATCH v3 11/15] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions

arch/x86/crypto/Makefile obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o · line 54

Quoted 10 lines · Chang S. Bae
arch/x86/crypto/Makefile +2 -0
index ef6c0b9f69c6..f696b037faa5 100644
@@ -50,6 +50,9 @@ obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o
 aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
 
+obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-intel.o
+aeskl-intel-y := aeskl-intel_asm.o aesni-intel_asm.o aeskl-intel_glue.o aes-intel_glue.o

This makes the object files aesni-intel_asm.o and aes-intel_glue.o each be built into two separate kernel modules. My understanding is that duplicating code like that is generally frowned upon. These files should either be built into a separate module, which both aesni-intel.ko and aeskl-intel.ko would depend on, or aeskl-intel.ko should depend on aesni-intel.ko.

arch/x86/crypto/aeskl-intel_asm.S

Quoted 5 lines
arch/x86/crypto/aeskl-intel_asm.S new file +0 -0
new file mode 100644
index 000000000000..d56ec8dd6644

This file gets very long after all the modes are added (> 1100 lines). Is there really no feasible way to share code between this and aesni-intel_asm.S, similar to how the arm64 AES implementations work? Surely most of the logic is the same, and it's just the actual AES instructions that differ?

arch/x86/crypto/aeskl-intel_asm.S

Quoted 4 lines
arch/x86/crypto/aeskl-intel_asm.S new file +4 -0
+config CRYPTO_AES_KL
+	tristate "AES cipher algorithms (AES-KL)"
+	depends on (LD_VERSION >= 23600) || (LLD_VERSION >= 120000)
+	depends on DM_CRYPT

'depends on DM_CRYPT' doesn't really make sense here, since there is no actual dependency on dm-crypt in the code.

- Eric

#16 lore raw

Re: [PATCH v3 11/15] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions

Quoted 9 lines · Eric Biggers
Quoted 8 lines · Chang S. Bae
arch/x86/crypto/Makefile +0 -0
index ef6c0b9f69c6..f696b037faa5 100644
@@ -50,6 +50,9 @@ obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o

aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o

Quoted 8 lines

arch/x86/crypto/Makefile obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o · line 54

Quoted 2 lines
arch/x86/crypto/Makefile +2 -0
+obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-intel.o
+aeskl-intel-y := aeskl-intel_asm.o aesni-intel_asm.o aeskl-intel_glue.o aes-intel_glue.o

This makes the object files aesni-intel_asm.o and aes-intel_glue.o each be built into two separate kernel modules. My understanding is that duplicating code like that is generally frowned upon. These files should either be built into a separate module, which both aesni-intel.ko and aeskl-intel.ko would depend on, or aeskl-intel.ko should depend on aesni-intel.ko.

The only reason to include the AES-NI object here is that AES-KL does not support the 192-bit key.

Maybe the fallback can be the aes-generic driver [1] instead of AES-NI here.

Quoted 10 lines

arch/x86/crypto/aeskl-intel_asm.S

Quoted 5 lines
arch/x86/crypto/aeskl-intel_asm.S new file +0 -0
new file mode 100644
index 000000000000..d56ec8dd6644

This file gets very long after all the modes are added (> 1100 lines). Is there really no feasible way to share code between this and aesni-intel_asm.S, similar to how the arm64 AES implementations work? Surely most of the logic is the same, and it's just the actual AES instructions that differ?

No, these two instruction sets are separate. So I think no room to share the ASM code.

Quoted 7 lines

arch/x86/crypto/aeskl-intel_asm.S

Quoted 4 lines
arch/x86/crypto/aeskl-intel_asm.S new file +4 -0
+config CRYPTO_AES_KL
+	tristate "AES cipher algorithms (AES-KL)"
+	depends on (LD_VERSION >= 23600) || (LLD_VERSION >= 120000)
+	depends on DM_CRYPT

'depends on DM_CRYPT' doesn't really make sense here, since there is no actual dependency on dm-crypt in the code.

I think the intention here is to build a policy that the library is available only when there is a clear use case.

But maybe putting such restriction is too much here.

Thanks
Chang

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/crypto/aes_generic.c

#17 lore raw

Re: [PATCH v3 11/15] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions

Quoted 44 lines · Chang Seok
Quoted 9 lines · Eric Biggers
Quoted 8 lines · Chang S. Bae
arch/x86/crypto/Makefile +0 -0
index ef6c0b9f69c6..f696b037faa5 100644
@@ -50,6 +50,9 @@ obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o

aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o

Quoted 8 lines

arch/x86/crypto/Makefile obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o · line 54

Quoted 2 lines
arch/x86/crypto/Makefile +2 -0
+obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-intel.o
+aeskl-intel-y := aeskl-intel_asm.o aesni-intel_asm.o aeskl-intel_glue.o aes-intel_glue.o

This makes the object files aesni-intel_asm.o and aes-intel_glue.o each be built into two separate kernel modules. My understanding is that duplicating code like that is generally frowned upon. These files should either be built into a separate module, which both aesni-intel.ko and aeskl-intel.ko would depend on, or aeskl-intel.ko should depend on aesni-intel.ko.

The only reason to include the AES-NI object here is that AES-KL does not support the 192-bit key.

Maybe the fallback can be the aes-generic driver [1] instead of AES-NI here.

Quoted 10 lines

arch/x86/crypto/aeskl-intel_asm.S

Quoted 5 lines
arch/x86/crypto/aeskl-intel_asm.S new file +0 -0
new file mode 100644
index 000000000000..d56ec8dd6644

This file gets very long after all the modes are added (> 1100 lines). Is there really no feasible way to share code between this and aesni-intel_asm.S, similar to how the arm64 AES implementations work? Surely most of the logic is the same, and it's just the actual AES instructions that differ?

No, these two instruction sets are separate. So I think no room to share the ASM code.

Quoted 5 lines
Quoted 4 lines
arch/x86/crypto/aeskl-intel_asm.S new file +4 -0
+config CRYPTO_AES_KL
+    tristate "AES cipher algorithms (AES-KL)"
+    depends on (LD_VERSION >= 23600) || (LLD_VERSION >= 120000)
+    depends on DM_CRYPT
Quoted 8 lines
Quoted 2 lines

'depends on DM_CRYPT' doesn't really make sense here, since there is no actual dependency on dm-crypt in the code.

I think the intention here is to build a policy that the library is available only when there is a clear use case.

But maybe putting such restriction is too much here.

Yeah, my bad the "depends on DM_CRYPT" can go. Even though the Key Locker support has no real pressing reason to be built without it, there is still no actual code dependency.

#18 lore raw

Re: [PATCH v3 11/15] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions

In reply to Bae, Chang Seok
Quoted 3 lines · Chang Seok
Quoted 1 line · Eric Biggers

On Wed, Nov 24, 2021 at 12:06:56PM -0800, Chang S. Bae wrote:

...

Quoted 11 lines
Quoted 10 lines

arch/x86/crypto/aeskl-intel_asm.S

Quoted 5 lines
arch/x86/crypto/aeskl-intel_asm.S new file +0 -0
new file mode 100644
index 000000000000..d56ec8dd6644

This file gets very long after all the modes are added (> 1100 lines). Is there really no feasible way to share code between this and aesni-intel_asm.S, similar to how the arm64 AES implementations work? Surely most of the logic is the same, and it's just the actual AES instructions that differ?

Quoted 3 lines

No, these two instruction sets are separate. So I think no room to share the ASM code.

On arm64, we have

aes-ce.S, which uses AES instructions to implement the AES core transforms

aes-neon.S, which uses plain NEON instructions to implement the AES core transforms

aes-modes.S, which can be combined with either of the above, and implements the various chaining modes (ECB, CBC, CTR, XTS, and a helper for CMAC, CBCMAC and XMAC)

If you have two different primitives for performing AES transforms (the original round by round one, and the KL one that does 10 or 14 rounds at a time), you should still be able to reuse most of the code that implements the non-trivial handling of the chaining modes.

Quoted 19 lines
Quoted 7 lines

arch/x86/crypto/aeskl-intel_asm.S

Quoted 4 lines
arch/x86/crypto/aeskl-intel_asm.S new file +4 -0
+config CRYPTO_AES_KL
+    tristate "AES cipher algorithms (AES-KL)"
+    depends on (LD_VERSION >= 23600) || (LLD_VERSION >= 120000)
+    depends on DM_CRYPT

'depends on DM_CRYPT' doesn't really make sense here, since there is no actual dependency on dm-crypt in the code.

I think the intention here is to build a policy that the library is available only when there is a clear use case.

But maybe putting such restriction is too much here.

Thanks
Chang

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/crypto/aes_generic.c

#19 lore raw

Re: [PATCH v3 11/15] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions

Quoted 13 lines · Ard Biesheuvel
Quoted 4 lines · Chang Seok

No, these two instruction sets are separate. So I think no room to share the ASM code.

On arm64, we have

aes-ce.S, which uses AES instructions to implement the AES core transforms

aes-neon.S, which uses plain NEON instructions to implement the AES core transforms

Quoted 8 lines

aes-modes.S, which can be combined with either of the above, and implements the various chaining modes (ECB, CBC, CTR, XTS, and a helper for CMAC, CBCMAC and XMAC)

If you have two different primitives for performing AES transforms (the original round by round one, and the KL one that does 10 or 14 rounds at a time), you should still be able to reuse most of the code that implements the non-trivial handling of the chaining modes.

Yes, no question about this for maintainability.

However, besides the fact that a KL instruction takes multiple rounds, some AES-KL instructions have register constraints. E.g. AESENCWIDE256KL always uses XMM0-7 for input blocks.

Today, AES-NI code maintains 32-bit compatibility, e.g. clobbering XMM2-3 for key and input vector, so sharing the code makes the AES-KL code inefficient and even ugly I think due to the register constraint. E.g. the AES-KL code does use XMM9-10 for key and an input vector, but it has to move them around just for code sharing.

Thanks,
Chang

#20 lore raw

Re: [PATCH v3 11/15] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions

In reply to Chang S. Bae

arch/x86/crypto/aeskl-intel_asm.S line 88

Quoted 1 line · Chang S. Bae
arch/x86/crypto/aeskl-intel_asm.S new file +1 -0
+	encodekey256 %eax, %eax

So this thing uses the fancy new keylocker instructions, however:

Quoted 5 lines
crypto/Kconfig +0 -0
index 285f82647d2b..784a04433549 100644

crypto/Kconfig config CRYPTO_AES_NI_INTEL · line 1121

Quoted 9 lines
crypto/Kconfig +6 -0
@@ -1113,6 +1113,50 @@ config CRYPTO_AES_NI_INTEL
 	  ECB, CBC, LRW, XTS. The 64 bit version has additional
 	  acceleration for CTR.
 
+config CRYPTO_AES_KL
+	tristate "AES cipher algorithms (AES-KL)"
+	depends on (LD_VERSION >= 23600) || (LLD_VERSION >= 120000)
+	depends on DM_CRYPT
+	select X86_KEYLOCKER
+	select CRYPTO_AES_NI_INTEL

There is no dependency on the compiler actually supporting them..

config AS_HAS_KEYLOCKER

	def_bool $(as-instr,encodekey256)

	depends on AS_HAS_KEYLOCKER

Hmm?

Quoted 44 lines
crypto/Kconfig +38 -0
+
+	help
+	  Key Locker provides AES SIMD instructions (AES-KL) for secure
+	  data encryption and decryption. While this new instruction
+	  set is analogous to AES-NI, AES-KL supports to encode an AES
+	  key to an encoded form ('key handle') and uses it to transform
+	  data instead of accessing the AES key.
+
+	  The setkey() transforms an AES key to a key handle, then the AES
+	  key is no longer needed for data transformation. A user may
+	  displace their keys from possible exposition.
+
+	  This key encryption is done by the CPU-internal wrapping key. The
+	  x86 core code loads a new random key at every boot time and
+	  restores it from deep sleep states. This wrapping key support is
+	  provided with X86_KEYLOCKER.
+
+	  AES-KL supports 128-/256-bit keys only. While giving a 192-bit
+	  key does not return an error, as AES-NI is chosen to process it,
+	  the claimed security property is not available with that.
+
+	  GNU binutils version 2.36 or above and LLVM version 12 or above
+	  are assemblers that support AES-KL instructions.
+
+	  Bare metal disk encryption is the preferred use case. Make it
+	  depend on DM_CRYPT.
+
+	  This selection enables an alternative crypto cipher for
+	  cryptsetup, e.g. "capi:xts-aes-aeskl-plain", to use with dm-crypt
+	  volumes. It trades off raw performance for reduced clear-text key
+	  exposure and has an additional failure mode compared to AES-NI.
+	  See Documentation/x86/keylocker.rst for more details. Key Locker
+	  usage requires explicit opt-in at cryptsetup time. So, select it
+	  if unsure.
+
+	  See also the CRYPTO_AES_NI_INTEL description for more about the
+	  AES cipher algorithm.
+
 config CRYPTO_AES_SPARC64
 	tristate "AES cipher algorithms (SPARC64)"
 	depends on SPARC64
Signature · 1 line
2.17.1
#21 lore raw

Re: [PATCH v3 11/15] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions

Quoted 23 lines · Peter Zijlstra

arch/x86/crypto/aeskl-intel_asm.S line 88

Quoted 1 line · Chang S. Bae
arch/x86/crypto/aeskl-intel_asm.S new file +1 -0
+	encodekey256 %eax, %eax

So this thing uses the fancy new keylocker instructions, however:

Quoted 5 lines
crypto/Kconfig +0 -0
index 285f82647d2b..784a04433549 100644

crypto/Kconfig config CRYPTO_AES_NI_INTEL · line 1121

Quoted 9 lines
crypto/Kconfig +6 -0
@@ -1113,6 +1113,50 @@ config CRYPTO_AES_NI_INTEL
 	  ECB, CBC, LRW, XTS. The 64 bit version has additional
 	  acceleration for CTR.
 
+config CRYPTO_AES_KL
+	tristate "AES cipher algorithms (AES-KL)"
+	depends on (LD_VERSION >= 23600) || (LLD_VERSION >= 120000)
+	depends on DM_CRYPT
+	select X86_KEYLOCKER
+	select CRYPTO_AES_NI_INTEL

There is no dependency on the compiler actually supporting them..

Quoted 6 lines

config AS_HAS_KEYLOCKER

	def_bool $(as-instr,encodekey256)

	depends on AS_HAS_KEYLOCKER

Hmm?

Well, LD_VERSION reflects the binutils version.

But yes, the as-instr macro looks to be useful here:

+config AS_HAS_KEYLOCKER
+ def_bool $(as-instr,encodekey256 %eax$(comma)%eax)
+ help
+ Supported by binutils >=2.36 and LLVM integrated assembler >= V12
+
config CRYPTO_AES_KL
tristate "AES cipher algorithms (AES-KL)"
- depends on (LD_VERSION >= 23600) || (LLD_VERSION >= 120000)
+ depends on AS_HAS_KEYLOCKER

Thanks,
Chang

#22 lore raw

PATCH v3 12/15 crypto: x86/aes-kl - Support ECB mode

In reply to Chang S. Bae

Implement ECB mode using AES-KL. Export the methods with a lower priority than AES-NI to avoid from selected by default.

---
Changes from RFC v2:
* Separate out the code as a new patch.

---
 arch/x86/crypto/aeskl-intel_asm.S  | 193 +++++++++++++++++++++++++++++
 arch/x86/crypto/aeskl-intel_glue.c |  92 +++++++++++++-
 2 files changed, 284 insertions(+), 1 deletion(-)
arch/x86/crypto/aeskl-intel_asm.S +193 -0
index d56ec8dd6644..833bb39ae903 100644
@@ -182,3 +182,196 @@ SYM_FUNC_START(_aeskl_dec)
 	ret
 SYM_FUNC_END(_aeskl_dec)
 
+/*
+ * int _aeskl_ecb_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
+ *		      size_t len)
+ */
+SYM_FUNC_START(_aeskl_ecb_enc)
+	FRAME_BEGIN
+#ifndef __x86_64__
+	pushl LEN
+	pushl HANDLEP
+	pushl KLEN
+	movl (FRAME_OFFSET+16)(%esp), HANDLEP	# ctx
+	movl (FRAME_OFFSET+20)(%esp), OUTP	# dst
+	movl (FRAME_OFFSET+24)(%esp), INP	# src
+	movl (FRAME_OFFSET+28)(%esp), LEN	# len
+#endif
+	test LEN, LEN
+	jz .Lecb_enc_noerr
+	mov 480(HANDLEP), KLEN
+	cmp $16, LEN
+	jb .Lecb_enc_noerr
+	cmp $128, LEN
+	jb .Lecb_enc1
+
+.align 4
+.Lecb_enc8:
+	movdqu (INP), STATE1
+	movdqu 0x10(INP), STATE2
+	movdqu 0x20(INP), STATE3
+	movdqu 0x30(INP), STATE4
+	movdqu 0x40(INP), STATE5
+	movdqu 0x50(INP), STATE6
+	movdqu 0x60(INP), STATE7
+	movdqu 0x70(INP), STATE8
+
+	cmp $16, KLEN
+	je .Lecb_enc8_128
+	aesencwide256kl (HANDLEP)
+	jz .Lecb_enc_err
+	jmp .Lecb_enc8_end
+.Lecb_enc8_128:
+	aesencwide128kl (HANDLEP)
+	jz .Lecb_enc_err
+
+.Lecb_enc8_end:
+	movdqu STATE1, (OUTP)
+	movdqu STATE2, 0x10(OUTP)
+	movdqu STATE3, 0x20(OUTP)
+	movdqu STATE4, 0x30(OUTP)
+	movdqu STATE5, 0x40(OUTP)
+	movdqu STATE6, 0x50(OUTP)
+	movdqu STATE7, 0x60(OUTP)
+	movdqu STATE8, 0x70(OUTP)
+
+	sub $128, LEN
+	add $128, INP
+	add $128, OUTP
+	cmp $128, LEN
+	jge .Lecb_enc8
+	cmp $16, LEN
+	jb .Lecb_enc_noerr
+
+.align 4
+.Lecb_enc1:
+	movdqu (INP), STATE1
+	cmp $16, KLEN
+	je .Lecb_enc1_128
+	aesenc256kl (HANDLEP), STATE
+	jz .Lecb_enc_err
+	jmp .Lecb_enc1_end
+.Lecb_enc1_128:
+	aesenc128kl (HANDLEP), STATE
+	jz .Lecb_enc_err
+
+.Lecb_enc1_end:
+	movdqu STATE1, (OUTP)
+	sub $16, LEN
+	add $16, INP
+	add $16, OUTP
+	cmp $16, LEN
+	jge .Lecb_enc1
+
+.Lecb_enc_noerr:
+	xor AREG, AREG
+	jmp .Lecb_enc_end
+.Lecb_enc_err:
+	mov $1, AREG
+.Lecb_enc_end:
+#ifndef __x86_64__
+	popl KLEN
+	popl HANDLEP
+	popl LEN
+#endif
+	FRAME_END
+	ret
+SYM_FUNC_END(_aeskl_ecb_enc)
+
+/*
+ * int _aeskl_ecb_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
+ *		      size_t len)
+ */
+SYM_FUNC_START(_aeskl_ecb_dec)
+	FRAME_BEGIN
+#ifndef __x86_64__
+	pushl LEN
+	pushl HANDLEP
+	pushl KLEN
+	movl (FRAME_OFFSET+16)(%esp), HANDLEP	# ctx
+	movl (FRAME_OFFSET+20)(%esp), OUTP	# dst
+	movl (FRAME_OFFSET+24)(%esp), INP	# src
+	movl (FRAME_OFFSET+28)(%esp), LEN	# len
+#endif
+
+	test LEN, LEN
+	jz .Lecb_dec_noerr
+	mov 480(HANDLEP), KLEN
+	cmp $16, LEN
+	jb .Lecb_dec_noerr
+	cmp $128, LEN
+	jb .Lecb_dec1
+
+.align 4
+.Lecb_dec8:
+	movdqu (INP), STATE1
+	movdqu 0x10(INP), STATE2
+	movdqu 0x20(INP), STATE3
+	movdqu 0x30(INP), STATE4
+	movdqu 0x40(INP), STATE5
+	movdqu 0x50(INP), STATE6
+	movdqu 0x60(INP), STATE7
+	movdqu 0x70(INP), STATE8
+
+	cmp $16, KLEN
+	je .Lecb_dec8_128
+	aesdecwide256kl (HANDLEP)
+	jz .Lecb_dec_err
+	jmp .Lecb_dec8_end
+.Lecb_dec8_128:
+	aesdecwide128kl (HANDLEP)
+	jz .Lecb_dec_err
+
+.Lecb_dec8_end:
+	movdqu STATE1, (OUTP)
+	movdqu STATE2, 0x10(OUTP)
+	movdqu STATE3, 0x20(OUTP)
+	movdqu STATE4, 0x30(OUTP)
+	movdqu STATE5, 0x40(OUTP)
+	movdqu STATE6, 0x50(OUTP)
+	movdqu STATE7, 0x60(OUTP)
+	movdqu STATE8, 0x70(OUTP)
+
+	sub $128, LEN
+	add $128, INP
+	add $128, OUTP
+	cmp $128, LEN
+	jge .Lecb_dec8
+	cmp $16, LEN
+	jb .Lecb_dec_noerr
+
+.align 4
+.Lecb_dec1:
+	movdqu (INP), STATE1
+	cmp $16, KLEN
+	je .Lecb_dec1_128
+	aesdec256kl (HANDLEP), STATE
+	jz .Lecb_dec_err
+	jmp .Lecb_dec1_end
+.Lecb_dec1_128:
+	aesdec128kl (HANDLEP), STATE
+	jz .Lecb_dec_err
+
+.Lecb_dec1_end:
+	movdqu STATE1, (OUTP)
+	sub $16, LEN
+	add $16, INP
+	add $16, OUTP
+	cmp $16, LEN
+	jge .Lecb_dec1
+
+.Lecb_dec_noerr:
+	xor AREG, AREG
+	jmp .Lecb_dec_end
+.Lecb_dec_err:
+	mov $1, AREG
+.Lecb_dec_end:
+#ifndef __x86_64__
+	popl KLEN
+	popl HANDLEP
+	popl LEN
+#endif
+	FRAME_END
+	ret
+SYM_FUNC_END(_aeskl_ecb_dec)
+
arch/x86/crypto/aeskl-intel_glue.c +91 -1

120 lines not on this page · raw message

Signature · 1 line
2.17.1
#23 lore raw

PATCH v3 13/15 crypto: x86/aes-kl - Support CBC mode

In reply to Chang S. Bae

Implement CBC using AES-KL. Export the methods with a lower priority than AES-NI to avoid from selected by default.

---
Changes from RFC v2:
* Separate out the code as a new patch.

---
 arch/x86/crypto/aeskl-intel_asm.S  | 188 +++++++++++++++++++++++++++++
 arch/x86/crypto/aeskl-intel_glue.c |  67 ++++++++++
 2 files changed, 255 insertions(+)
arch/x86/crypto/aeskl-intel_asm.S +188 -0
index 833bb39ae903..5ee7b24ee3c8 100644
@@ -375,3 +375,191 @@ SYM_FUNC_START(_aeskl_ecb_dec)
 	ret
 SYM_FUNC_END(_aeskl_ecb_dec)
 
+/*
+ * int _aeskl_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
+ *		      size_t len, u8 *iv)
+ */
+SYM_FUNC_START(_aeskl_cbc_enc)
+	FRAME_BEGIN
+#ifndef __x86_64__
+	pushl IVP
+	pushl LEN
+	pushl HANDLEP
+	pushl KLEN
+	movl (FRAME_OFFSET+20)(%esp), HANDLEP	# ctx
+	movl (FRAME_OFFSET+24)(%esp), OUTP	# dst
+	movl (FRAME_OFFSET+28)(%esp), INP	# src
+	movl (FRAME_OFFSET+32)(%esp), LEN	# len
+	movl (FRAME_OFFSET+36)(%esp), IVP	# iv
+#endif
+
+	cmp $16, LEN
+	jb .Lcbc_enc_noerr
+	mov 480(HANDLEP), KLEN
+	movdqu (IVP), STATE
+
+.align 4
+.Lcbc_enc1:
+	movdqu (INP), IN
+	pxor IN, STATE
+
+	cmp $16, KLEN
+	je .Lcbc_enc1_128
+	aesenc256kl (HANDLEP), STATE
+	jz .Lcbc_enc_err
+	jmp .Lcbc_enc1_end
+.Lcbc_enc1_128:
+	aesenc128kl (HANDLEP), STATE
+	jz .Lcbc_enc_err
+
+.Lcbc_enc1_end:
+	movdqu STATE, (OUTP)
+	sub $16, LEN
+	add $16, INP
+	add $16, OUTP
+	cmp $16, LEN
+	jge .Lcbc_enc1
+	movdqu STATE, (IVP)
+
+.Lcbc_enc_noerr:
+	xor AREG, AREG
+	jmp .Lcbc_enc_end
+.Lcbc_enc_err:
+	mov $1, AREG
+.Lcbc_enc_end:
+#ifndef __x86_64__
+	popl KLEN
+	popl HANDLEP
+	popl LEN
+	popl IVP
+#endif
+	FRAME_END
+	ret
+SYM_FUNC_END(_aeskl_cbc_enc)
+
+/*
+ * int _aeskl_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
+ *		      size_t len, u8 *iv)
+ */
+SYM_FUNC_START(_aeskl_cbc_dec)
+	FRAME_BEGIN
+#ifndef __x86_64__
+	pushl IVP
+	pushl LEN
+	pushl HANDLEP
+	pushl KLEN
+	movl (FRAME_OFFSET+20)(%esp), HANDLEP	# ctx
+	movl (FRAME_OFFSET+24)(%esp), OUTP	# dst
+	movl (FRAME_OFFSET+28)(%esp), INP	# src
+	movl (FRAME_OFFSET+32)(%esp), LEN	# len
+	movl (FRAME_OFFSET+36)(%esp), IVP	# iv
+#endif
+
+	cmp $16, LEN
+	jb .Lcbc_dec_noerr
+	mov 480(HANDLEP), KLEN
+#ifdef __x86_64__
+	cmp $128, LEN
+	jb .Lcbc_dec1_pre
+
+.align 4
+.Lcbc_dec8:
+	movdqu 0x0(INP), STATE1
+	movdqu 0x10(INP), STATE2
+	movdqu 0x20(INP), STATE3
+	movdqu 0x30(INP), STATE4
+	movdqu 0x40(INP), STATE5
+	movdqu 0x50(INP), STATE6
+	movdqu 0x60(INP), STATE7
+	movdqu 0x70(INP), STATE8
+
+	movdqu (IVP), IN1
+	movdqa STATE1, IN2
+	movdqa STATE2, IN3
+	movdqa STATE3, IN4
+	movdqa STATE4, IN5
+	movdqa STATE5, IN6
+	movdqa STATE6, IN7
+	movdqa STATE7, IN8
+	movdqu STATE8, (IVP)
+
+	cmp $16, KLEN
+	je .Lcbc_dec8_128
+	aesdecwide256kl (HANDLEP)
+	jz .Lcbc_dec_err
+	jmp .Lcbc_dec8_end
+.Lcbc_dec8_128:
+	aesdecwide128kl (HANDLEP)
+	jz .Lcbc_dec_err
+
+.Lcbc_dec8_end:
+	pxor IN1, STATE1
+	pxor IN2, STATE2
+	pxor IN3, STATE3
+	pxor IN4, STATE4
+	pxor IN5, STATE5
+	pxor IN6, STATE6
+	pxor IN7, STATE7
+	pxor IN8, STATE8
+
+	movdqu STATE1, 0x0(OUTP)
+	movdqu STATE2, 0x10(OUTP)
+	movdqu STATE3, 0x20(OUTP)
+	movdqu STATE4, 0x30(OUTP)
+	movdqu STATE5, 0x40(OUTP)
+	movdqu STATE6, 0x50(OUTP)
+	movdqu STATE7, 0x60(OUTP)
+	movdqu STATE8, 0x70(OUTP)
+
+	sub $128, LEN
+	add $128, INP
+	add $128, OUTP
+	cmp $128, LEN
+	jge .Lcbc_dec8
+	cmp $16, LEN
+	jb .Lcbc_dec_noerr
+#endif
+
+.align 4
+.Lcbc_dec1_pre:
+	movdqu (IVP), STATE3
+.Lcbc_dec1:
+	movdqu (INP), STATE2
+	movdqa STATE2, STATE1
+
+	cmp $16, KLEN
+	je .Lcbc_dec1_128
+	aesdec256kl (HANDLEP), STATE1
+	jz .Lcbc_dec_err
+	jmp .Lcbc_dec1_end
+.Lcbc_dec1_128:
+	aesdec128kl (HANDLEP), STATE1
+	jz .Lcbc_dec_err
+
+.Lcbc_dec1_end:
+	pxor STATE3, STATE1
+	movdqu STATE1, (OUTP)
+	movdqa STATE2, STATE3
+	sub $16, LEN
+	add $16, INP
+	add $16, OUTP
+	cmp $16, LEN
+	jge .Lcbc_dec1
+	movdqu STATE3, (IVP)
+
+.Lcbc_dec_noerr:
+	xor AREG, AREG
+	jmp .Lcbc_dec_end
+.Lcbc_dec_err:
+	mov $1, AREG
+.Lcbc_dec_end:
+#ifndef __x86_64__
+	popl KLEN
+	popl HANDLEP
+	popl LEN
+	popl IVP
+#endif
+	FRAME_END
+	ret
+SYM_FUNC_END(_aeskl_cbc_dec)
+
arch/x86/crypto/aeskl-intel_glue.c +67 -0

91 lines not on this page · raw message

Signature · 1 line
2.17.1
#24 lore raw

PATCH v3 14/15 crypto: x86/aes-kl - Support CTR mode

In reply to Chang S. Bae

Implement CTR mode using AES-KL. Export the methods with a lower priority than AES-NI to avoid from selected by default.

---
Changes from RFC v2:
* Separate out the code as a new patch.

---
 arch/x86/crypto/aeskl-intel_asm.S  | 174 +++++++++++++++++++++++++++++
 arch/x86/crypto/aeskl-intel_glue.c |  55 +++++++++
 2 files changed, 229 insertions(+)
arch/x86/crypto/aeskl-intel_asm.S +174 -0
index 5ee7b24ee3c8..ffde0cd3dd42 100644
@@ -563,3 +563,177 @@ SYM_FUNC_START(_aeskl_cbc_dec)
 	ret
 SYM_FUNC_END(_aeskl_cbc_dec)
 
+#ifdef __x86_64__
+
+/*
+ * _aeskl_ctr_inc_init:	internal ABI
+ *	setup registers used by _aesni_inc
+ * input:
+ *	IV
+ * output:
+ *	CTR:	== IV, in little endian
+ *	TCTR_LOW: == lower qword of CTR
+ *	INC:	== 1, in little endian
+ *	BSWAP_MASK == endian swapping mask
+ */
+SYM_FUNC_START_LOCAL(_aeskl_ctr_inc_init)
+	movaps .Lbswap_mask, BSWAP_MASK
+	movaps IV, CTR
+	pshufb BSWAP_MASK, CTR
+	mov $1, TCTR_LOW
+	movq TCTR_LOW, INC
+	movq CTR, TCTR_LOW
+	ret
+SYM_FUNC_END(_aeskl_ctr_inc_init)
+
+/*
+ * _aeskl_ctr_inc:		internal ABI
+ *	Increase IV by 1, IV is in big endian
+ * input:
+ *	IV
+ *	CTR:	== IV, in little endian
+ *	TCTR_LOW: == lower qword of CTR
+ *	INC:	== 1, in little endian
+ *	BSWAP_MASK == endian swapping mask
+ * output:
+ *	IV:	Increase by 1
+ * changed:
+ *	CTR:	== output IV, in little endian
+ *	TCTR_LOW: == lower qword of CTR
+ */
+SYM_FUNC_START_LOCAL(_aeskl_ctr_inc)
+	paddq INC, CTR
+	add $1, TCTR_LOW
+	jnc .Linc_low
+	pslldq $8, INC
+	paddq INC, CTR
+	psrldq $8, INC
+.Linc_low:
+	movaps CTR, IV
+	pshufb BSWAP_MASK, IV
+	ret
+SYM_FUNC_END(_aeskl_ctr_inc)
+
+/*
+ * CTR implementations
+ */
+
+/*
+ * int _aeskl_ctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
+ *		      size_t len, u8 *iv)
+ */
+SYM_FUNC_START(_aeskl_ctr_enc)
+	FRAME_BEGIN
+	cmp $16, LEN
+	jb .Lctr_enc_noerr
+	mov 480(HANDLEP), KLEN
+	movdqu (IVP), IV
+	call _aeskl_ctr_inc_init
+	cmp $128, LEN
+	jb .Lctr_enc1
+
+.align 4
+.Lctr_enc8:
+	movaps IV, STATE1
+	call _aeskl_ctr_inc
+	movaps IV, STATE2
+	call _aeskl_ctr_inc
+	movaps IV, STATE3
+	call _aeskl_ctr_inc
+	movaps IV, STATE4
+	call _aeskl_ctr_inc
+	movaps IV, STATE5
+	call _aeskl_ctr_inc
+	movaps IV, STATE6
+	call _aeskl_ctr_inc
+	movaps IV, STATE7
+	call _aeskl_ctr_inc
+	movaps IV, STATE8
+	call _aeskl_ctr_inc
+
+	cmp $16, KLEN
+	je .Lctr_enc8_128
+	aesencwide256kl (%rdi)
+	jz .Lctr_enc_err
+	jmp .Lctr_enc8_end
+.Lctr_enc8_128:
+	aesencwide128kl (%rdi)
+	jz .Lctr_enc_err
+.Lctr_enc8_end:
+
+	movups (INP), IN1
+	pxor IN1, STATE1
+	movups STATE1, (OUTP)
+
+	movups 0x10(INP), IN1
+	pxor IN1, STATE2
+	movups STATE2, 0x10(OUTP)
+
+	movups 0x20(INP), IN1
+	pxor IN1, STATE3
+	movups STATE3, 0x20(OUTP)
+
+	movups 0x30(INP), IN1
+	pxor IN1, STATE4
+	movups STATE4, 0x30(OUTP)
+
+	movups 0x40(INP), IN1
+	pxor IN1, STATE5
+	movups STATE5, 0x40(OUTP)
+
+	movups 0x50(INP), IN1
+	pxor IN1, STATE6
+	movups STATE6, 0x50(OUTP)
+
+	movups 0x60(INP), IN1
+	pxor IN1, STATE7
+	movups STATE7, 0x60(OUTP)
+
+	movups 0x70(INP), IN1
+	pxor IN1, STATE8
+	movups STATE8, 0x70(OUTP)
+
+	sub $128, LEN
+	add $128, INP
+	add $128, OUTP
+	cmp $128, LEN
+	jge .Lctr_enc8
+	cmp $16, LEN
+	jb .Lctr_enc_end
+
+.align 4
+.Lctr_enc1:
+	movaps IV, STATE1
+	call _aeskl_ctr_inc
+
+	cmp $16, KLEN
+	je .Lctr_enc1_128
+	aesenc256kl (HANDLEP), STATE1
+	jz .Lctr_enc_err
+	jmp .Lctr_enc1_end
+.Lctr_enc1_128:
+	aesenc128kl (HANDLEP), STATE1
+	jz .Lctr_enc_err
+
+.Lctr_enc1_end:
+	movups (INP), IN1
+	pxor IN1, STATE1
+	movups STATE1, (OUTP)
+	sub $16, LEN
+	add $16, INP
+	add $16, OUTP
+	cmp $16, LEN
+	jge .Lctr_enc1
+
+.Lctr_enc_end:
+	movdqu IV, (IVP)
+.Lctr_enc_noerr:
+	xor AREG, AREG
+	jmp .Lctr_enc_ret
+.Lctr_enc_err:
+	mov $1, AREG
+.Lctr_enc_ret:
+	FRAME_END
+	ret
+SYM_FUNC_END(_aeskl_ctr_enc)
+
arch/x86/crypto/aeskl-intel_glue.c +55 -0

79 lines not on this page · raw message

Signature · 1 line
2.17.1
#25 lore raw

PATCH v3 15/15 crypto: x86/aes-kl - Support XTS mode

In reply to Chang S. Bae

Implement XTS mode using AES-KL. Export the methods with a lower priority than AES-NI to avoid from selected by default.

The assembly code clobbers more than eight 128-bit registers to make use of performant wide instructions that process eight 128-bit blocks at once. But that many 128-bit registers are not available in 32-bit mode, so support 64-bit mode only.

---
Changes from RFC v2:
* Separate out the code as a new patch.

---
 arch/x86/crypto/aeskl-intel_asm.S  | 447 +++++++++++++++++++++++++++++
 arch/x86/crypto/aeskl-intel_glue.c |  74 +++++
 2 files changed, 521 insertions(+)
arch/x86/crypto/aeskl-intel_asm.S +447 -0
index ffde0cd3dd42..d542324071d4 100644
@@ -737,3 +737,450 @@ SYM_FUNC_START(_aeskl_ctr_enc)
 	ret
 SYM_FUNC_END(_aeskl_ctr_enc)
 
+/*
+ * XTS implementation
+ */
+
+/*
+ * _aeskl_gf128mul_x_ble: 	internal ABI
+ *	Multiply in GF(2^128) for XTS IVs
+ * input:
+ *	IV:	current IV
+ *	GF128MUL_MASK == mask with 0x87 and 0x01
+ * output:
+ *	IV:	next IV
+ * changed:
+ *	CTR:	== temporary value
+ */
+#define _aeskl_gf128mul_x_ble() \
+	pshufd $0x13, IV, KEY; \
+	paddq IV, IV; \
+	psrad $31, KEY; \
+	pand GF128MUL_MASK, KEY; \
+	pxor KEY, IV;
+
+/*
+ * int _aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst,
+ *			  const u8 *src, unsigned int len, le128 *iv)
+ */
+SYM_FUNC_START(_aeskl_xts_encrypt)
+	FRAME_BEGIN
+	movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK
+	movups (IVP), IV
+
+	mov 480(HANDLEP), KLEN
+
+.Lxts_enc8:
+	sub $128, LEN
+	jl .Lxts_enc1_pre
+
+	movdqa IV, STATE1
+	movdqu (INP), INC
+	pxor INC, STATE1
+	movdqu IV, (OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE2
+	movdqu 0x10(INP), INC
+	pxor INC, STATE2
+	movdqu IV, 0x10(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE3
+	movdqu 0x20(INP), INC
+	pxor INC, STATE3
+	movdqu IV, 0x20(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE4
+	movdqu 0x30(INP), INC
+	pxor INC, STATE4
+	movdqu IV, 0x30(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE5
+	movdqu 0x40(INP), INC
+	pxor INC, STATE5
+	movdqu IV, 0x40(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE6
+	movdqu 0x50(INP), INC
+	pxor INC, STATE6
+	movdqu IV, 0x50(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE7
+	movdqu 0x60(INP), INC
+	pxor INC, STATE7
+	movdqu IV, 0x60(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE8
+	movdqu 0x70(INP), INC
+	pxor INC, STATE8
+	movdqu IV, 0x70(OUTP)
+
+	cmp $16, KLEN
+	je .Lxts_enc8_128
+	aesencwide256kl (%rdi)
+	jz .Lxts_enc_ret_err
+	jmp .Lxts_enc8_end
+.Lxts_enc8_128:
+	aesencwide128kl (%rdi)
+	jz .Lxts_enc_ret_err
+
+.Lxts_enc8_end:
+	movdqu 0x00(OUTP), INC
+	pxor INC, STATE1
+	movdqu STATE1, 0x00(OUTP)
+
+	movdqu 0x10(OUTP), INC
+	pxor INC, STATE2
+	movdqu STATE2, 0x10(OUTP)
+
+	movdqu 0x20(OUTP), INC
+	pxor INC, STATE3
+	movdqu STATE3, 0x20(OUTP)
+
+	movdqu 0x30(OUTP), INC
+	pxor INC, STATE4
+	movdqu STATE4, 0x30(OUTP)
+
+	movdqu 0x40(OUTP), INC
+	pxor INC, STATE5
+	movdqu STATE5, 0x40(OUTP)
+
+	movdqu 0x50(OUTP), INC
+	pxor INC, STATE6
+	movdqu STATE6, 0x50(OUTP)
+
+	movdqu 0x60(OUTP), INC
+	pxor INC, STATE7
+	movdqu STATE7, 0x60(OUTP)
+
+	movdqu 0x70(OUTP), INC
+	pxor INC, STATE8
+	movdqu STATE8, 0x70(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+
+	add $128, INP
+	add $128, OUTP
+	test LEN, LEN
+	jnz .Lxts_enc8
+
+.Lxts_enc_ret_iv:
+	movups IV, (IVP)
+.Lxts_enc_ret_noerr:
+	xor AREG, AREG
+	jmp .Lxts_enc_ret
+.Lxts_enc_ret_err:
+	mov $1, AREG
+.Lxts_enc_ret:
+	FRAME_END
+	ret
+
+.Lxts_enc1_pre:
+	add $128, LEN
+	jz .Lxts_enc_ret_iv
+	sub $16, LEN
+	jl .Lxts_enc_cts4
+
+.Lxts_enc1:
+	movdqu (INP), STATE1
+	pxor IV, STATE1
+
+	cmp $16, KLEN
+	je .Lxts_enc1_128
+	aesenc256kl (HANDLEP), STATE1
+	jz .Lxts_enc_ret_err
+	jmp .Lxts_enc1_end
+.Lxts_enc1_128:
+	aesenc128kl (HANDLEP), STATE1
+	jz .Lxts_enc_ret_err
+
+.Lxts_enc1_end:
+	pxor IV, STATE1
+	_aeskl_gf128mul_x_ble()
+
+	test LEN, LEN
+	jz .Lxts_enc1_out
+
+	add $16, INP
+	sub $16, LEN
+	jl .Lxts_enc_cts1
+
+	movdqu STATE1, (OUTP)
+	add $16, OUTP
+	jmp .Lxts_enc1
+
+.Lxts_enc1_out:
+	movdqu STATE1, (OUTP)
+	jmp .Lxts_enc_ret_iv
+
+.Lxts_enc_cts4:
+	movdqu STATE8, STATE1
+	sub $16, OUTP
+
+.Lxts_enc_cts1:
+	lea .Lcts_permute_table(%rip), T1
+	add LEN, INP		/* rewind input pointer */
+	add $16, LEN		/* # bytes in final block */
+	movups (INP), IN1
+
+	mov T1, IVP
+	add $32, IVP
+	add LEN, T1
+	sub LEN, IVP
+	add OUTP, LEN
+
+	movups (T1), STATE2
+	movaps STATE1, STATE3
+	pshufb STATE2, STATE1
+	movups STATE1, (LEN)
+
+	movups (IVP), STATE1
+	pshufb STATE1, IN1
+	pblendvb STATE3, IN1
+	movaps IN1, STATE1
+
+	pxor IV, STATE1
+
+	cmp $16, KLEN
+	je .Lxts_enc1_cts_128
+	aesenc256kl (HANDLEP), STATE1
+	jz .Lxts_enc_ret_err
+	jmp .Lxts_enc1_cts_end
+.Lxts_enc1_cts_128:
+	aesenc128kl (HANDLEP), STATE1
+	jz .Lxts_enc_ret_err
+
+.Lxts_enc1_cts_end:
+	pxor IV, STATE1
+	movups STATE1, (OUTP)
+	jmp .Lxts_enc_ret_noerr
+SYM_FUNC_END(_aeskl_xts_encrypt)
+
+/*
+ * int _aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst,
+ *			  const u8 *src, unsigned int len, le128 *iv)
+ */
+SYM_FUNC_START(_aeskl_xts_decrypt)
+	FRAME_BEGIN
+	movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK
+	movups (IVP), IV
+
+	mov 480(HANDLEP), KLEN
+
+	test $15, LEN
+	jz .Lxts_dec8
+	sub $16, LEN
+
+.Lxts_dec8:
+	sub $128, LEN
+	jl .Lxts_dec1_pre
+
+	movdqa IV, STATE1
+	movdqu (INP), INC
+	pxor INC, STATE1
+	movdqu IV, (OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE2
+	movdqu 0x10(INP), INC
+	pxor INC, STATE2
+	movdqu IV, 0x10(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE3
+	movdqu 0x20(INP), INC
+	pxor INC, STATE3
+	movdqu IV, 0x20(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE4
+	movdqu 0x30(INP), INC
+	pxor INC, STATE4
+	movdqu IV, 0x30(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE5
+	movdqu 0x40(INP), INC
+	pxor INC, STATE5
+	movdqu IV, 0x40(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE6
+	movdqu 0x50(INP), INC
+	pxor INC, STATE6
+	movdqu IV, 0x50(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE7
+	movdqu 0x60(INP), INC
+	pxor INC, STATE7
+	movdqu IV, 0x60(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+	movdqa IV, STATE8
+	movdqu 0x70(INP), INC
+	pxor INC, STATE8
+	movdqu IV, 0x70(OUTP)
+
+	cmp $16, KLEN
+	je .Lxts_dec8_128
+	aesdecwide256kl (%rdi)
+	jz .Lxts_dec_ret_err
+	jmp .Lxts_dec8_end
+.Lxts_dec8_128:
+	aesdecwide128kl (%rdi)
+	jz .Lxts_dec_ret_err
+
+.Lxts_dec8_end:
+	movdqu 0x00(OUTP), INC
+	pxor INC, STATE1
+	movdqu STATE1, 0x00(OUTP)
+
+	movdqu 0x10(OUTP), INC
+	pxor INC, STATE2
+	movdqu STATE2, 0x10(OUTP)
+
+	movdqu 0x20(OUTP), INC
+	pxor INC, STATE3
+	movdqu STATE3, 0x20(OUTP)
+
+	movdqu 0x30(OUTP), INC
+	pxor INC, STATE4
+	movdqu STATE4, 0x30(OUTP)
+
+	movdqu 0x40(OUTP), INC
+	pxor INC, STATE5
+	movdqu STATE5, 0x40(OUTP)
+
+	movdqu 0x50(OUTP), INC
+	pxor INC, STATE6
+	movdqu STATE6, 0x50(OUTP)
+
+	movdqu 0x60(OUTP), INC
+	pxor INC, STATE7
+	movdqu STATE7, 0x60(OUTP)
+
+	movdqu 0x70(OUTP), INC
+	pxor INC, STATE8
+	movdqu STATE8, 0x70(OUTP)
+
+	_aeskl_gf128mul_x_ble()
+
+	add $128, INP
+	add $128, OUTP
+	test LEN, LEN
+	jnz .Lxts_dec8
+
+.Lxts_dec_ret_iv:
+	movups IV, (IVP)
+.Lxts_dec_ret_noerr:
+	xor AREG, AREG
+	jmp .Lxts_dec_ret
+.Lxts_dec_ret_err:
+	mov $1, AREG
+.Lxts_dec_ret:
+	FRAME_END
+	ret
+
+.Lxts_dec1_pre:
+	add $128, LEN
+	jz .Lxts_dec_ret_iv
+
+.Lxts_dec1:
+	movdqu (INP), STATE1
+
+	add $16, INP
+	sub $16, LEN
+	jl .Lxts_dec_cts1
+
+	pxor IV, STATE1
+
+	cmp $16, KLEN
+	je .Lxts_dec1_128
+	aesdec256kl (HANDLEP), STATE1
+	jz .Lxts_dec_ret_err
+	jmp .Lxts_dec1_end
+.Lxts_dec1_128:
+	aesdec128kl (HANDLEP), STATE1
+	jz .Lxts_dec_ret_err
+
+.Lxts_dec1_end:
+	pxor IV, STATE1
+	_aeskl_gf128mul_x_ble()
+
+	test LEN, LEN
+	jz .Lxts_dec1_out
+
+	movdqu STATE1, (OUTP)
+	add $16, OUTP
+	jmp .Lxts_dec1
+
+.Lxts_dec1_out:
+	movdqu STATE1, (OUTP)
+	jmp .Lxts_dec_ret_iv
+
+.Lxts_dec_cts1:
+	movdqa IV, STATE5
+	_aeskl_gf128mul_x_ble()
+
+	pxor IV, STATE1
+
+	cmp $16, KLEN
+	je .Lxts_dec1_cts_pre_128
+	aesdec256kl (HANDLEP), STATE1
+	jz .Lxts_dec_ret_err
+	jmp .Lxts_dec1_cts_pre_end
+.Lxts_dec1_cts_pre_128:
+	aesdec128kl (HANDLEP), STATE1
+	jz .Lxts_dec_ret_err
+
+.Lxts_dec1_cts_pre_end:
+	pxor IV, STATE1
+
+	lea .Lcts_permute_table(%rip), T1
+	add LEN, INP		/* rewind input pointer */
+	add $16, LEN		/* # bytes in final block */
+	movups (INP), IN1
+
+	mov T1, IVP
+	add $32, IVP
+	add LEN, T1
+	sub LEN, IVP
+	add OUTP, LEN
+
+	movups (T1), STATE2
+	movaps STATE1, STATE3
+	pshufb STATE2, STATE1
+	movups STATE1, (LEN)
+
+	movups (IVP), STATE1
+	pshufb STATE1, IN1
+	pblendvb STATE3, IN1
+	movaps IN1, STATE1
+
+	pxor STATE5, STATE1
+
+	cmp $16, KLEN
+	je .Lxts_dec1_cts_128
+	aesdec256kl (HANDLEP), STATE1
+	jz .Lxts_dec_ret_err
+	jmp .Lxts_dec1_cts_end
+.Lxts_dec1_cts_128:
+	aesdec128kl (HANDLEP), STATE1
+	jz .Lxts_dec_ret_err
+
+.Lxts_dec1_cts_end:
+	pxor STATE5, STATE1
+
+	movups STATE1, (OUTP)
+	jmp .Lxts_dec_ret_noerr
+
+SYM_FUNC_END(_aeskl_xts_decrypt)
+
+#endif
arch/x86/crypto/aeskl-intel_glue.c +74 -0

98 lines not on this page · raw message

Signature · 1 line
2.17.1
#26 lore raw
In reply to Chang S. Bae
Quoted 4 lines · Chang S. Bae

== Non Use Cases ==

Bare metal disk encryption is the only use case intended by these patches.

If that's the case, why are so many encryption modes being added (ECB, CTR, CBC, and XTS)? Wouldn't just XTS be sufficient?

Quoted 4 lines

* PATCH10-15: For the x86 crypto library, it first prepares the AES-NI code to accommodate the new AES implementation. Then incrementally add base functions and various modes support -- ECB, CBC, CTR, and XTS. The code was found to pass the crypto test.

Did you test with CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y?

- Eric

#27 lore raw
Quoted 8 lines · Eric Biggers
Quoted 4 lines · Chang S. Bae

== Non Use Cases ==

Bare metal disk encryption is the only use case intended by these patches.

If that's the case, why are so many encryption modes being added (ECB, CTR, CBC, and XTS)? Wouldn't just XTS be sufficient?

Right, it would reduce the crypt library changes significantly. But it is clueless whether XTS is sufficient to support DM-crypt, because a user may select the kernel’s crypto API via ‘capi:', [1].

Quoted 6 lines
Quoted 4 lines

* PATCH10-15: For the x86 crypto library, it first prepares the AES-NI code to accommodate the new AES implementation. Then incrementally add base functions and various modes support -- ECB, CBC, CTR, and XTS. The code was found to pass the crypto test.

Did you test with CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y?

Yes.

Thanks,
Chang

[1] https://gitlab.com/cryptsetup/cryptsetup/-/wikis/DMCrypt

#28 lore raw
Quoted 5 lines · Chang Seok
Quoted 4 lines · Eric Biggers
Quoted 3 lines · Chang S. Bae

== Non Use Cases ==

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

Bare metal disk encryption is the only use case intended by these patches.

If that's the case, why are so many encryption modes being added (ECB, CTR, CBC, and XTS)? Wouldn't just XTS be sufficient?

Right, it would reduce the crypt library changes significantly. But it is clueless whether XTS is sufficient to support DM-crypt, because a user may select the kernel’s crypto API via ‘capi:', [1].

Just because dm-crypt allows you to create a ECB or CTR encrypted disk does not mean that it is a good idea.

- Eric

#29 lore raw
Quoted 11 lines · Eric Biggers
Quoted 10 lines · Chang Seok
Quoted 8 lines · Eric Biggers
Quoted 4 lines · Chang S. Bae

== Non Use Cases ==

Bare metal disk encryption is the only use case intended by these patches.

If that's the case, why are so many encryption modes being added (ECB, CTR, CBC, and XTS)? Wouldn't just XTS be sufficient?

Quoted 6 lines
Quoted 3 lines

Right, it would reduce the crypt library changes significantly. But it is clueless whether XTS is sufficient to support DM-crypt, because a user may select the kernel’s crypto API via ‘capi:', [1].

Just because dm-crypt allows you to create a ECB or CTR encrypted disk does not mean that it is a good idea.

Okay, it sounds like at least ECB and CTR supports are not practically needed.

I have no problem trimming down to exclude them.

Thanks,
Chang