KVM SGX virtualization support
Subsystems (6)
v2 cover KVM SGX virtualization support
--- Disclaimer ---
These patches were originally written by Sean Christopherson while at Intel. Now that Sean has left Intel, I (Kai) have taken over getting them upstream. This series needs more review before it can be merged. It is being posted publicly and under RFC so Sean and others can review it. Maintainers are safe ignoring it for now.
------------------
Hi all,
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Please help to review this series. Also I'd like to hear what is the proper way to merge this series, since it contains change to both x86/SGX and KVM subsystem. Any feedback is highly appreciated. And please let me know if I forgot to CC anyone, or anyone wants to be removed from CC. Thanks in advance!
This series is based against upstream v5.11-rc3. You can also get the code from upstream branch of kvm-sgx repo on github:
https://github.com/intel/kvm-sgx.git upstream
It also requires Qemu changes to create VM with SGX support. You can find Qemu repo here:
https://github.com/intel/qemu-sgx.git next
Please refer to README.md of above qemu-sgx repo for detail on how to create guest with SGX support. At meantime, for your quick reference you can use below command to create SGX guest:
#qemu-system-x86_64 -smp 4 -m 2G -drive file=<your_vm_image>,if=virtio \ -cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,prealloc
Please note that the SGX relevant part is:
-cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,prealloc
And you can change other parameters of your qemu command based on your needs.
=========
Changelog:
(Changelog here is for global changes. Please see each patch's changelog for
changes made to specific patch.)
v1->v2:
- Refined this cover letter by addressing comments from Dave and Jarkko.
- The original patch which introduced new X86_FEATURE_SGX1/SGX2 were replaced
by 3 new patches from Sean, following Boris and Sean's discussion.
[RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
[RFC PATCH v2 18/26] KVM: x86: Add support for reverse CPUID lookup of scattered features [RFC PATCH v2 19/26] KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
- The original patch 1
x86/sgx: Split out adding EPC page to free list to separate helper
was replaced with 2 new patches from Jarkko
[RFC PATCH v2 02/26] x86/sgx: Remove a warn from sgx_free_epc_page()
[RFC PATCH v2 03/26] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
addressing Jarkko's comments.
- Moved modifying sgx_init() to always initialize sgx_virt_epc_init() out of
patch
x86/sgx: Introduce virtual EPC for use by KVM guests
to a separate patch:
[RFC PATCH v2 07/26] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
to address Dave's comment that patch ordering can be improved due to before patch "Allow SGX virtualization without Launch Control support", all SGX, including SGX virtualization, is actually disabled when SGX LC is not
present.
KVM part patches are not changed comparing to v1 (except changes due to X86_FEATURE_SGX1/2 patches). For changes to each x86 patch, please see changelog in each indudival patch. If no changelog, then no change was made to it.
=========
KVM SGX virtualization Overview
- Virtual EPC
SGX enclave memory is special and is reserved specifically for enclave use. In bare-metal SGX enclaves, the kernel allocates enclave pages, copies data into the pages with privileged instructions, then allows the enclave to start. In this scenario, only initialized pages already assigned to an enclave are mapped to userspace.
In virtualized environments, the hypervisor still needs to do the physical enclave page allocation. The guest kernel is responsible for the data copying (among other things). This means that the job of starting an enclave is now split between hypervisor and guest.
This series introduces a new misc device: /dev/sgx_virt_epc. This device allows the host to map *uninitialized* enclave memory into userspace, which can then be passed into a guest.
While it might be *possible* to start a host-side enclave with /dev/sgx_enclave and pass its memory into a guest, it would be wasteful and convoluted.
Implement the *raw* EPC allocation in the x86 core-SGX subsystem via /dev/sgx_virt_epc rather than in KVM. Doing so has two major advantages:
- Does not require changes to KVM's uAPI, e.g. EPC gets handled as just another memory backend for guests.
- EPC management is wholly contained in the SGX subsystem, e.g. SGX does not have to export any symbols, changes to reclaim flows don't need to be routed through KVM, SGX's dirty laundry doesn't have to get aired out for the world to see, and so on and so forth.
The virtual EPC pages allocated to guests are currently not reclaimable. Reclaiming EPC page used by enclave requires a special reclaim mechanism separate from normal page reclaim, and that mechanism is not supported for virutal EPC pages. Due to the complications of handling reclaim conflicts between guest and host, reclaiming virtual EPC pages is significantly more complex than basic support for SGX virtualization.
- Support SGX virtualization without SGX Flexible Launch Control
SGX hardware supports two "launch control" modes to limit which enclaves can run. In the "locked" mode, the hardware prevents enclaves from running unless they are blessed by a third party. In the unlocked mode, the kernel is in full control of which enclaves can run. The bare-metal SGX code refuses to launch enclaves unless it is in the unlocked mode.
This sgx_virt_epc driver does not have such a restriction. This allows guests which are OK with the locked mode to use SGX, even if the host kernel refuses to.
- Support exposing SGX2
Due to the same reason above, SGX2 feature detection is added to core SGX code to allow KVM to expose SGX2 to guest, even currently SGX driver doesn't support SGX2, because SGX2 can work just fine in guest w/o any interaction to host SGX driver.
- Restricit SGX guest access to provisioning key
To grant guest being able to fully use SGX, guest needs to be able to access provisioning key. The provisioning key is sensitive, and accessing to it should be restricted. In bare-metal driver, allowing enclave to access provisioning key is restricted by being able to open /dev/sgx_provision.
Add a new KVM_CAP_SGX_ATTRIBUTE to KVM uAPI to extend above mechanism to KVM guests as well. When userspace hypervisor creates a new VM, the new cap is only added to VM when userspace hypervisior is able to open /dev/sgx_provision, following the same role as in bare-metal driver. KVM then traps ECREATE from guest, and only allows ECREATE with provisioning key bit to run when guest supports KVM_CAP_SGX_ATTRIBUTE.
Kai Huang (2):
x86/sgx: Initialize virtual EPC driver even when SGX driver is
disabled
x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
Sean Christopherson (22):
x86/cpufeatures: Add SGX1 and SGX2 sub-features
x86/sgx: Add SGX_CHILD_PRESENT hardware error code x86/sgx: Introduce virtual EPC for use by KVM guests x86/cpu/intel: Allow SGX virtualization without Launch Control support x86/sgx: Expose SGX architectural definitions to the kernel x86/sgx: Move ENCLS leaf definitions to sgx_arch.h x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT) x86/sgx: Add encls_faulted() helper
x86/sgx: Add helpers to expose ECREATE and EINIT to KVM x86/sgx: Move provisioning device creation out of SGX driver
KVM: VMX: Convert vcpu_vmx.exit_reason to a union
KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)
KVM: x86: Define new #PF SGX error code bit
KVM: x86: Add support for reverse CPUID lookup of scattered features
KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
KVM: VMX: Add basic handling of VM-Exit from SGX enclave
KVM: VMX: Frame in ENCLS handler for SGX virtualization
KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions
KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs
KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)
KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC
KVM: x86: Add capability to grant VM access to privileged SGX
attribute
[email protected] (2):
x86/sgx: Remove a warn from sgx_free_epc_page() x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
Documentation/virt/kvm/api.rst | 23 +
arch/x86/Kconfig | 12 +
arch/x86/include/asm/cpufeatures.h | 2 +
arch/x86/include/asm/kvm_host.h | 5 +
arch/x86/include/asm/sgx.h | 19 +
.../cpu/sgx/arch.h => include/asm/sgx_arch.h} | 20 +
arch/x86/include/asm/vmx.h | 1 +
arch/x86/include/uapi/asm/vmx.h | 1 +
arch/x86/kernel/cpu/cpuid-deps.c | 3 +
arch/x86/kernel/cpu/feat_ctl.c | 63 ++-
arch/x86/kernel/cpu/scattered.c | 2 +
arch/x86/kernel/cpu/sgx/Makefile | 1 +
arch/x86/kernel/cpu/sgx/driver.c | 17 -
arch/x86/kernel/cpu/sgx/encl.c | 15 +-
arch/x86/kernel/cpu/sgx/encls.h | 29 +-
arch/x86/kernel/cpu/sgx/ioctl.c | 23 +-
arch/x86/kernel/cpu/sgx/main.c | 67 ++-
arch/x86/kernel/cpu/sgx/sgx.h | 4 +-
arch/x86/kernel/cpu/sgx/virt.c | 316 ++++++++++++
arch/x86/kernel/cpu/sgx/virt.h | 14 +
arch/x86/kvm/Makefile | 2 +
arch/x86/kvm/cpuid.c | 89 +++-
arch/x86/kvm/cpuid.h | 50 +-
arch/x86/kvm/vmx/nested.c | 70 ++-
arch/x86/kvm/vmx/nested.h | 5 +
arch/x86/kvm/vmx/sgx.c | 462 ++++++++++++++++++
arch/x86/kvm/vmx/sgx.h | 34 ++
arch/x86/kvm/vmx/vmcs12.c | 1 +
arch/x86/kvm/vmx/vmcs12.h | 4 +-
arch/x86/kvm/vmx/vmx.c | 171 +++++--
arch/x86/kvm/vmx/vmx.h | 27 +-
arch/x86/kvm/x86.c | 24 +
include/uapi/linux/kvm.h | 1 +
tools/testing/selftests/sgx/defines.h | 2 +-
34 files changed, 1432 insertions(+), 147 deletions(-)
create mode 100644 arch/x86/include/asm/sgx.h
rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx_arch.h} (96%)
create mode 100644 arch/x86/kernel/cpu/sgx/virt.c
create mode 100644 arch/x86/kernel/cpu/sgx/virt.h
create mode 100644 arch/x86/kvm/vmx/sgx.c
create mode 100644 arch/x86/kvm/vmx/sgx.hSignature · 1 line
2.29.2
RFC PATCH v2 1/26 x86/cpufeatures: Add SGX1 and SGX2 sub-features
From: Sean Christopherson <[email protected]>
Add SGX1 and SGX2 feature flags, via CPUID.0x12.0x0.EAX, as scattered features. As part of virtualizing SGX, KVM will expose the SGX CPUID leafs to its guest, and to do so correctly needs to query hardware and kernel support for SGX1 and SGX2.
Also add SGX related feature bits to CPUID dependency table to make clearing SGX feature easier.
--- arch/x86/include/asm/cpufeatures.h | 2 ++ arch/x86/kernel/cpu/cpuid-deps.c | 3 +++ arch/x86/kernel/cpu/feat_ctl.c | 1 - arch/x86/kernel/cpu/scattered.c | 2 ++ 4 files changed, 7 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeatures.h +2 -0
@@ -292,6 +292,8 @@#define X86_FEATURE_FENCE_SWAPGS_KERNEL (11*32+ 5)/* "" LFENCE in kernel entry SWAPGS path */ #define X86_FEATURE_SPLIT_LOCK_DETECT (11*32+ 6)/* #AC for split lock */ #define X86_FEATURE_PER_THREAD_MBA (11*32+ 7)/* "" Per-thread Memory Bandwidth Allocation */ +#define X86_FEATURE_SGX1 (11*32+ 8)/* Software Guard Extensions sub-feature SGX1 */ +#define X86_FEATURE_SGX2 (11*32+ 9)/* Software Guard Extensions sub-feature SGX2 */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5)/* AVX512 BFLOAT16 instructions */
arch/x86/kernel/cpu/cpuid-deps.c +3 -0
@@ -72,6 +72,9 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_AVX512_FP16, X86_FEATURE_AVX512BW }, { X86_FEATURE_ENQCMD, X86_FEATURE_XSAVES }, { X86_FEATURE_PER_THREAD_MBA, X86_FEATURE_MBA }, + { X86_FEATURE_SGX_LC, X86_FEATURE_SGX }, + { X86_FEATURE_SGX1, X86_FEATURE_SGX }, + { X86_FEATURE_SGX2, X86_FEATURE_SGX1 }, {} };
arch/x86/kernel/cpu/feat_ctl.c +0 -1
@@ -96,7 +96,6 @@ static void init_vmx_capabilities(struct cpuinfo_x86 *c)static void clear_sgx_caps (void ) {setup_clear_cpu_cap (X86_FEATURE_SGX ); -setup_clear_cpu_cap (X86_FEATURE_SGX_LC ); }static int __init nosgx (char *str)
arch/x86/kernel/cpu/scattered.c +2 -0
@@ -36,6 +36,8 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_CDP_L2, CPUID_ECX, 2, 0x00000010, 2 }, { X86_FEATURE_MBA, CPUID_EBX, 3, 0x00000010, 0 }, { X86_FEATURE_PER_THREAD_MBA, CPUID_ECX, 0, 0x00000010, 3 }, + { X86_FEATURE_SGX1, CPUID_EAX, 0, 0x00000012, 0 }, + { X86_FEATURE_SGX2, CPUID_EAX, 1, 0x00000012, 0 }, { X86_FEATURE_HW_PSTATE, CPUID_EDX, 7, 0x80000007, 0 }, { X86_FEATURE_CPB, CPUID_EDX, 9, 0x80000007, 0 }, { X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 },
Signature · 1 line
2.29.2
RFC PATCH v2 2/26 x86/sgx: Remove a warn from sgx_free_epc_page()
In reply to Kai HuangFrom: "[email protected]" <[email protected]>
Remove SGX_EPC_PAGE_RECLAIMER_TRACKED check and warning. This cannot happen, as enclave pages are freed only at the time when encl->refcount triggers, i.e. when both VFS and the page reclaimer have given up on their references.
--- arch/x86/kernel/cpu/sgx/main.c | 2 -- 1 file changed, 2 deletions(-)
arch/x86/kernel/cpu/sgx/main.c +0 -2
@@ -605,8 +605,6 @@ void sgx_free_epc_page(struct sgx_epc_page *page)struct sgx_epc_section *section = &sgx_epc_sections[page->section];int ret; -WARN_ON_ONCE (page->flags &SGX_EPC_PAGE_RECLAIMER_TRACKED ); - ret =__eremove (sgx_get_epc_virt_addr (page));if (WARN_ONCE (ret,"EREMOVE returned %d (0x%x)" , ret, ret))return ;
Signature · 1 line
2.29.2
RFC PATCH v2 3/26 x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Kai HuangFrom: "[email protected]" <[email protected]>
Encapsulate the snippet in sgx_free_epc_page() concerning EREMOVE to sgx_reset_epc_page(), which is a static helper function for sgx_encl_release(). It's the only function existing, which deals with initialized pages.
--- arch/x86/kernel/cpu/sgx/encl.c | 13 +++++++++++++ arch/x86/kernel/cpu/sgx/main.c | 10 ++++------ 2 files changed, 17 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/sgx/encl.c +13 -0
@@ -389,6 +389,16 @@ const struct vm_operations_struct sgx_vm_ops = { .access = sgx_vma_access, }; + +static void sgx_reset_epc_page (struct sgx_epc_page *epc_page) +{ +int ret; + + ret =__eremove (sgx_get_epc_virt_addr (epc_page)); +if (WARN_ONCE (ret,"EREMOVE returned %d (0x%x)" , ret, ret)) +return ; +} +/** * sgx_encl_release - Destroy an enclave instance * @kref: address of a kref inside &sgx_encl
@@ -412,6 +422,7 @@ void sgx_encl_release(struct kref *ref)if (sgx_unmark_page_reclaimable (entry->epc_page))continue ; +sgx_reset_epc_page (entry->epc_page);sgx_free_epc_page (entry->epc_page); encl->secs_child_cnt--; entry->epc_page =NULL ;
@@ -423,6 +434,7 @@ void sgx_encl_release(struct kref *ref)xa_destroy (&encl->page_array);if (!encl->secs_child_cnt && encl->secs.epc_page) { +sgx_reset_epc_page (encl->secs.epc_page);sgx_free_epc_page (encl->secs.epc_page); encl->secs.epc_page =NULL ; }
@@ -431,6 +443,7 @@ void sgx_encl_release(struct kref *ref) va_page = list_first_entry(&encl->va_pages, struct sgx_va_page, list);list_del (&va_page->list); +sgx_reset_epc_page (va_page->epc_page);sgx_free_epc_page (va_page->epc_page);kfree (va_page); }
arch/x86/kernel/cpu/sgx/main.c +4 -6
@@ -598,16 +598,14 @@ struct sgx_epc_page *sgx_alloc_epc_page(void *owner, bool reclaim)* sgx_free_epc_page() - Free an EPC page * @page: an EPC page * -* Call EREMOVE for an EPC page and insert it back to the list of free pages. +* Put the EPC page back to the list of free pages. It's the callers +* responsibility to make sure that the page is in uninitialized state In other +* words, do EREMOVE, EWB or whatever operation is necessary before calling +* this function. */ void sgx_free_epc_page (struct sgx_epc_page *page) {struct sgx_epc_section *section = &sgx_epc_sections[page->section]; -int ret; - - ret =__eremove (sgx_get_epc_virt_addr (page)); -if (WARN_ONCE (ret,"EREMOVE returned %d (0x%x)" , ret, ret)) -return ;spin_lock (§ion->lock);list_add_tail (&page->list, §ion->page_list);
Signature · 1 line
2.29.2
RFC PATCH v2 4/26 x86/sgx: Add SGX_CHILD_PRESENT hardware error code
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
SGX virtualization requires to allocate "raw" EPC and use it as "virtual EPC" for SGX guest. Unlike EPC used by SGX driver, virtual EPC doesn't track how EPC pages are used in VM, e.g. (de)construction of enclaves, so it cannot guarantee EREMOVE success, e.g. it doesn't have a priori knowledge of which pages are SECS with non-zero child counts.
Add SGX_CHILD_PRESENT for use by SGX virtualization to assert EREMOVE failures are expected, but only due to SGX_CHILD_PRESENT.
---v1->v2:
- Change title to reflect hardware error code.
--- arch/x86/kernel/cpu/sgx/arch.h | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kernel/cpu/sgx/arch.h +2 -0
@@ -26,12 +26,14 @@* enum sgx_return_code - The return code type for ENCLS, ENCLU and ENCLV * %SGX_NOT_TRACKED: Previous ETRACK's shootdown sequence has not * been completed yet. +* %SGX_CHILD_PRESENT Enclave has child pages present in the EPC. * %SGX_INVALID_EINITTOKEN: EINITTOKEN is invalid and enclave signer's * public key does not match IA32_SGXLEPUBKEYHASH. * %SGX_UNMASKED_EVENT: An unmasked event, e.g. INTR, was received */ enum sgx_return_code {SGX_NOT_TRACKED =11 , +SGX_CHILD_PRESENT =13 ,SGX_INVALID_EINITTOKEN =16 ,SGX_UNMASKED_EVENT =128 , };
Signature · 1 line
2.29.2
RFC PATCH v2 5/26 x86/sgx: Introduce virtual EPC for use by KVM guests
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add a misc device /dev/sgx_virt_epc to allow userspace to allocate "raw" EPC without an associated enclave. The intended and only known use case for raw EPC allocation is to expose EPC to a KVM guest, hence the virt_epc moniker, virt.{c,h} files and X86_SGX_VIRTUALIZATION Kconfig.
More specifically, to allocate a virtual EPC instance with particular size, the userspace hypervisor opens the device node, and uses mmap() with the intended size to get an address range of virtual EPC. Then it may use the address range to create one KVM memory slot as virtual EPC for guest.
Implement the "raw" EPC allocation in the x86 core-SGX subsystem via /dev/sgx_virt_epc rather than in KVM. Doing so has two major advantages:
- Does not require changes to KVM's uAPI, e.g. EPC gets handled as just another memory backend for guests.
- EPC management is wholly contained in the SGX subsystem, e.g. SGX does not have to export any symbols, changes to reclaim flows don't need to be routed through KVM, SGX's dirty laundry doesn't have to get aired out for the world to see, and so on and so forth.
The virtual EPC pages allocated to guests are currently not reclaimable. Reclaiming EPC page used by enclave requires a special reclaim mechanism separate from normal page reclaim, and that mechanism is not supported for virutal EPC pages. Due to the complications of handling reclaim conflicts between guest and host, reclaiming virtual EPC pages is significantly more complex than basic support for SGX virtualization.
---v1->v2:
- Added one paragraph to explain fops of virtual EPC, per Jarkko's suggestion.
- Moved change to sgx_init() out of this patch to a separate patch, as stated in cover letter.
- In sgx_virt_epc_init(), return error if VMX is not supported, or CONFIG_KVM_INTEL is not enabled, because there's no point to create /dev/sgx_virt_epc if KVM is not supported.
- Removed 'struct mm_struct *mm' in 'struct sgx_virt_epc', and related logic in sgx_virt_epc_open/release/mmap(), per Dave's comment.
- Renamed 'virtual_epc_zombie_pages' and 'virt_epc_lock' to 'zombie_secs_pages' 'zombie_secs_pages_lock', per Dave's suggestion.
- Changed __sgx_free_epc_page() to sgx_free_epc_page() due to Jarkko's patch removes EREMOVE in sgx_free_epc_page().
- Changed all struct sgx_virt_epc *epc to struct sgx_virt_epc *vepc.
- In __sgx_virt_epc_fault(), changed comment to use WARN_ON() to make sure vepc->lock has already been hold, per Dave's suggestion.
- In sgx_virt_epc_free_page(), added comments to explain SGX_ENCLAVE_ACT is not expected; and changed to use WARN_ONCE() to dump actual error code, per Dave's comment.
- Removed NULL page check in sgx_virt_epc_free_page(), per Dave's comment.
--- arch/x86/Kconfig | 12 ++ arch/x86/kernel/cpu/sgx/Makefile | 1 + arch/x86/kernel/cpu/sgx/virt.c | 255 +++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/sgx/virt.h | 14 ++ 4 files changed, 282 insertions(+) create mode 100644 arch/x86/kernel/cpu/sgx/virt.c create mode 100644 arch/x86/kernel/cpu/sgx/virt.h
arch/x86/Kconfig +12 -0
@@ -1951,6 +1951,18 @@ config X86_SGX If unsure, say N. +config X86_SGX_VIRTUALIZATION + bool "Software Guard eXtensions (SGX) Virtualization" + depends on X86_SGX && KVM_INTEL + help + + Enables KVM guests to create SGX enclaves. + + This includes support to expose "raw" unreclaimable enclave memory to + guests via a device node, e.g. /dev/sgx_virt_epc. + + If unsure, say N. + config EFI bool "EFI runtime service support" depends on ACPI
arch/x86/kernel/cpu/sgx/Makefile +1 -0
@@ -3,3 +3,4 @@ obj-y += \ encl.o \ ioctl.o \ main.o +obj-$(CONFIG_X86_SGX_VIRTUALIZATION) += virt.o
arch/x86/kernel/cpu/sgx/virt.c new file +255 -0
255 lines not on this page · raw message
arch/x86/kernel/cpu/sgx/virt.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */ +#ifndef _ASM_X86_SGX_VIRT_H +#define _ASM_X86_SGX_VIRT_H + +#ifdef CONFIG_X86_SGX_VIRTUALIZATION +int __init sgx_virt_epc_init (void ); +#else +static inline int __init sgx_virt_epc_init (void ) +{ +return -ENODEV ; +} +#endif + +#endif /* _ASM_X86_SGX_VIRT_H */
Signature · 1 line
2.29.2
RFC PATCH v2 6/26 x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
The kernel will currently disable all SGX support if the hardware does not support launch control. Make it more permissive to allow SGX virtualization on systems without Launch Control support. This will allow KVM to expose SGX to guests that have less-strict requirements on the availability of flexible launch control.
Improve error message to distinguish between three cases. There are two cases where SGX support is completely disabled:
1) SGX has been disabled completely by the BIOS
2) SGX LC is locked by the BIOS. Bare-metal support is disabled because of LC unavailability. SGX virtualization is unavailable (because of Kconfig).
One where it is partially available:
3) SGX LC is locked by the BIOS. Bare-metal support is disabled because of LC unavailability. SGX virtualization is supported.
---v1->v2:
- Refined commit message per Dave's comments.
- Added check to only enable SGX virtualization when VMX is supported, per Dave's comment.
- Refined error msg print to explicitly call out SGX virtualization will be supported when LC is locked by BIOS, per Dave's comment.
--- arch/x86/kernel/cpu/feat_ctl.c | 64 +++++++++++++++++++++++++++------- 1 file changed, 51 insertions(+), 13 deletions(-)
arch/x86/kernel/cpu/feat_ctl.c +51 -13
@@ -98,6 +98,11 @@ static void clear_sgx_caps(void)setup_clear_cpu_cap (X86_FEATURE_SGX ); } +static void clear_sgx_lc (void ) +{ +setup_clear_cpu_cap (X86_FEATURE_SGX_LC ); +} +static int __init nosgx (char *str) {clear_sgx_caps ();
@@ -110,7 +115,7 @@ early_param("nosgx", nosgx);void init_ia32_feat_ctl (struct cpuinfo_x86 *c) {bool tboot =tboot_enabled (); -bool enable_sgx; +bool enable_vmx, enable_sgx_virt, enable_sgx_driver;u64 msr;if (rdmsrl_safe (MSR_IA32_FEAT_CTL , &msr)) {
@@ -119,13 +124,24 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c)return ; } + enable_vmx =cpu_has (c,X86_FEATURE_VMX ) && +IS_ENABLED (CONFIG_KVM_INTEL ); +/* -* Enable SGX if and only if the kernel supports SGX and Launch Control -* is supported, i.e. disable SGX if the LE hash MSRs can't be written. +* Enable SGX if and only if the kernel supports SGX. Require Launch +* Control support if SGX virtualization is *not* supported, i.e. +* disable SGX if the LE hash MSRs can't be written and SGX can't be +* exposed to a KVM guest (which might support non-LC configurations). */ - enable_sgx =cpu_has (c,X86_FEATURE_SGX ) && -cpu_has (c,X86_FEATURE_SGX_LC ) && -IS_ENABLED (CONFIG_X86_SGX ); + enable_sgx_driver =cpu_has (c,X86_FEATURE_SGX ) && +cpu_has (c,X86_FEATURE_SGX1 ) && +IS_ENABLED (CONFIG_X86_SGX ) && +cpu_has (c,X86_FEATURE_SGX_LC ); + enable_sgx_virt =cpu_has (c,X86_FEATURE_SGX ) && +cpu_has (c,X86_FEATURE_SGX1 ) && +IS_ENABLED (CONFIG_X86_SGX ) && +IS_ENABLED (CONFIG_X86_SGX_VIRTUALIZATION ) && + enable_vmx;if (msr &FEAT_CTL_LOCKED ) goto update_caps;
@@ -141,15 +157,18 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c)* i.e. KVM is enabled, to avoid unnecessarily adding an attack vector * for the kernel, e.g. using VMX to hide malicious code. */ -if (cpu_has (c,X86_FEATURE_VMX ) &&IS_ENABLED (CONFIG_KVM_INTEL )) { +if (enable_vmx) { msr |=FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX ;if (tboot) msr |=FEAT_CTL_VMX_ENABLED_INSIDE_SMX ; } -if (enable_sgx) - msr |=FEAT_CTL_SGX_ENABLED |FEAT_CTL_SGX_LC_ENABLED ; +if (enable_sgx_driver || enable_sgx_virt) { + msr |=FEAT_CTL_SGX_ENABLED ; +if (enable_sgx_driver) + msr |=FEAT_CTL_SGX_LC_ENABLED ; + }wrmsrl (MSR_IA32_FEAT_CTL , msr);
@@ -172,10 +191,29 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c) } update_sgx: -if (!(msr &FEAT_CTL_SGX_ENABLED ) || - !(msr &FEAT_CTL_SGX_LC_ENABLED ) || !enable_sgx) { -if (enable_sgx) -pr_err_once ("SGX disabled by BIOS\n" ); +if (!(msr &FEAT_CTL_SGX_ENABLED )) { +if (enable_sgx_driver || enable_sgx_virt) +pr_err_once ("SGX disabled by BIOS.\n" );clear_sgx_caps (); +return ; + } + +/* +* VMX feature bit may be cleared due to being disabled in BIOS, +* in which case SGX virtualization cannot be supported either. +*/ +if (!cpu_has (c,X86_FEATURE_VMX ) && enable_sgx_virt) { +pr_err_once ("SGX virtualization disabled due to lack of VMX.\n" ); + enable_sgx_virt =0 ; + } + +if (!(msr &FEAT_CTL_SGX_LC_ENABLED ) && enable_sgx_driver) { +if (!enable_sgx_virt) { +pr_err_once ("SGX Launch Control is locked. Disable SGX.\n" ); +clear_sgx_caps (); + }else { +pr_err_once ("SGX Launch Control is locked. Support SGX virtualization only.\n" ); +clear_sgx_lc (); + } } }
Signature · 1 line
2.29.2
RFC PATCH v2 7/26 x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Kai HuangModify sgx_init() to always try to initialize the virtual EPC driver, even if the bare-metal SGX driver is disabled. The bare-metal driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.
--- arch/x86/kernel/cpu/sgx/main.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/main.c +3 -1
@@ -12,6 +12,7 @@#include "driver.h" #include "encl.h" #include "encls.h" +#include "virt.h" struct sgx_epc_section sgx_epc_sections[SGX_MAX_EPC_SECTIONS ];static int sgx_nr_epc_sections;
@@ -710,7 +711,8 @@ static void __init sgx_init(void)if (!sgx_page_reclaimer_init ()) goto err_page_cache; - ret =sgx_drv_init (); +/* Success if the native *or* virtual EPC driver initialized cleanly. */ + ret = !!sgx_drv_init () & !!sgx_virt_epc_init ();if (ret) goto err_kthread;
Signature · 1 line
2.29.2
RFC PATCH v2 8/26 x86/sgx: Expose SGX architectural definitions to the kernel
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
KVM will use many of the architectural constants and structs to virtualize SGX.
---
arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx_arch.h} | 0
arch/x86/kernel/cpu/sgx/encl.c | 2 +-
arch/x86/kernel/cpu/sgx/sgx.h | 2 +-
tools/testing/selftests/sgx/defines.h | 2 +-
4 files changed, 3 insertions(+), 3 deletions(-)
rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx_arch.h} (100%)arch/x86/include/asm/sgx_arch.h renamed from arch/x86/kernel/cpu/sgx/arch.h +0 -0
arch/x86/kernel/cpu/sgx/encl.c +1 -1
@@ -7,7 +7,7 @@#include <linux/shmem_fs.h> #include <linux/suspend.h> #include <linux/sched/mm.h> -#include "arch.h" +#include <asm/sgx_arch.h> #include "encl.h" #include "encls.h" #include "sgx.h"
arch/x86/kernel/cpu/sgx/sgx.h +1 -1
@@ -8,7 +8,7 @@#include <linux/rwsem.h> #include <linux/types.h> #include <asm/asm.h> -#include "arch.h" +#include <asm/sgx_arch.h> #undef pr_fmt#define pr_fmt (fmt) "sgx: " fmt
tools/testing/selftests/sgx/defines.h +1 -1
@@ -14,7 +14,7 @@#define __aligned (x) __attribute__((__aligned__(x)))#define __packed __attribute__((packed)) -#include "../../../../arch/x86/kernel/cpu/sgx/arch.h" +#include "../../../../arch/x86/include/asm/sgx_arch.h" #include "../../../../arch/x86/include/asm/enclu.h" #include "../../../../arch/x86/include/uapi/asm/sgx.h"
Signature · 1 line
2.29.2
RFC PATCH v2 9/26 x86/sgx: Move ENCLS leaf definitions to sgx_arch.h
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Move the ENCLS leaf definitions to sgx_arch.h so that they can be used by KVM. And because they're architectural.
--- arch/x86/include/asm/sgx_arch.h | 15 +++++++++++++++ arch/x86/kernel/cpu/sgx/encls.h | 15 --------------- 2 files changed, 15 insertions(+), 15 deletions(-)
arch/x86/include/asm/sgx_arch.h +15 -0
@@ -22,6 +22,21 @@/* The bitmask for the EPC section type. */ #define SGX_CPUID_EPC_MASK GENMASK(3, 0) +enum sgx_encls_function { +ECREATE =0x00 , +EADD =0x01 , +EINIT =0x02 , +EREMOVE =0x03 , +EDGBRD =0x04 , +EDGBWR =0x05 , +EEXTEND =0x06 , +ELDU =0x08 , +EBLOCK =0x09 , +EPA =0x0A , +EWB =0x0B , +ETRACK =0x0C , +}; +/** * enum sgx_return_code - The return code type for ENCLS, ENCLU and ENCLV * %SGX_NOT_TRACKED: Previous ETRACK's shootdown sequence has not
arch/x86/kernel/cpu/sgx/encls.h +0 -15
@@ -11,21 +11,6 @@#include <asm/traps.h> #include "sgx.h" -enum sgx_encls_function { -ECREATE =0x00 , -EADD =0x01 , -EINIT =0x02 , -EREMOVE =0x03 , -EDGBRD =0x04 , -EDGBWR =0x05 , -EEXTEND =0x06 , -ELDU =0x08 , -EBLOCK =0x09 , -EPA =0x0A , -EWB =0x0B , -ETRACK =0x0C , -}; -/** * ENCLS_FAULT_FLAG - flag signifying an ENCLS return code is a trapnr *
Signature · 1 line
2.29.2
RFC PATCH v2 10/26 x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT)
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Define the ENCLS leafs that are available with SGX2, also referred to as Enclave Dynamic Memory Management (EDMM). The leafs will be used by KVM to conditionally expose SGX2 capabilities to guests.
--- arch/x86/include/asm/sgx_arch.h | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/include/asm/sgx_arch.h +3 -0
@@ -35,6 +35,9 @@ enum sgx_encls_function {EPA =0x0A ,EWB =0x0B ,ETRACK =0x0C , +EAUG =0x0D , +EMODPR =0x0E , + EMODT = 0x0F, };/**
Signature · 1 line
2.29.2
RFC PATCH v2 11/26 x86/sgx: Add encls_faulted() helper
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add a helper to extract the fault indicator from an encoded ENCLS return value. SGX virtualization will also need to detect ENCLS faults.
--- arch/x86/kernel/cpu/sgx/encls.h | 14 +++++++++++++- arch/x86/kernel/cpu/sgx/ioctl.c | 2 +- 2 files changed, 14 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/sgx/encls.h +13 -1
@@ -40,6 +40,18 @@ }while (0 ); \ } +/* +* encls_faulted() - Check if an ENCLS leaf faulted given an error code +* @ret the return value of an ENCLS leaf function call +* +* Return: +* %true if @ret indicates a fault, %false otherwise +*/ +static inline bool encls_faulted (int ret) +{ +return ret &ENCLS_FAULT_FLAG ; +} +/** * encls_failed() - Check if an ENCLS function failed * @ret: the return value of an ENCLS function call
@@ -50,7 +62,7 @@*/ static inline bool encls_failed (int ret) { -if (ret &ENCLS_FAULT_FLAG ) +if (encls_faulted (ret))return ENCLS_TRAPNR (ret) !=X86_TRAP_PF ;return !!ret;
arch/x86/kernel/cpu/sgx/ioctl.c +1 -1
@@ -568,7 +568,7 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct, } } -if (ret &ENCLS_FAULT_FLAG ) { +if (encls_faulted (ret)) {if (encls_failed (ret))ENCLS_WARN (ret,"EINIT" );
Signature · 1 line
2.29.2
RFC PATCH v2 12/26 x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Kai HuangAdd a helper to update SGX_LEPUBKEYHASHn MSRs. SGX virtualization also needs to update those MSRs based on guest's "virtual" SGX_LEPUBKEYHASHn before EINIT from guest.
--- arch/x86/kernel/cpu/sgx/ioctl.c | 5 ++--- arch/x86/kernel/cpu/sgx/main.c | 8 ++++++++ arch/x86/kernel/cpu/sgx/sgx.h | 2 ++ 3 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/sgx/ioctl.c +2 -3
@@ -495,7 +495,7 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct, void *token) {u64 mrsigner[4 ]; -int i, j, k; +int i, j;void *addr;int ret;
@@ -544,8 +544,7 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,preempt_disable (); -for (k =0 ; k <4 ; k++) -wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 + k, mrsigner[k]); +sgx_update_lepubkeyhash (mrsigner); ret =__einit (sigstruct, token, addr);
arch/x86/kernel/cpu/sgx/main.c +8 -0
@@ -697,6 +697,14 @@ static bool __init sgx_page_cache_init(void)return true; } +void sgx_update_lepubkeyhash (u64 *lepubkeyhash) +{ +int i; + +for (i =0 ; i <4 ; i++) +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 + i, lepubkeyhash[i]); +} +static void __init sgx_init (void ) {int ret;
arch/x86/kernel/cpu/sgx/sgx.h +2 -0
@@ -83,4 +83,6 @@ void sgx_mark_page_reclaimable(struct sgx_epc_page *page);int sgx_unmark_page_reclaimable (struct sgx_epc_page *page);struct sgx_epc_page *sgx_alloc_epc_page (void *owner,bool reclaim); +void sgx_update_lepubkeyhash (u64 *lepubkeyhash); +#endif /* _X86_SGX_H */
Signature · 1 line
2.29.2
RFC PATCH v2 13/26 x86/sgx: Add helpers to expose ECREATE and EINIT to KVM
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
The bare-metal kernel must intercept ECREATE to be able to impose policies on guests. When it does this, the bare-metal kernel runs ECREATE against the userspace mapping of the virtualized EPC.
Provide wrappers around __ecreate() and __einit() to hide the ugliness of overloading the ENCLS return value to encode multiple error formats in a single int. KVM will trap-and-execute ECREATE and EINIT as part of SGX virtualization, and on an exception, KVM needs the trapnr so that it can inject the correct fault into the guest.
---v1->v2:
- Refined commit msg based on Dave's comment.
- Added comment to explain why to use __uaccess_xxx().
--- arch/x86/include/asm/sgx.h | 16 +++++++++ arch/x86/kernel/cpu/sgx/virt.c | 61 ++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+) create mode 100644 arch/x86/include/asm/sgx.h
arch/x86/include/asm/sgx.h new file +16 -0
@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_SGX_H +#define _ASM_X86_SGX_H + +#include <linux/types.h> + +#ifdef CONFIG_X86_SGX_VIRTUALIZATION +struct sgx_pageinfo ; + +int sgx_virt_ecreate (struct sgx_pageinfo *pageinfo,void __user *secs, +int *trapnr); +int sgx_virt_einit (void __user *sigstruct,void __user *token, +void __user *secs,u64 *lepubkeyhash,int *trapnr); +#endif + +#endif /* _ASM_X86_SGX_H */
arch/x86/kernel/cpu/sgx/virt.c +61 -0
64 lines not on this page · raw message
Signature · 1 line
2.29.2
RFC PATCH v2 14/26 x86/sgx: Move provisioning device creation out of SGX driver
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
And extract sgx_set_attribute() out of sgx_ioc_enclave_provision() and export it as symbol for KVM to use.
Provisioning key is sensitive. SGX driver only allows to create enclave which can access provisioning key when enclave creator has permission to open /dev/sgx_provision. It should apply to VM as well, as provisioning key is platform specific, thus unrestricted VM can also potentially compromise provisioning key.
Move provisioning device creation out of sgx_drv_init() to sgx_init() as preparation for adding SGX virtualization support, so that even SGX driver is not enabled due to flexible launch control is not available, SGX virtualization can still be enabled, and use it to restrict VM's capability of being able to access provisioning key.
--- arch/x86/include/asm/sgx.h | 3 +++ arch/x86/kernel/cpu/sgx/driver.c | 17 ------------ arch/x86/kernel/cpu/sgx/ioctl.c | 16 ++---------- arch/x86/kernel/cpu/sgx/main.c | 45 +++++++++++++++++++++++++++++++- 4 files changed, 49 insertions(+), 32 deletions(-)
arch/x86/include/asm/sgx.h +3 -0
@@ -4,6 +4,9 @@#include <linux/types.h> +int sgx_set_attribute (unsigned long *allowed_attributes, +unsigned int attribute_fd); +#ifdef CONFIG_X86_SGX_VIRTUALIZATION struct sgx_pageinfo ;
arch/x86/kernel/cpu/sgx/driver.c +0 -17
@@ -133,10 +133,6 @@ static const struct file_operations sgx_encl_fops = { .get_unmapped_area = sgx_get_unmapped_area, }; -const struct file_operations sgx_provision_fops = { - .owner =THIS_MODULE , -}; -static struct miscdevice sgx_dev_enclave = { .minor =MISC_DYNAMIC_MINOR , .name ="sgx_enclave" ,
@@ -144,13 +140,6 @@ static struct miscdevice sgx_dev_enclave = { .fops = &sgx_encl_fops, }; -static struct miscdevice sgx_dev_provision = { - .minor =MISC_DYNAMIC_MINOR , - .name ="sgx_provision" , - .nodename ="sgx_provision" , - .fops = &sgx_provision_fops, -}; -int __init sgx_drv_init (void ) {unsigned int eax, ebx, ecx, edx;
@@ -184,11 +173,5 @@ int __init sgx_drv_init(void)if (ret)return ret; - ret =misc_register (&sgx_dev_provision); -if (ret) { -misc_deregister (&sgx_dev_enclave); -return ret; - } -return 0 ; }
arch/x86/kernel/cpu/sgx/ioctl.c +2 -14
@@ -2,6 +2,7 @@/* Copyright(c) 2016-20 Intel Corporation. */ #include <asm/mman.h> +#include <asm/sgx.h> #include <linux/mman.h> #include <linux/delay.h> #include <linux/file.h>
@@ -664,24 +665,11 @@ static long sgx_ioc_enclave_init(struct sgx_encl *encl, void __user *arg)static long sgx_ioc_enclave_provision (struct sgx_encl *encl,void __user *arg) {struct sgx_enclave_provision params; -struct file *file;if (copy_from_user (¶ms, arg,sizeof (params)))return -EFAULT ; - file =fget (params.fd); -if (!file) -return -EINVAL ; - -if (file->f_op != &sgx_provision_fops) { -fput (file); -return -EINVAL ; - } - - encl->attributes_mask |=SGX_ATTR_PROVISIONKEY ; - -fput (file); -return 0 ; +return sgx_set_attribute (&encl->attributes_mask, params.fd); }long sgx_ioctl (struct file *filep,unsigned int cmd,unsigned long arg)
arch/x86/kernel/cpu/sgx/main.c +44 -1
77 lines not on this page · raw message
Signature · 1 line
2.29.2
RFC PATCH v2 15/26 KVM: VMX: Convert vcpu_vmx.exit_reason to a union
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Convert vcpu_vmx.exit_reason from a u32 to a union (of size u32). The full VM_EXIT_REASON field is comprised of a 16-bit basic exit reason in bits 15:0, and single-bit modifiers in bits 31:16.
Historically, KVM has only had to worry about handling the "failed VM-Entry" modifier, which could only be set in very specific flows and required dedicated handling. I.e. manually stripping the FAILED_VMENTRY bit was a somewhat viable approach. But even with only a single bit to worry about, KVM has had several bugs related to comparing a basic exit reason against the full exit reason store in vcpu_vmx.
Upcoming Intel features, e.g. SGX, will add new modifier bits that can be set on more or less any VM-Exit, as opposed to the significantly more restricted FAILED_VMENTRY, i.e. correctly handling everything in one-off flows isn't scalable. Tracking exit reason in a union forces code to explicitly choose between consuming the full exit reason and the basic exit, and is a convenient way to document and access the modifiers.
No functional change intended.
--- arch/x86/kvm/vmx/nested.c | 42 +++++++++++++++--------- arch/x86/kvm/vmx/vmx.c | 68 ++++++++++++++++++++------------------- arch/x86/kvm/vmx/vmx.h | 25 +++++++++++++- 3 files changed, 86 insertions(+), 49 deletions(-)
arch/x86/kvm/vmx/nested.c +27 -15
@@ -3311,7 +3311,11 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);enum vm_entry_failure_code entry_failure_code;bool evaluate_pending_interrupts; -u32 exit_reason, failed_index; +u32 failed_index; +union vmx_exit_reason exit_reason = { + .basic =-1 , + .failed_vmentry =1 , + };if (kvm_check_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu))kvm_vcpu_flush_tlb_current (vcpu);
@@ -3363,7 +3367,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,if (nested_vmx_check_guest_state (vcpu, vmcs12, &entry_failure_code)) { - exit_reason =EXIT_REASON_INVALID_STATE ; + exit_reason.basic =EXIT_REASON_INVALID_STATE ; vmcs12->exit_qualification = entry_failure_code; goto vmentry_fail_vmexit; }
@@ -3374,7 +3378,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, vcpu->arch.tsc_offset += vmcs12->tsc_offset;if (prepare_vmcs02 (vcpu, vmcs12, &entry_failure_code)) { - exit_reason =EXIT_REASON_INVALID_STATE ; + exit_reason.basic =EXIT_REASON_INVALID_STATE ; vmcs12->exit_qualification = entry_failure_code; goto vmentry_fail_vmexit_guest_mode; }
@@ -3384,7 +3388,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, vmcs12->vm_entry_msr_load_addr, vmcs12->vm_entry_msr_load_count);if (failed_index) { - exit_reason =EXIT_REASON_MSR_LOAD_FAIL ; + exit_reason.basic =EXIT_REASON_MSR_LOAD_FAIL ; vmcs12->exit_qualification = failed_index; goto vmentry_fail_vmexit_guest_mode; }
@@ -3452,7 +3456,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,return NVMX_VMENTRY_VMEXIT ;load_vmcs12_host_state (vcpu, vmcs12); - vmcs12->vm_exit_reason = exit_reason |VMX_EXIT_REASONS_FAILED_VMENTRY ; + vmcs12->vm_exit_reason = exit_reason.full;if (enable_shadow_vmcs || vmx->nested.hv_evmcs) vmx->nested.need_vmcs12_to_shadow_sync = true;return NVMX_VMENTRY_VMEXIT ;
@@ -5540,7 +5544,12 @@ static int handle_vmfunc(struct kvm_vcpu *vcpu)return kvm_skip_emulated_instruction (vcpu); fail: -nested_vmx_vmexit (vcpu, vmx->exit_reason, +/* +* This is effectively a reflected VM-Exit, as opposed to a synthesized +* nested VM-Exit. Pass the original exit reason, i.e. don't hardcode +* EXIT_REASON_VMFUNC as the exit reason. +*/ +nested_vmx_vmexit (vcpu, vmx->exit_reason.full,vmx_get_intr_info (vcpu),vmx_get_exit_qual (vcpu));return 1 ;
@@ -5608,7 +5617,8 @@ static bool nested_vmx_exit_handled_io(struct kvm_vcpu *vcpu,* MSR bitmap. This may be the case even when L0 doesn't use MSR bitmaps. */ static bool nested_vmx_exit_handled_msr (struct kvm_vcpu *vcpu, -struct vmcs12 *vmcs12,u32 exit_reason) +struct vmcs12 *vmcs12, +union vmx_exit_reason exit_reason) {u32 msr_index =kvm_rcx_read (vcpu);gpa_t bitmap;
@@ -5622,7 +5632,7 @@ static bool nested_vmx_exit_handled_msr(struct kvm_vcpu *vcpu,* First we need to figure out which of the four to use: */ bitmap = vmcs12->msr_bitmap; -if (exit_reason ==EXIT_REASON_MSR_WRITE ) +if (exit_reason.basic ==EXIT_REASON_MSR_WRITE ) bitmap +=2048 ;if (msr_index >=0xc0000000 ) { msr_index -=0xc0000000 ;
@@ -5759,11 +5769,12 @@ static bool nested_vmx_exit_handled_mtf(struct vmcs12 *vmcs12)* Return true if L0 wants to handle an exit from L2 regardless of whether or not * L1 wants the exit. Only call this when in is_guest_mode (L2). */ -static bool nested_vmx_l0_wants_exit (struct kvm_vcpu *vcpu,u32 exit_reason) +static bool nested_vmx_l0_wants_exit (struct kvm_vcpu *vcpu, +union vmx_exit_reason exit_reason) {u32 intr_info; -switch ((u16 )exit_reason) { +switch (exit_reason.basic) {case EXIT_REASON_EXCEPTION_NMI : intr_info =vmx_get_intr_info (vcpu);if (is_nmi (intr_info))
@@ -5819,12 +5830,13 @@ static bool nested_vmx_l0_wants_exit(struct kvm_vcpu *vcpu, u32 exit_reason)* Return 1 if L1 wants to intercept an exit from L2. Only call this when in * is_guest_mode (L2). */ -static bool nested_vmx_l1_wants_exit (struct kvm_vcpu *vcpu,u32 exit_reason) +static bool nested_vmx_l1_wants_exit (struct kvm_vcpu *vcpu, +union vmx_exit_reason exit_reason) {struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);u32 intr_info; -switch ((u16 )exit_reason) { +switch (exit_reason.basic) {case EXIT_REASON_EXCEPTION_NMI : intr_info =vmx_get_intr_info (vcpu);if (is_nmi (intr_info))
@@ -5943,7 +5955,7 @@ static bool nested_vmx_l1_wants_exit(struct kvm_vcpu *vcpu, u32 exit_reason)bool nested_vmx_reflect_vmexit (struct kvm_vcpu *vcpu) {struct vcpu_vmx *vmx =to_vmx (vcpu); -u32 exit_reason = vmx->exit_reason; +union vmx_exit_reason exit_reason = vmx->exit_reason;unsigned long exit_qual;u32 exit_intr_info;
@@ -5962,7 +5974,7 @@ bool nested_vmx_reflect_vmexit(struct kvm_vcpu *vcpu) goto reflect_vmexit; } -trace_kvm_nested_vmexit (exit_reason, vcpu,KVM_ISA_VMX ); +trace_kvm_nested_vmexit (exit_reason.full, vcpu,KVM_ISA_VMX );/* If L0 (KVM) wants the exit, it trumps L1's desires. */ if (nested_vmx_l0_wants_exit (vcpu, exit_reason))
@@ -5988,7 +6000,7 @@ bool nested_vmx_reflect_vmexit(struct kvm_vcpu *vcpu) exit_qual =vmx_get_exit_qual (vcpu); reflect_vmexit: -nested_vmx_vmexit (vcpu, exit_reason, exit_intr_info, exit_qual); +nested_vmx_vmexit (vcpu, exit_reason.full, exit_intr_info, exit_qual);return true; }
arch/x86/kvm/vmx/vmx.c +35 -33
157 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.h +24 -1
@@ -70,6 +70,29 @@ struct pt_desc {struct pt_ctx guest; }; +union vmx_exit_reason { +struct { +u32 basic :16 ; +u32 reserved16 :1 ; +u32 reserved17 :1 ; +u32 reserved18 :1 ; +u32 reserved19 :1 ; +u32 reserved20 :1 ; +u32 reserved21 :1 ; +u32 reserved22 :1 ; +u32 reserved23 :1 ; +u32 reserved24 :1 ; +u32 reserved25 :1 ; +u32 reserved26 :1 ; +u32 sgx_enclave_mode :1 ; +u32 smi_pending_mtf :1 ; +u32 smi_from_vmx_root :1 ; +u32 reserved30 :1 ; +u32 failed_vmentry :1 ; + }; +u32 full; +}; +/* * The nested_vmx structure is part of vcpu_vmx, and holds information we need * for correct emulation of VMX (i.e., nested VMX) on this vcpu.
@@ -244,7 +267,7 @@ struct vcpu_vmx {int vpid;bool emulation_required; -u32 exit_reason; +union vmx_exit_reason exit_reason;/* Posted interrupt descriptor */ struct pi_desc pi_desc;
Signature · 1 line
2.29.2
RFC PATCH v2 16/26 KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Export the gva_to_gpa() helpers for use by SGX virtualization when executing ENCLS[ECREATE] and ENCLS[EINIT] on behalf of the guest. To execute ECREATE and EINIT, KVM must obtain the GPA of the target Secure Enclave Control Structure (SECS) in order to get its corresponding HVA.
Because the SECS must reside in the Enclave Page Cache (EPC), copying the SECS's data to a host-controlled buffer via existing exported helpers is not a viable option as the EPC is not readable or writable by the kernel.
SGX virtualization will also use gva_to_gpa() to obtain HVAs for non-EPC pages in order to pass user pointers directly to ECREATE and EINIT, which avoids having to copy pages worth of data into the kernel.
--- arch/x86/kvm/x86.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kvm/x86.c +2 -0
@@ -5891,6 +5891,7 @@ gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva,u32 access = (kvm_x86_ops.get_cpl (vcpu) ==3 ) ?PFERR_USER_MASK :0 ;return vcpu->arch.walk_mmu->gva_to_gpa (vcpu, gva, access, exception); } +EXPORT_SYMBOL_GPL (kvm_mmu_gva_to_gpa_read);gpa_t kvm_mmu_gva_to_gpa_fetch (struct kvm_vcpu *vcpu,gva_t gva,struct x86_exception *exception)
@@ -5907,6 +5908,7 @@ gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, access |=PFERR_WRITE_MASK ;return vcpu->arch.walk_mmu->gva_to_gpa (vcpu, gva, access, exception); } +EXPORT_SYMBOL_GPL (kvm_mmu_gva_to_gpa_write);/* uses this to access any guest's mapped memory without checking CPL */ gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva,
Signature · 1 line
2.29.2
RFC PATCH v2 17/26 KVM: x86: Define new #PF SGX error code bit
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Page faults that are signaled by the SGX Enclave Page Cache Map (EPCM), as opposed to the traditional IA32/EPT page tables, set an SGX bit in the error code to indicate that the #PF was induced by SGX. KVM will need to emulate this behavior as part of its trap-and-execute scheme for virtualizing SGX Launch Control, e.g. to inject SGX-induced #PFs if EINIT faults in the host, and to support live migration.
--- arch/x86/include/asm/kvm_host.h | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -216,6 +216,7 @@ enum x86_intercept_stage;#define PFERR_RSVD_BIT 3#define PFERR_FETCH_BIT 4#define PFERR_PK_BIT 5 +#define PFERR_SGX_BIT 15#define PFERR_GUEST_FINAL_BIT 32#define PFERR_GUEST_PAGE_BIT 33
@@ -225,6 +226,7 @@ enum x86_intercept_stage;#define PFERR_RSVD_MASK (1U << PFERR_RSVD_BIT)#define PFERR_FETCH_MASK (1U << PFERR_FETCH_BIT)#define PFERR_PK_MASK (1U << PFERR_PK_BIT) +#define PFERR_SGX_MASK (1U << PFERR_SGX_BIT)#define PFERR_GUEST_FINAL_MASK (1ULL << PFERR_GUEST_FINAL_BIT)#define PFERR_GUEST_PAGE_MASK (1ULL << PFERR_GUEST_PAGE_BIT)
Signature · 1 line
2.29.2
RFC PATCH v2 18/26 KVM: x86: Add support for reverse CPUID lookup of scattered features
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Introduce a scheme that allows KVM's CPUID magic to support features that are scattered in the kernel's feature words. To advertise and/or query guest support for CPUID-based features, KVM requires the bit number of an X86_FEATURE_* to match the bit number in its associated CPUID entry. For scattered features, this does not hold true.
Add a framework to allow defining KVM-only words, stored in kvm_cpu_caps after the shared kernel caps, that can be used to gather the scattered feature bits by translating X86_FEATURE_* flags into their KVM-defined feature.
Note, because reverse_cpuid_check() effectively forces kvm_cpu_caps lookups to be resolved at compile time, there is no runtime cost for translating from kernel-defined to kvm-defined features.
More details here: https://lkml.kernel.org/r/X/[email protected]
--- arch/x86/kvm/cpuid.c | 32 +++++++++++++++++++++++++++----- arch/x86/kvm/cpuid.h | 39 ++++++++++++++++++++++++++++++++++----- 2 files changed, 61 insertions(+), 10 deletions(-)
arch/x86/kvm/cpuid.c +27 -5
@@ -28,7 +28,7 @@* Unlike "struct cpuinfo_x86.x86_capability", kvm_cpu_caps doesn't need to be * aligned to sizeof(unsigned long) because it's not accessed via bitops. */ -u32 kvm_cpu_caps[NCAPINTS ]__read_mostly ; +u32 kvm_cpu_caps[NR_KVM_CPU_CAPS ]__read_mostly ;EXPORT_SYMBOL_GPL (kvm_cpu_caps);static u32 xstate_required_size (u64 xstate_bv,bool compacted)
@@ -53,6 +53,7 @@ static u32 xstate_required_size(u64 xstate_bv, bool compacted) }#define F feature_bit +#define SF (name) (boot_cpu_has(X86_FEATURE_##name) ? F(name) : 0)static inline struct kvm_cpuid_entry2 *cpuid_entry2_find (struct kvm_cpuid_entry2 *entries,int nent,u32 function,u32 index)
@@ -331,13 +332,13 @@ int kvm_vcpu_ioctl_get_cpuid2(struct kvm_vcpu *vcpu,return r; } -static __always_inline void kvm_cpu_cap_mask(enum cpuid_leafs leaf, u32 mask) +/* Mask kvm_cpu_caps for @leaf with the raw CPUID capabilities of this CPU. */ +static __always_inline void __kvm_cpu_cap_mask(enum cpuid_leafs leaf) {const struct cpuid_reg cpuid =x86_feature_cpuid (leaf *32 );struct kvm_cpuid_entry2 entry;reverse_cpuid_check (leaf); - kvm_cpu_caps[leaf] &= mask;cpuid_count (cpuid.function, cpuid.index, &entry.eax, &entry.ebx, &entry.ecx, &entry.edx);
@@ -345,6 +346,26 @@ static __always_inline void kvm_cpu_cap_mask(enum cpuid_leafs leaf, u32 mask) kvm_cpu_caps[leaf] &= *__cpuid_entry_get_reg (&entry, cpuid.reg); } +static __always_inline void kvm_cpu_cap_mask(enum cpuid_leafs leaf, u32 mask) +{ +/* Use the "init" variant for scattered leafs. */ +BUILD_BUG_ON (leaf >=NCAPINTS ); + + kvm_cpu_caps[leaf] &= mask; + +__kvm_cpu_cap_mask (leaf); +} + +static __always_inline void kvm_cpu_cap_init(enum cpuid_leafs leaf, u32 mask) +{ +/* Use the "mask" variant for hardwared-defined leafs. */ +BUILD_BUG_ON (leaf <NCAPINTS ); + + kvm_cpu_caps[leaf] = mask; + +__kvm_cpu_cap_mask (leaf); +} +void kvm_set_cpu_caps (void ) {unsigned int f_nx =is_efer_nx () ?F (NX ) :0 ;
@@ -355,12 +376,13 @@ void kvm_set_cpu_caps(void)unsigned int f_gbpages =0 ;unsigned int f_lm =0 ;#endif +memset (kvm_cpu_caps,0 ,sizeof (kvm_cpu_caps)); -BUILD_BUG_ON (sizeof (kvm_cpu_caps) > +BUILD_BUG_ON (sizeof (kvm_cpu_caps) - (NKVMCAPINTS *sizeof (*kvm_cpu_caps)) >sizeof (boot_cpu_data.x86_capability));memcpy (&kvm_cpu_caps, &boot_cpu_data.x86_capability, -sizeof (kvm_cpu_caps)); +sizeof (kvm_cpu_caps) - (NKVMCAPINTS *sizeof (*kvm_cpu_caps))); kvm_cpu_cap_mask(CPUID_1_ECX,/*
arch/x86/kvm/cpuid.h +34 -5
@@ -7,7 +7,20 @@#include <asm/processor.h> #include <uapi/asm/kvm_para.h> -extern u32 kvm_cpu_caps[NCAPINTS ]__read_mostly ; +/* +* Hardware-defined CPUID leafs that are scattered in the kernel, but need to +* be directly used by KVM. Note, these word values conflict with the kernel's +* "bug" caps, but KVM doesn't use those. +*/ +enum kvm_only_cpuid_leafs { +NR_KVM_CPU_CAPS =NCAPINTS , + +NKVMCAPINTS =NR_KVM_CPU_CAPS -NCAPINTS , +}; + +#define X86_KVM_FEATURE (w, f) ((w)*32 + (f)) + +extern u32 kvm_cpu_caps[NR_KVM_CPU_CAPS ]__read_mostly ;void kvm_set_cpu_caps (void );void kvm_update_cpuid_runtime (struct kvm_vcpu *vcpu);
@@ -83,6 +96,20 @@ static __always_inline void reverse_cpuid_check(unsigned int x86_leaf)BUILD_BUG_ON (reverse_cpuid[x86_leaf].function ==0 ); } +/* +* Translate feature bits that are scattered in the kernel's cpufeatures word +* into KVM feature words that align with hardware's definitions. +*/ +static __always_inline u32 __feature_translate(int x86_feature) +{ +return x86_feature; +} + +static __always_inline u32 __feature_leaf(int x86_feature) +{ +return __feature_translate (x86_feature) /32 ; +} +/* * Retrieve the bit mask from an X86_FEATURE_* definition. Features contain * the hardware defined bit number (stored in bits 4:0) and a software defined
@@ -91,6 +118,8 @@ static __always_inline void reverse_cpuid_check(unsigned int x86_leaf)*/ static __always_inline u32 __feature_bit(int x86_feature) { + x86_feature =__feature_translate (x86_feature); +reverse_cpuid_check (x86_feature /32 );return 1 << (x86_feature &31 ); }
@@ -99,7 +128,7 @@ static __always_inline u32 __feature_bit(int x86_feature) static __always_inline struct cpuid_reg x86_feature_cpuid(unsigned int x86_feature) { -unsigned int x86_leaf = x86_feature /32 ; +unsigned int x86_leaf =__feature_leaf (x86_feature);reverse_cpuid_check (x86_leaf);return reverse_cpuid[x86_leaf];
@@ -291,7 +320,7 @@ static inline bool cpuid_fault_enabled(struct kvm_vcpu *vcpu) static __always_inline void kvm_cpu_cap_clear(unsigned int x86_feature) { -unsigned int x86_leaf = x86_feature /32 ; +unsigned int x86_leaf =__feature_leaf (x86_feature);reverse_cpuid_check (x86_leaf); kvm_cpu_caps[x86_leaf] &= ~__feature_bit (x86_feature);
@@ -299,7 +328,7 @@ static __always_inline void kvm_cpu_cap_clear(unsigned int x86_feature) static __always_inline void kvm_cpu_cap_set(unsigned int x86_feature) { -unsigned int x86_leaf = x86_feature /32 ; +unsigned int x86_leaf =__feature_leaf (x86_feature);reverse_cpuid_check (x86_leaf); kvm_cpu_caps[x86_leaf] |=__feature_bit (x86_feature);
@@ -307,7 +336,7 @@ static __always_inline void kvm_cpu_cap_set(unsigned int x86_feature) static __always_inline u32 kvm_cpu_cap_get(unsigned int x86_feature) { -unsigned int x86_leaf = x86_feature /32 ; +unsigned int x86_leaf =__feature_leaf (x86_feature);reverse_cpuid_check (x86_leaf);return kvm_cpu_caps[x86_leaf] &__feature_bit (x86_feature);
Signature · 1 line
2.29.2
RFC PATCH v2 19/26 KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Define a new KVM-only feature word for advertising and querying SGX sub-features in CPUID.0x12.0x0.EAX. Because SGX1 and SGX2 are scattered in the kernel's feature word, they need to be translated so that the bit numbers match those of hardware.
--- arch/x86/kvm/cpuid.h | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
arch/x86/kvm/cpuid.h +12 -1
@@ -13,13 +13,18 @@* "bug" caps, but KVM doesn't use those. */ enum kvm_only_cpuid_leafs { -NR_KVM_CPU_CAPS =NCAPINTS , +CPUID_12_EAX =NCAPINTS , +NR_KVM_CPU_CAPS ,NKVMCAPINTS =NR_KVM_CPU_CAPS -NCAPINTS , };#define X86_KVM_FEATURE (w, f) ((w)*32 + (f)) +/* Intel-defined SGX sub-features, CPUID level 0x12 (EAX). */ +#define __X86_FEATURE_SGX1 X86_KVM_FEATURE(CPUID_12_EAX, 0) +#define __X86_FEATURE_SGX2 X86_KVM_FEATURE(CPUID_12_EAX, 1) +extern u32 kvm_cpu_caps[NR_KVM_CPU_CAPS ]__read_mostly ;void kvm_set_cpu_caps (void );
@@ -76,6 +81,7 @@ static const struct cpuid_reg reverse_cpuid[] = { [CPUID_8000_0007_EBX] = {0x80000007, 0, CPUID_EBX}, [CPUID_7_EDX] = { 7, 0, CPUID_EDX}, [CPUID_7_1_EAX] = { 7, 1, CPUID_EAX}, + [CPUID_12_EAX] = {0x00000012, 0, CPUID_EAX}, };/*
@@ -102,6 +108,11 @@ static __always_inline void reverse_cpuid_check(unsigned int x86_leaf)*/ static __always_inline u32 __feature_translate(int x86_feature) { +if (x86_feature ==X86_FEATURE_SGX1 ) +return __X86_FEATURE_SGX1; +else if (x86_feature ==X86_FEATURE_SGX2 ) +return __X86_FEATURE_SGX2; +return x86_feature; }
Signature · 1 line
2.29.2
RFC PATCH v2 20/26 KVM: VMX: Add basic handling of VM-Exit from SGX enclave
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add support for handling VM-Exits that originate from a guest SGX enclave. In SGX, an "enclave" is a new CPL3-only execution environment, wherein the CPU and memory state is protected by hardware to make the state inaccesible to code running outside of the enclave. When exiting an enclave due to an asynchronous event (from the perspective of the enclave), e.g. exceptions, interrupts, and VM-Exits, the enclave's state is automatically saved and scrubbed (the CPU loads synthetic state), and then reloaded when re-entering the enclave. E.g. after an instruction based VM-Exit from an enclave, vmcs.GUEST_RIP will not contain the RIP of the enclave instruction that trigered VM-Exit, but will instead point to a RIP in the enclave's untrusted runtime (the guest userspace code that coordinates entry/exit to/from the enclave).
To help a VMM recognize and handle exits from enclaves, SGX adds bits to existing VMCS fields, VM_EXIT_REASON.VMX_EXIT_REASON_FROM_ENCLAVE and GUEST_INTERRUPTIBILITY_INFO.GUEST_INTR_STATE_ENCLAVE_INTR. Define the new architectural bits, and add a boolean to struct vcpu_vmx to cache VMX_EXIT_REASON_FROM_ENCLAVE. Clear the bit in exit_reason so that checks against exit_reason do not need to account for SGX, e.g. "if (exit_reason == EXIT_REASON_EXCEPTION_NMI)" continues to work.
KVM is a largely a passive observer of the new bits, e.g. KVM needs to account for the bits when propagating information to a nested VMM, but otherwise doesn't need to act differently for the majority of VM-Exits from enclaves.
The one scenario that is directly impacted is emulation, which is for all intents and purposes impossible[1] since KVM does not have access to the RIP or instruction stream that triggered the VM-Exit. The inability to emulate is a non-issue for KVM, as most instructions that might trigger VM-Exit unconditionally #UD in an enclave (before the VM-Exit check. For the few instruction that conditionally #UD, KVM either never sets the exiting control, e.g. PAUSE_EXITING[2], or sets it if and only if the feature is not exposed to the guest in order to inject a #UD, e.g. RDRAND_EXITING.
But, because it is still possible for a guest to trigger emulation, e.g. MMIO, inject a #UD if KVM ever attempts emulation after a VM-Exit from an enclave. This is architecturally accurate for instruction VM-Exits, and for MMIO it's the least bad choice, e.g. it's preferable to killing the VM. In practice, only broken or particularly stupid guests should ever encounter this behavior.
Add a WARN in skip_emulated_instruction to detect any attempt to modify the guest's RIP during an SGX enclave VM-Exit as all such flows should either be unreachable or must handle exits from enclaves before getting to skip_emulated_instruction.
[1] Impossible for all practical purposes. Not truly impossible since KVM could implement some form of para-virtualization scheme.
[2] PAUSE_LOOP_EXITING only affects CPL0 and enclaves exist only at CPL3, so we also don't need to worry about that interaction.
--- arch/x86/include/asm/vmx.h | 1 + arch/x86/include/uapi/asm/vmx.h | 1 + arch/x86/kvm/vmx/nested.c | 2 ++ arch/x86/kvm/vmx/vmx.c | 38 +++++++++++++++++++++++++++++++-- 4 files changed, 40 insertions(+), 2 deletions(-)
arch/x86/include/asm/vmx.h +1 -0
@@ -372,6 +372,7 @@ enum vmcs_field {#define GUEST_INTR_STATE_MOV_SS 0x00000002#define GUEST_INTR_STATE_SMI 0x00000004#define GUEST_INTR_STATE_NMI 0x00000008 +#define GUEST_INTR_STATE_ENCLAVE_INTR 0x00000010/* GUEST_ACTIVITY_STATE flags */ #define GUEST_ACTIVITY_ACTIVE 0
arch/x86/include/uapi/asm/vmx.h +1 -0
@@ -27,6 +27,7 @@#define VMX_EXIT_REASONS_FAILED_VMENTRY 0x80000000 +#define VMX_EXIT_REASONS_SGX_ENCLAVE_MODE 0x08000000#define EXIT_REASON_EXCEPTION_NMI 0#define EXIT_REASON_EXTERNAL_INTERRUPT 1
arch/x86/kvm/vmx/nested.c +2 -0
@@ -4126,6 +4126,8 @@ static void prepare_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, {/* update exit information fields: */ vmcs12->vm_exit_reason = vm_exit_reason; +if (to_vmx (vcpu)->exit_reason.sgx_enclave_mode) + vmcs12->vm_exit_reason |=VMX_EXIT_REASONS_SGX_ENCLAVE_MODE ; vmcs12->exit_qualification = exit_qualification; vmcs12->vm_exit_intr_info = exit_intr_info;
arch/x86/kvm/vmx/vmx.c +36 -2
@@ -1561,12 +1561,18 @@ static int vmx_rtit_ctl_check(struct kvm_vcpu *vcpu, u64 data)static bool vmx_can_emulate_instruction (struct kvm_vcpu *vcpu,void *insn,int insn_len) { +if (to_vmx (vcpu)->exit_reason.sgx_enclave_mode) { +kvm_queue_exception (vcpu,UD_VECTOR ); +return false; + }return true; }static int skip_emulated_instruction (struct kvm_vcpu *vcpu) { +union vmx_exit_reason exit_reason =to_vmx (vcpu)->exit_reason;unsigned long rip, orig_rip; +u32 instr_len;/* * Using VMCS.VM_EXIT_INSTRUCTION_LEN on EPT misconfig depends on
@@ -1577,9 +1583,33 @@ static int skip_emulated_instruction(struct kvm_vcpu *vcpu)* i.e. we end up advancing IP with some random value. */ if (!static_cpu_has (X86_FEATURE_HYPERVISOR ) || -to_vmx (vcpu)->exit_reason.basic !=EXIT_REASON_EPT_MISCONFIG ) { + exit_reason.basic !=EXIT_REASON_EPT_MISCONFIG ) { + instr_len =vmcs_read32 (VM_EXIT_INSTRUCTION_LEN ); + +/* +* Emulating an enclave's instructions isn't supported as KVM +* cannot access the enclave's memory or its true RIP, e.g. the +* vmcs.GUEST_RIP points at the exit point of the enclave, not +* the RIP that actually triggered the VM-Exit. But, because +* most instructions that cause VM-Exit will #UD in an enclave, +* most instruction-based VM-Exits simply do not occur. +* +* There are a few exceptions, notably the debug instructions +* INT1ICEBRK and INT3, as they are allowed in debug enclaves +* and generate #DB/#BP as expected, which KVM might intercept. +* But again, the CPU does the dirty work and saves an instr +* length of zero so VMMs don't shoot themselves in the foot. +* WARN if KVM tries to skip a non-zero length instruction on +* a VM-Exit from an enclave. +*/ +if (!instr_len) + goto rip_updated; + +WARN (exit_reason.sgx_enclave_mode, +"KVM: skipping instruction after SGX enclave VM-Exit" ); + orig_rip =kvm_rip_read (vcpu); - rip = orig_rip +vmcs_read32 (VM_EXIT_INSTRUCTION_LEN ); + rip = orig_rip + instr_len;#ifdef CONFIG_X86_64 /* * We need to mask out the high 32 bits of RIP if not in 64-bit
@@ -1595,6 +1625,7 @@ static int skip_emulated_instruction(struct kvm_vcpu *vcpu)return 0 ; } +rip_updated:/* skipping an emulated instruction also counts */ vmx_set_interrupt_shadow (vcpu,0 );
@@ -5341,6 +5372,9 @@ static int handle_ept_misconfig(struct kvm_vcpu *vcpu) {gpa_t gpa; +if (!vmx_can_emulate_instruction (vcpu,NULL ,0 )) +return 1 ; +/* * A nested guest cannot optimize MMIO vmexits, because we have an * nGPA here instead of the required GPA.
Signature · 1 line
2.29.2
RFC PATCH v2 21/26 KVM: VMX: Frame in ENCLS handler for SGX virtualization
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Introduce sgx.c and sgx.h, along with the framework for handling ENCLS VM-Exits. Add a bool, enable_sgx, that will eventually be wired up to a module param to control whether or not SGX virtualization is enabled at runtime.
--- arch/x86/kvm/Makefile | 2 ++ arch/x86/kvm/vmx/sgx.c | 51 ++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/sgx.h | 15 +++++++++++++ arch/x86/kvm/vmx/vmx.c | 9 +++++--- 4 files changed, 74 insertions(+), 3 deletions(-) create mode 100644 arch/x86/kvm/vmx/sgx.c create mode 100644 arch/x86/kvm/vmx/sgx.h
arch/x86/kvm/Makefile +2 -0
@@ -21,6 +21,8 @@ kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \ kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \ vmx/evmcs.o vmx/nested.o vmx/posted_intr.o +kvm-intel-$(CONFIG_X86_SGX_VIRTUALIZATION) += vmx/sgx.o + kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o svm/sev.o obj-$(CONFIG_KVM) += kvm.o
arch/x86/kvm/vmx/sgx.c new file +51 -0
51 lines not on this page · raw message
arch/x86/kvm/vmx/sgx.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_SGX_H +#define __KVM_X86_SGX_H + +#include <linux/kvm_host.h> + +#ifdef CONFIG_X86_SGX_VIRTUALIZATION +extern bool __read_mostly enable_sgx; + +int handle_encls (struct kvm_vcpu *vcpu); +#else +#define enable_sgx 0 +#endif + +#endif /* __KVM_X86_SGX_H */
arch/x86/kvm/vmx/vmx.c +6 -3
@@ -56,6 +56,7 @@#include "mmu.h" #include "nested.h" #include "pmu.h" +#include "sgx.h" #include "trace.h" #include "vmcs.h" #include "vmcs12.h"
@@ -5623,16 +5624,18 @@ static int handle_vmx_instruction(struct kvm_vcpu *vcpu)return 1 ; } +#ifndef CONFIG_X86_SGX_VIRTUALIZATION static int handle_encls (struct kvm_vcpu *vcpu) {/* -* SGX virtualization is not yet supported. There is no software -* enable bit for SGX, so we have to trap ENCLS and inject a #UD -* to prevent the guest from executing ENCLS. +* SGX virtualization is disabled. There is no software enable bit for +* SGX, so KVM intercepts all ENCLS leafs and injects a #UD to prevent +* the guest from executing ENCLS (when SGX is supported by hardware). */ kvm_queue_exception (vcpu,UD_VECTOR );return 1 ; } +#endif /* CONFIG_X86_SGX_VIRTUALIZATION */ /* * The exit handlers return 1 if the exit was handled fully and guest execution
Signature · 1 line
2.29.2
RFC PATCH v2 22/26 KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add an ECREATE handler that will be used to intercept ECREATE for the purpose of enforcing and enclave's MISCSELECT, ATTRIBUTES and XFRM, i.e. to allow userspace to restrict SGX features via CPUID. ECREATE will be intercepted when any of the aforementioned masks diverges from hardware in order to enforce the desired CPUID model, i.e. inject #GP if the guest attempts to set a bit that hasn't been enumerated as allowed-1 in CPUID.
Note, access to the PROVISIONKEY is not yet supported.
--- arch/x86/include/asm/kvm_host.h | 3 + arch/x86/kvm/vmx/sgx.c | 243 ++++++++++++++++++++++++++++++++ 2 files changed, 246 insertions(+)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -1000,6 +1000,9 @@ struct kvm_arch {struct msr_bitmap_range ranges[16 ]; } msr_filter; +/* Guest can access the SGX PROVISIONKEY. */ +bool sgx_provisioning_allowed; +struct kvm_pmu_event_filter *pmu_event_filter;struct task_struct *nx_lpage_recovery_thread;
arch/x86/kvm/vmx/sgx.c +243 -0
255 lines not on this page · raw message
Signature · 1 line
2.29.2
RFC PATCH v2 23/26 KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Emulate the four Launch Enclave public key hash MSRs (LE hash MSRs) that exist on CPUs that support SGX Launch Control (LC). SGX LC modifies the behavior of ENCLS[EINIT] to use the LE hash MSRs when verifying the key used to sign an enclave. On CPUs without LC support, the LE hash is hardwired into the CPU to an Intel controlled key (the Intel key is also the reset value of the LE hash MSRs). Track the guest's desired hash so that a future patch can stuff the hash into the hardware MSRs when executing EINIT on behalf of the guest, when those MSRs are writable in host.
Note, KVM allows writes to the LE hash MSRs if IA32_FEATURE_CONTROL is unlocked. This is technically not architectural behavior, but it's roughly equivalent to the arch behavior of the MSRs being writable prior to activating SGX[1]. Emulating SGX activation is feasible, but adds no tangible benefits and would just create extra work for KVM and guest firmware.
[1] SGX related bits in IA32_FEATURE_CONTROL cannot be set until SGX is activated, e.g. by firmware. SGX activation is triggered by setting bit 0 in MSR 0x7a. Until SGX is activated, the LE hash MSRs are writable, e.g. to allow firmware to lock down the LE root key with a non-Intel value.
--- arch/x86/kvm/vmx/sgx.c | 35 +++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/sgx.h | 6 ++++++ arch/x86/kvm/vmx/vmx.c | 20 ++++++++++++++++++++ arch/x86/kvm/vmx/vmx.h | 2 ++ 4 files changed, 63 insertions(+)
arch/x86/kvm/vmx/sgx.c +35 -0
@@ -12,6 +12,9 @@bool __read_mostly enable_sgx; +/* Initial value of guest's virtual SGX_LEPUBKEYHASHn MSRs */ +static u64 sgx_pubkey_hash[4 ]__ro_after_init ; +/* * ENCLS's memory operands use a fixed segment (DS) and a fixed * address size based on the mode. Related prefixes are ignored.
@@ -292,3 +295,35 @@ int handle_encls(struct kvm_vcpu *vcpu) }return 1 ; } + +void setup_default_sgx_lepubkeyhash (void ) +{ +/* +* Use Intel's default value for Skylake hardware if Launch Control is +* not supported, i.e. Intel's hash is hardcoded into silicon, or if +* Launch Control is supported and enabled, i.e. mimic the reset value +* and let the guest write the MSRs at will. If Launch Control is +* supported but disabled, then use the current MSR values as the hash +* MSRs exist but are read-only (locked and not writable). +*/ +if (!enable_sgx || !boot_cpu_has (X86_FEATURE_SGX_LC ) || +rdmsrl_safe (MSR_IA32_SGXLEPUBKEYHASH0 , &sgx_pubkey_hash[0 ])) { + sgx_pubkey_hash[0 ] =0xa6053e051270b7acULL ; + sgx_pubkey_hash[1 ] =0x6cfbe8ba8b3b413dULL ; + sgx_pubkey_hash[2 ] =0xc4916d99f2b3735dULL ; + sgx_pubkey_hash[3 ] =0xd4f8c05909f9bb3bULL ; + }else { +/* MSR_IA32_SGXLEPUBKEYHASH0 is read above */ +rdmsrl (MSR_IA32_SGXLEPUBKEYHASH1 , sgx_pubkey_hash[1 ]); +rdmsrl (MSR_IA32_SGXLEPUBKEYHASH2 , sgx_pubkey_hash[2 ]); +rdmsrl (MSR_IA32_SGXLEPUBKEYHASH3 , sgx_pubkey_hash[3 ]); + } +} + +void vcpu_setup_sgx_lepubkeyhash (struct kvm_vcpu *vcpu) +{ +struct vcpu_vmx *vmx =to_vmx (vcpu); + +memcpy (vmx->msr_ia32_sgxlepubkeyhash, sgx_pubkey_hash, +sizeof (sgx_pubkey_hash)); +}
arch/x86/kvm/vmx/sgx.h +6 -0
@@ -8,8 +8,14 @@extern bool __read_mostly enable_sgx;int handle_encls (struct kvm_vcpu *vcpu); + +void setup_default_sgx_lepubkeyhash (void ); +void vcpu_setup_sgx_lepubkeyhash (struct kvm_vcpu *vcpu);#else #define enable_sgx 0 + +static inline void setup_default_sgx_lepubkeyhash (void ) { } +static inline void vcpu_setup_sgx_lepubkeyhash (struct kvm_vcpu *vcpu) { }#endif #endif /* __KVM_X86_SGX_H */
arch/x86/kvm/vmx/vmx.c +20 -0
@@ -1888,6 +1888,13 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)case MSR_IA32_FEAT_CTL : msr_info->data = vmx->msr_ia32_feature_control;break ; + case MSR_IA32_SGXLEPUBKEYHASH0 ... MSR_IA32_SGXLEPUBKEYHASH3: +if (!msr_info->host_initiated && + !guest_cpuid_has (vcpu,X86_FEATURE_SGX_LC )) +return 1 ; + msr_info->data =to_vmx (vcpu)->msr_ia32_sgxlepubkeyhash + [msr_info->index -MSR_IA32_SGXLEPUBKEYHASH0 ]; +break ; case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:if (!nested_vmx_allowed (vcpu))return 1 ;
@@ -2154,6 +2161,15 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (msr_info->host_initiated && data ==0 )vmx_leave_nested (vcpu);break ; + case MSR_IA32_SGXLEPUBKEYHASH0 ... MSR_IA32_SGXLEPUBKEYHASH3: +if (!msr_info->host_initiated && + (!guest_cpuid_has (vcpu,X86_FEATURE_SGX_LC ) || + ((vmx->msr_ia32_feature_control &FEAT_CTL_LOCKED ) && + !(vmx->msr_ia32_feature_control &FEAT_CTL_SGX_LC_ENABLED )))) +return 1 ; + vmx->msr_ia32_sgxlepubkeyhash + [msr_index -MSR_IA32_SGXLEPUBKEYHASH0 ] = data; +break ; case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:if (!msr_info->host_initiated)return 1 ;/* they are read-only */
@@ -6957,6 +6973,8 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu)else memset (&vmx->nested.msrs,0 ,sizeof (vmx->nested.msrs)); +vcpu_setup_sgx_lepubkeyhash (vcpu); + vmx->nested.posted_intr_nv =-1 ; vmx->nested.current_vmptr =-1ull ;
@@ -7907,6 +7925,8 @@ static __init int hardware_setup(void)if (!enable_ept || !cpu_has_vmx_intel_pt ()) pt_mode =PT_MODE_SYSTEM ; +setup_default_sgx_lepubkeyhash (); +if (nested) {nested_vmx_setup_ctls_msrs (&vmcs_config.nested, vmx_capability.ept);
arch/x86/kvm/vmx/vmx.h +2 -0
@@ -299,6 +299,8 @@ struct vcpu_vmx {*/ u64 msr_ia32_feature_control;u64 msr_ia32_feature_control_valid_bits; +/* SGX Launch Control public key hash */ +u64 msr_ia32_sgxlepubkeyhash[4 ];u64 ept_pointer;struct pt_desc pt_desc;
Signature · 1 line
2.29.2
RFC PATCH v2 26/26 KVM: x86: Add capability to grant VM access to privileged SGX attribute
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add a capability, KVM_CAP_SGX_ATTRIBUTE, that can be used by userspace to grant a VM access to a priveleged attribute, with args[0] holding a file handle to a valid SGX attribute file.
The SGX subsystem restricts access to a subset of enclave attributes to provide additional security for an uncompromised kernel, e.g. to prevent malware from using the PROVISIONKEY to ensure its nodes are running inside a geniune SGX enclave and/or to obtain a stable fingerprint.
To prevent userspace from circumventing such restrictions by running an enclave in a VM, KVM restricts guest access to privileged attributes by default.
--- Documentation/virt/kvm/api.rst | 23 +++++++++++++++++++++++ arch/x86/kvm/cpuid.c | 2 +- arch/x86/kvm/x86.c | 22 ++++++++++++++++++++++ include/uapi/linux/kvm.h | 1 + 4 files changed, 47 insertions(+), 1 deletion(-)
Documentation/virt/kvm/api.rst +23 -0
@@ -6037,6 +6037,29 @@ KVM_EXIT_X86_RDMSR and KVM_EXIT_X86_WRMSR exit notifications which user space can then handle to implement model specific MSR handling and/or user notifications to inform a user that an MSR was not handled. +7.22 KVM_CAP_SGX_ATTRIBUTE +---------------------- + +:Architectures: x86 +:Target: VM +:Parameters: args[0] is a file handle of a SGX attribute file in securityfs +:Returns: 0 on success, -EINVAL if the file handle is invalid or if a requested + attribute is not supported by KVM. + +KVM_CAP_SGX_ATTRIBUTE enables a userspace VMM to grant a VM access to one or +more priveleged enclave attributes. args[0] must hold a file handle to a valid +SGX attribute file corresponding to an attribute that is supported/restricted +by KVM (currently only PROVISIONKEY). + +The SGX subsystem restricts access to a subset of enclave attributes to provide +additional security for an uncompromised kernel, e.g. use of the PROVISIONKEY +is restricted to deter malware from using the PROVISIONKEY to obtain a stable +system fingerprint. To prevent userspace from circumventing such restrictions +by running an enclave in a VM, KVM prevents access to privileged attributes by +default. + +See Documentation/x86/sgx/2.Kernel-internals.rst for more details. + 8. Other capabilities. ======================
arch/x86/kvm/cpuid.c +1 -1
@@ -833,7 +833,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* expected to derive it from supported XCR0. */ entry->eax &=SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT | -/* PROVISIONKEY | */ SGX_ATTR_EINITTOKENKEY | +SGX_ATTR_PROVISIONKEY |SGX_ATTR_EINITTOKENKEY |SGX_ATTR_KSS ; entry->ebx &=0 ;break ;
arch/x86/kvm/x86.c +22 -0
@@ -74,6 +74,8 @@#include <asm/tlbflush.h> #include <asm/intel_pt.h> #include <asm/emulate_prefix.h> +#include <asm/sgx.h> +#include <asm/sgx_arch.h> #include <clocksource/hyperv_timer.h> #define CREATE_TRACE_POINTS
@@ -3767,6 +3769,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_X86_USER_SPACE_MSR :case KVM_CAP_X86_MSR_FILTER :case KVM_CAP_ENFORCE_PV_FEATURE_CPUID : +#ifdef CONFIG_X86_SGX_VIRTUALIZATION +case KVM_CAP_SGX_ATTRIBUTE : +#endif r =1 ;break ;case KVM_CAP_SYNC_REGS :
@@ -5295,6 +5300,23 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, kvm->arch.user_space_msr_mask = cap->args[0 ]; r =0 ;break ; +#ifdef CONFIG_X86_SGX_VIRTUALIZATION +case KVM_CAP_SGX_ATTRIBUTE : { +unsigned long allowed_attributes =0 ; + + r =sgx_set_attribute (&allowed_attributes, cap->args[0 ]); +if (r) +break ; + +/* KVM only supports the PROVISIONKEY privileged attribute. */ +if ((allowed_attributes &SGX_ATTR_PROVISIONKEY ) && + !(allowed_attributes & ~SGX_ATTR_PROVISIONKEY )) + kvm->arch.sgx_provisioning_allowed = true; +else + r = -EINVAL ; +break ; + } +#endif default : r = -EINVAL ;break ;
include/uapi/linux/kvm.h +1 -0
@@ -1058,6 +1058,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_ENFORCE_PV_FEATURE_CPUID 190#define KVM_CAP_SYS_HYPERV_CPUID 191#define KVM_CAP_DIRTY_LOG_RING 192 +#define KVM_CAP_SGX_ATTRIBUTE 200#ifdef KVM_CAP_IRQ_ROUTING
Signature · 1 line
2.29.2
RFC PATCH v2 24/26 KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add a VM-Exit handler to trap-and-execute EINIT when SGX LC is enabled in the host. When SGX LC is enabled, the host kernel may rewrite the hardware values at will, e.g. to launch enclaves with different signers, thus KVM needs to intercept EINIT to ensure it is executed with the correct LE hash (even if the guest sees a hardwired hash).
Switching the LE hash MSRs on VM-Enter/VM-Exit is not a viable option as writing the MSRs is prohibitively expensive, e.g. on SKL hardware each WRMSR is ~400 cycles. And because EINIT takes tens of thousands of cycles to execute, the ~1500 cycle overhead to trap-and-execute EINIT is unlikely to be noticed by the guest, let alone impact its overall SGX performance.
--- arch/x86/kvm/vmx/sgx.c | 55 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)
arch/x86/kvm/vmx/sgx.c +55 -0
@@ -256,6 +256,59 @@ static int handle_encls_ecreate(struct kvm_vcpu *vcpu)return kvm_skip_emulated_instruction (vcpu); } +static int handle_encls_einit (struct kvm_vcpu *vcpu) +{ +unsigned long sig_hva, secs_hva, token_hva, rflags; +struct vcpu_vmx *vmx =to_vmx (vcpu); +gva_t sig_gva, secs_gva, token_gva; +gpa_t sig_gpa, secs_gpa, token_gpa; +int ret, trapnr; + +if (sgx_get_encls_gva (vcpu,kvm_rbx_read (vcpu),1808 ,4096 , &sig_gva) || +sgx_get_encls_gva (vcpu,kvm_rcx_read (vcpu),4096 ,4096 , &secs_gva) || +sgx_get_encls_gva (vcpu,kvm_rdx_read (vcpu),304 ,512 , &token_gva)) +return 1 ; + +/* +* Translate the SIGSTRUCT, SECS and TOKEN pointers from GVA to GPA. +* Resume the guest on failure to inject a #PF. +*/ +if (sgx_gva_to_gpa (vcpu, sig_gva, false, &sig_gpa) || +sgx_gva_to_gpa (vcpu, secs_gva, true, &secs_gpa) || +sgx_gva_to_gpa (vcpu, token_gva, false, &token_gpa)) +return 1 ; + +/* +* ...and then to HVA. The order of accesses isn't architectural, i.e. +* KVM doesn't have to fully process one address at a time. Exit to +* userspace if a GPA is invalid. Note, all structures are aligned and +* cannot split pages. +*/ +if (sgx_gpa_to_hva (vcpu, sig_gpa, &sig_hva) || +sgx_gpa_to_hva (vcpu, secs_gpa, &secs_hva) || +sgx_gpa_to_hva (vcpu, token_gpa, &token_hva)) +return 0 ; + + ret =sgx_virt_einit ((void __user *)sig_hva, (void __user *)token_hva, + (void __user *)secs_hva, + vmx->msr_ia32_sgxlepubkeyhash, &trapnr); + +if (ret == -EFAULT ) +return sgx_inject_fault (vcpu, secs_gva, trapnr); + + rflags =vmx_get_rflags (vcpu) & ~(X86_EFLAGS_CF |X86_EFLAGS_PF | +X86_EFLAGS_AF |X86_EFLAGS_SF | +X86_EFLAGS_OF ); +if (ret) + rflags |=X86_EFLAGS_ZF ; +else + rflags &= ~X86_EFLAGS_ZF ; +vmx_set_rflags (vcpu, rflags); + +kvm_rax_write (vcpu, ret); +return kvm_skip_emulated_instruction (vcpu); +} +static inline bool encls_leaf_enabled_in_guest (struct kvm_vcpu *vcpu,u32 leaf) {if (!enable_sgx || !guest_cpuid_has (vcpu,X86_FEATURE_SGX ))
@@ -288,6 +341,8 @@ int handle_encls(struct kvm_vcpu *vcpu) }else {if (leaf ==ECREATE )return handle_encls_ecreate (vcpu); +if (leaf ==EINIT ) +return handle_encls_einit (vcpu);WARN (1 ,"KVM: unexpected exit on ENCLS[%u]" , leaf); vcpu->run->exit_reason =KVM_EXIT_UNKNOWN ; vcpu->run->hw.hardware_exit_reason =EXIT_REASON_ENCLS ;
Signature · 1 line
2.29.2
RFC PATCH v2 25/26 KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Enable SGX virtualization now that KVM has the VM-Exit handlers needed to trap-and-execute ENCLS to ensure correctness and/or enforce the CPU model exposed to the guest. Add a KVM module param, "sgx", to allow an admin to disable SGX virtualization independent of the kernel.
When supported in hardware and the kernel, advertise SGX1, SGX2 and SGX LC to userspace via CPUID and wire up the ENCLS_EXITING bitmap based on the guest's SGX capabilities, i.e. to allow ENCLS to be executed in an SGX-enabled guest. With the exception of the provision key, all SGX attribute bits may be exposed to the guest. Guest access to the provision key, which is controlled via securityfs, will be added in a future patch.
Note, KVM does not yet support exposing ENCLS_C leafs or ENCLV leafs.
---v1->v2:
- Changed to use kvm_cpu_cap_init() instead of kvm_cpu_cap_mask() to expose SGX1 and SGX2 due to they are scattered in X86_FEATURE_SGX1/SGX2.
--- arch/x86/kvm/cpuid.c | 57 +++++++++++++++++++++++++++- arch/x86/kvm/vmx/nested.c | 26 +++++++++++-- arch/x86/kvm/vmx/nested.h | 5 +++ arch/x86/kvm/vmx/sgx.c | 80 ++++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/sgx.h | 13 +++++++ arch/x86/kvm/vmx/vmcs12.c | 1 + arch/x86/kvm/vmx/vmcs12.h | 4 +- arch/x86/kvm/vmx/vmx.c | 38 ++++++++++++++++++- 8 files changed, 215 insertions(+), 9 deletions(-)
arch/x86/kvm/cpuid.c +55 -2
@@ -18,6 +18,7 @@#include <asm/processor.h> #include <asm/user.h> #include <asm/fpu/xstate.h> +#include <asm/sgx_arch.h> #include "cpuid.h" #include "lapic.h" #include "mmu.h"
@@ -171,6 +172,21 @@ static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) vcpu->arch.guest_supported_xcr0 = (best->eax | ((u64 )best->edx <<32 )) & supported_xcr0; +/* +* Bits 127:0 of the allowed SECS.ATTRIBUTES (CPUID.0x12.0x1) enumerate +* the supported XSAVE Feature Request Mask (XFRM), i.e. the enclave's +* requested XCR0 value. The enclave's XFRM must be a subset of XCRO +* at the time of EENTER, thus adjust the allowed XFRM by the guest's +* supported XCR0. Similar to XCR0 handling, FP and SSE are forced to +* '1' even on CPUs that don't support XSAVE. +*/ + best =kvm_find_cpuid_entry (vcpu,0x12 ,0x1 ); +if (best) { + best->ecx &= vcpu->arch.guest_supported_xcr0 &0xffffffff ; + best->edx &= vcpu->arch.guest_supported_xcr0 >>32 ; + best->ecx |=XFEATURE_MASK_FPSSE ; + } +kvm_update_pv_runtime (vcpu); vcpu->arch.maxphyaddr =cpuid_query_maxphyaddr (vcpu);
@@ -413,7 +429,7 @@ void kvm_set_cpu_caps(void) ); kvm_cpu_cap_mask(CPUID_7_0_EBX, - F(FSGSBASE) | F(BMI1) | F(HLE) | F(AVX2) | F(SMEP) | + F(FSGSBASE) | F(SGX) | F(BMI1) | F(HLE) | F(AVX2) | F(SMEP) | F(BMI2) | F(ERMS) | 0 /*INVPCID*/ | F(RTM) | 0 /*MPX*/ | F(RDSEED) | F(ADX) | F(SMAP) | F(AVX512IFMA) | F(AVX512F) | F(AVX512PF) | F(AVX512ER) | F(AVX512CD) | F(CLFLUSHOPT) | F(CLWB) | F(AVX512DQ) |
@@ -424,7 +440,8 @@ void kvm_set_cpu_caps(void)F (AVX512VBMI ) |F (LA57 ) |F (PKU ) |0 /*OSPKE*/ |F (RDPID ) |F (AVX512_VPOPCNTDQ ) |F (UMIP ) |F (AVX512_VBMI2 ) |F (GFNI ) |F (VAES ) |F (VPCLMULQDQ ) |F (AVX512_VNNI ) |F (AVX512_BITALG ) | -F (CLDEMOTE ) |F (MOVDIRI ) |F (MOVDIR64B ) |0 /*WAITPKG*/ +F (CLDEMOTE ) |F (MOVDIRI ) |F (MOVDIR64B ) |0 /*WAITPKG*/ | +F (SGX_LC ) );/* Set LA57 based on hardware capability. */ if (cpuid_ecx (7 ) &F (LA57 ))
@@ -463,6 +480,10 @@ void kvm_set_cpu_caps(void)F (XSAVEOPT ) |F (XSAVEC ) |F (XGETBV1 ) |F (XSAVES ) ); +kvm_cpu_cap_init (CPUID_12_EAX , +SF (SGX1 ) |SF (SGX2 ) + ); + kvm_cpu_cap_mask(CPUID_8000_0001_ECX, F(LAHF_LM) | F(CMP_LEGACY) | 0 /*SVM*/ | 0 /* ExtApicSpace */ | F(CR8_LEGACY) | F(ABM) | F(SSE4A) | F(MISALIGNSSE) |
@@ -784,6 +805,38 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) entry->edx =0 ; }break ; +case 0x12 : +/* Intel SGX */ +if (!kvm_cpu_cap_has (X86_FEATURE_SGX )) { + entry->eax = entry->ebx = entry->ecx = entry->edx =0 ; +break ; + } + +/* +* Index 0: Sub-features, MISCSELECT (a.k.a extended features) +* and max enclave sizes. The SGX sub-features and MISCSELECT +* are restricted by kernel and KVM capabilities (like most +* feature flags), while enclave size is unrestricted. +*/ +cpuid_entry_override (entry,CPUID_12_EAX ); + entry->ebx &=SGX_MISC_EXINFO ; + + entry =do_host_cpuid (array, function,1 ); +if (!entry) + goto out; + +/* +* Index 1: SECS.ATTRIBUTES. ATTRIBUTES are restricted a la +* feature flags. Advertise all supported flags, including +* privileged attributes that require explicit opt-in from +* userspace. ATTRIBUTES.XFRM is not adjusted as userspace is +* expected to derive it from supported XCR0. +*/ + entry->eax &=SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT | +/* PROVISIONKEY | */ SGX_ATTR_EINITTOKENKEY | +SGX_ATTR_KSS ; + entry->ebx &=0 ; +break ;/* Intel PT */ case 0x14 :if (!kvm_cpu_cap_has (X86_FEATURE_INTEL_PT )) {
arch/x86/kvm/vmx/nested.c +23 -3
@@ -11,6 +11,7 @@#include "mmu.h" #include "nested.h" #include "pmu.h" +#include "sgx.h" #include "trace.h" #include "x86.h"
@@ -2318,6 +2319,9 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)if (!nested_cpu_has2 (vmcs12,SECONDARY_EXEC_UNRESTRICTED_GUEST )) exec_control &= ~SECONDARY_EXEC_UNRESTRICTED_GUEST ; +if (exec_control &SECONDARY_EXEC_ENCLS_EXITING ) +vmx_write_encls_bitmap (&vmx->vcpu, vmcs12); +secondary_exec_controls_set (vmx, exec_control); }
@@ -5726,6 +5730,20 @@ static bool nested_vmx_exit_handled_cr(struct kvm_vcpu *vcpu,return false; } +static bool nested_vmx_exit_handled_encls (struct kvm_vcpu *vcpu, +struct vmcs12 *vmcs12) +{ +u32 encls_leaf; + +if (!nested_cpu_has2 (vmcs12,SECONDARY_EXEC_ENCLS_EXITING )) +return false; + + encls_leaf =kvm_rax_read (vcpu); +if (encls_leaf >62 ) + encls_leaf =63 ; +return vmcs12->encls_exiting_bitmap &BIT_ULL (encls_leaf); +} +static bool nested_vmx_exit_handled_vmcs_access (struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12,gpa_t bitmap) {
@@ -5819,9 +5837,6 @@ static bool nested_vmx_l0_wants_exit(struct kvm_vcpu *vcpu,case EXIT_REASON_VMFUNC :/* VM functions are emulated through L2->L0 vmexits. */ return true; -case EXIT_REASON_ENCLS : -/* SGX is never exposed to L1 */ -return true;default :break ; }
@@ -5945,6 +5960,8 @@ static bool nested_vmx_l1_wants_exit(struct kvm_vcpu *vcpu,case EXIT_REASON_TPAUSE :return nested_cpu_has2 (vmcs12,SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE ); +case EXIT_REASON_ENCLS : +return nested_vmx_exit_handled_encls (vcpu, vmcs12);default :return true; }
@@ -6517,6 +6534,9 @@ void nested_vmx_setup_ctls_msrs(struct nested_vmx_msrs *msrs, u32 ept_caps) msrs->secondary_ctls_high |=SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; +if (enable_sgx) + msrs->secondary_ctls_high |=SECONDARY_EXEC_ENCLS_EXITING ; +/* miscellaneous data */ rdmsr(MSR_IA32_VMX_MISC, msrs->misc_low,
arch/x86/kvm/vmx/nested.h +5 -0
@@ -244,6 +244,11 @@ static inline bool nested_exit_on_intr(struct kvm_vcpu *vcpu)PIN_BASED_EXT_INTR_MASK ; } +static inline bool nested_cpu_has_encls_exit (struct vmcs12 *vmcs12) +{ +return nested_cpu_has2 (vmcs12,SECONDARY_EXEC_ENCLS_EXITING ); +} +/* * if fixed0[i] == 1: val[i] must be 1 * if fixed1[i] == 0: val[i] must be 0
arch/x86/kvm/vmx/sgx.c +79 -1
93 lines not on this page · raw message
arch/x86/kvm/vmx/sgx.h +13 -0
@@ -4,6 +4,9 @@#include <linux/kvm_host.h> +#include "capabilities.h" +#include "vmx_ops.h" +#ifdef CONFIG_X86_SGX_VIRTUALIZATION extern bool __read_mostly enable_sgx;
@@ -11,11 +14,21 @@ int handle_encls(struct kvm_vcpu *vcpu);void setup_default_sgx_lepubkeyhash (void );void vcpu_setup_sgx_lepubkeyhash (struct kvm_vcpu *vcpu); + +void vmx_write_encls_bitmap (struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12);#else #define enable_sgx 0static inline void setup_default_sgx_lepubkeyhash (void ) { }static inline void vcpu_setup_sgx_lepubkeyhash (struct kvm_vcpu *vcpu) { } + +static inline void vmx_write_encls_bitmap (struct kvm_vcpu *vcpu, +struct vmcs12 *vmcs12) +{ +/* Nothing to do if hardware doesn't support SGX */ +if (cpu_has_vmx_encls_vmexit ()) +vmcs_write64 (ENCLS_EXITING_BITMAP ,-1ull ); +}#endif #endif /* __KVM_X86_SGX_H */
arch/x86/kvm/vmx/vmcs12.c +1 -0
@@ -50,6 +50,7 @@ const unsigned short vmcs_field_to_offset_table[] = {FIELD64 (VMREAD_BITMAP , vmread_bitmap),FIELD64 (VMWRITE_BITMAP , vmwrite_bitmap),FIELD64 (XSS_EXIT_BITMAP , xss_exit_bitmap), +FIELD64 (ENCLS_EXITING_BITMAP , encls_exiting_bitmap),FIELD64 (GUEST_PHYSICAL_ADDRESS , guest_physical_address),FIELD64 (VMCS_LINK_POINTER , vmcs_link_pointer), FIELD64(GUEST_IA32_DEBUGCTL, guest_ia32_debugctl),
arch/x86/kvm/vmx/vmcs12.h +3 -1
@@ -69,7 +69,8 @@ struct __packed vmcs12 {u64 vm_function_control;u64 eptp_list_address;u64 pml_address; -u64 padding64[3 ];/* room for future expansion */ +u64 encls_exiting_bitmap; +u64 padding64[2 ];/* room for future expansion */ /* * To allow migration of L1 (complete with its L2 guests) between * machines of different natural widths (32 or 64 bit), we cannot have
@@ -256,6 +257,7 @@ static inline void vmx_check_vmcs12_offsets(void)CHECK_OFFSET (vm_function_control,296 );CHECK_OFFSET (eptp_list_address,304 );CHECK_OFFSET (pml_address,312 ); +CHECK_OFFSET (encls_exiting_bitmap,320 );CHECK_OFFSET (cr0_guest_host_mask,344 );CHECK_OFFSET (cr4_guest_host_mask,352 );CHECK_OFFSET (cr0_read_shadow,360 );
arch/x86/kvm/vmx/vmx.c +36 -2
@@ -2160,6 +2160,9 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) vmx->msr_ia32_feature_control = data;if (msr_info->host_initiated && data ==0 )vmx_leave_nested (vcpu); + +/* SGX may be enabled/disabled by guest's firmware */ +vmx_write_encls_bitmap (vcpu,NULL );break ; case MSR_IA32_SGXLEPUBKEYHASH0 ... MSR_IA32_SGXLEPUBKEYHASH3: if (!msr_info->host_initiated &&
@@ -4317,6 +4320,15 @@ static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx)vmx_adjust_sec_exec_control (vmx, &exec_control, waitpkg,WAITPKG ,ENABLE_USR_WAIT_PAUSE , false); +if (cpu_has_vmx_encls_vmexit () && nested) { +if (guest_cpuid_has (vcpu,X86_FEATURE_SGX )) + vmx->nested.msrs.secondary_ctls_high |= +SECONDARY_EXEC_ENCLS_EXITING ; +else + vmx->nested.msrs.secondary_ctls_high &= + ~SECONDARY_EXEC_ENCLS_EXITING ; + } + vmx->secondary_exec_control = exec_control; }
@@ -4416,8 +4428,7 @@ static void init_vmcs(struct vcpu_vmx *vmx)vmcs_write16 (GUEST_PML_INDEX ,PML_ENTITY_NUM -1 ); } -if (cpu_has_vmx_encls_vmexit ()) -vmcs_write64 (ENCLS_EXITING_BITMAP ,-1ull ); +vmx_write_encls_bitmap (&vmx->vcpu,NULL );if (vmx_pt_mode_is_host_guest ()) {memset (&vmx->pt_desc,0 ,sizeof (vmx->pt_desc));
@@ -7301,6 +7312,22 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)set_cr4_guest_host_mask (vmx); +vmx_write_encls_bitmap (vcpu,NULL ); +if (guest_cpuid_has (vcpu,X86_FEATURE_SGX )) + vmx->msr_ia32_feature_control_valid_bits |=FEAT_CTL_SGX_ENABLED ; +else + vmx->msr_ia32_feature_control_valid_bits &= ~FEAT_CTL_SGX_ENABLED ; +/* +* Only allow guest to write its virtual SGX_LEPUBKEYHASHn MSRs when +* host is writable, otherwise it is meaningless. +*/ +if (guest_cpuid_has (vcpu,X86_FEATURE_SGX_LC )) + vmx->msr_ia32_feature_control_valid_bits |= +FEAT_CTL_SGX_LC_ENABLED ; +else + vmx->msr_ia32_feature_control_valid_bits &= + ~FEAT_CTL_SGX_LC_ENABLED ; +/* Refresh #PF interception to account for MAXPHYADDR changes. */ update_exception_bitmap (vcpu); }
@@ -7321,6 +7348,13 @@ static __init void vmx_set_cpu_caps(void)if (vmx_pt_mode_is_host_guest ())kvm_cpu_cap_check_and_set (X86_FEATURE_INTEL_PT ); +if (!enable_sgx) { +kvm_cpu_cap_clear (X86_FEATURE_SGX ); +kvm_cpu_cap_clear (X86_FEATURE_SGX_LC ); +kvm_cpu_cap_clear (X86_FEATURE_SGX1 ); +kvm_cpu_cap_clear (X86_FEATURE_SGX2 ); + } +if (vmx_umip_emulated ())kvm_cpu_cap_set (X86_FEATURE_UMIP );
Signature · 1 line
2.29.2
Jarkko Sakkinen
maintainerCan you send a new version that applies:
$ git pw series apply 416463 Applying: x86/cpufeatures: Add SGX1 and SGX2 sub-features Applying: x86/sgx: Remove a warn from sgx_free_epc_page() Applying: x86/sgx: Wipe out EREMOVE from sgx_free_epc_page() Applying: x86/sgx: Add SGX_CHILD_PRESENT hardware error code Applying: x86/sgx: Introduce virtual EPC for use by KVM guests Applying: x86/cpu/intel: Allow SGX virtualization without Launch Control support Applying: x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled error: sha1 information is lacking or useless (arch/x86/kernel/cpu/sgx/main.c). error: could not build fake ancestor hint: Use 'git am --show-current-patch=diff' to see the failed patch Patch failed at 0007 x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".
Thanks.
/Jarkko
Quoted 238 lines · Kai Huang
--- Disclaimer ---
These patches were originally written by Sean Christopherson while at Intel. Now that Sean has left Intel, I (Kai) have taken over getting them upstream. This series needs more review before it can be merged. It is being posted publicly and under RFC so Sean and others can review it. Maintainers are safe ignoring it for now.
------------------
Hi all,
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Please help to review this series. Also I'd like to hear what is the proper way to merge this series, since it contains change to both x86/SGX and KVM subsystem. Any feedback is highly appreciated. And please let me know if I forgot to CC anyone, or anyone wants to be removed from CC. Thanks in advance!
This series is based against upstream v5.11-rc3. You can also get the code from upstream branch of kvm-sgx repo on github:
https://github.com/intel/kvm-sgx.git upstream
It also requires Qemu changes to create VM with SGX support. You can find Qemu repo here:
https://github.com/intel/qemu-sgx.git next
Please refer to README.md of above qemu-sgx repo for detail on how to create guest with SGX support. At meantime, for your quick reference you can use below command to create SGX guest:
#qemu-system-x86_64 -smp 4 -m 2G -drive file=<your_vm_image>,if=virtio \ -cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,preallocPlease note that the SGX relevant part is:
-cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,preallocAnd you can change other parameters of your qemu command based on your needs.
=========
Changelog:(Changelog here is for global changes. Please see each patch's changelog for
changes made to specific patch.)
v1->v2:- Refined this cover letter by addressing comments from Dave and Jarkko.
- The original patch which introduced new X86_FEATURE_SGX1/SGX2 were replaced
by 3 new patches from Sean, following Boris and Sean's discussion.
[RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
[RFC PATCH v2 18/26] KVM: x86: Add support for reverse CPUID lookup of scattered features [RFC PATCH v2 19/26] KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
- The original patch 1
x86/sgx: Split out adding EPC page to free list to separate helper
was replaced with 2 new patches from Jarkko
[RFC PATCH v2 02/26] x86/sgx: Remove a warn fromsgx_free_epc_page()
[RFC PATCH v2 03/26] x86/sgx: Wipe out EREMOVE fromsgx_free_epc_page()
addressing Jarkko's comments.
- Moved modifyingsgx_init()to always initializesgx_virt_epc_init()out of
patch
x86/sgx: Introduce virtual EPC for use by KVM guests
to a separate patch:
[RFC PATCH v2 07/26] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
to address Dave's comment that patch ordering can be improved due to before patch "Allow SGX virtualization without Launch Control support", all SGX, including SGX virtualization, is actually disabled when SGX LC is not
present.KVM part patches are not changed comparing to v1 (except changes due to X86_FEATURE_SGX1/2 patches). For changes to each x86 patch, please see changelog in each indudival patch. If no changelog, then no change was made to it.
=========
KVM SGX virtualization Overview- Virtual EPC
SGX enclave memory is special and is reserved specifically for enclave use. In bare-metal SGX enclaves, the kernel allocates enclave pages, copies data into the pages with privileged instructions, then allows the enclave to start. In this scenario, only initialized pages already assigned to an enclave are mapped to userspace.
In virtualized environments, the hypervisor still needs to do the physical enclave page allocation. The guest kernel is responsible for the data copying (among other things). This means that the job of starting an enclave is now split between hypervisor and guest.
This series introduces a new misc device:
/dev/sgx_virt_epc. This device allows the host to map *uninitialized* enclave memory into userspace, which can then be passed into a guest.While it might be *possible* to start a host-side enclave with
/dev/sgx_enclaveand pass its memory into a guest, it would be wasteful and convoluted.Implement the *raw* EPC allocation in the x86 core-SGX subsystem via
/dev/sgx_virt_epcrather than in KVM. Doing so has two major advantages:- Does not require changes to KVM's uAPI, e.g. EPC gets handled as just another memory backend for guests.
- EPC management is wholly contained in the SGX subsystem, e.g. SGX does not have to export any symbols, changes to reclaim flows don't need to be routed through KVM, SGX's dirty laundry doesn't have to get aired out for the world to see, and so on and so forth.
The virtual EPC pages allocated to guests are currently not reclaimable. Reclaiming EPC page used by enclave requires a special reclaim mechanism separate from normal page reclaim, and that mechanism is not supported for virutal EPC pages. Due to the complications of handling reclaim conflicts between guest and host, reclaiming virtual EPC pages is significantly more complex than basic support for SGX virtualization.
- Support SGX virtualization without SGX Flexible Launch Control
SGX hardware supports two "launch control" modes to limit which enclaves can run. In the "locked" mode, the hardware prevents enclaves from running unless they are blessed by a third party. In the unlocked mode, the kernel is in full control of which enclaves can run. The bare-metal SGX code refuses to launch enclaves unless it is in the unlocked mode.
This
sgx_virt_epcdriver does not have such a restriction. This allows guests which are OK with the locked mode to use SGX, even if the host kernel refuses to.- Support exposing SGX2
Due to the same reason above, SGX2 feature detection is added to core SGX code to allow KVM to expose SGX2 to guest, even currently SGX driver doesn't support SGX2, because SGX2 can work just fine in guest w/o any interaction to host SGX driver.
- Restricit SGX guest access to provisioning key
To grant guest being able to fully use SGX, guest needs to be able to access provisioning key. The provisioning key is sensitive, and accessing to it should be restricted. In bare-metal driver, allowing enclave to access provisioning key is restricted by being able to open
/dev/sgx_provision.Add a new
KVM_CAP_SGX_ATTRIBUTEto KVM uAPI to extend above mechanism to KVM guests as well. When userspace hypervisor creates a new VM, the new cap is only added to VM when userspace hypervisior is able to open/dev/sgx_provision, following the same role as in bare-metal driver. KVM then traps ECREATE from guest, and only allows ECREATE with provisioning key bit to run when guest supportsKVM_CAP_SGX_ATTRIBUTE.Kai Huang (2):
x86/sgx: Initialize virtual EPC driver even when SGX driver is
disabled
x86/sgx: Add helper to updateSGX_LEPUBKEYHASHnMSRsSean Christopherson (22):
x86/cpufeatures: Add SGX1 and SGX2 sub-features
x86/sgx: AddSGX_CHILD_PRESENThardware error code x86/sgx: Introduce virtual EPC for use by KVM guests x86/cpu/intel: Allow SGX virtualization without Launch Control support x86/sgx: Expose SGX architectural definitions to the kernel x86/sgx: Move ENCLS leaf definitions tosgx_arch.hx86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT) x86/sgx: Addencls_faulted()helper
x86/sgx: Add helpers to expose ECREATE and EINIT to KVM x86/sgx: Move provisioning device creation out of SGX driver
KVM: VMX: Convert vcpu_vmx.exit_reason to a union
KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)
KVM: x86: Define new #PF SGX error code bit
KVM: x86: Add support for reverse CPUID lookup of scattered features
KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
KVM: VMX: Add basic handling of VM-Exit from SGX enclave
KVM: VMX: Frame in ENCLS handler for SGX virtualization
KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions
KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs
KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)
KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC
KVM: x86: Add capability to grant VM access to privileged SGX
attribute[email protected] (2):
x86/sgx: Remove a warn fromsgx_free_epc_page()x86/sgx: Wipe out EREMOVE fromsgx_free_epc_page()Documentation/virt/kvm/api.rst | 23 + arch/x86/Kconfig | 12 + arch/x86/include/asm/cpufeatures.h | 2 + arch/x86/include/asm/kvm_host.h | 5 + arch/x86/include/asm/sgx.h | 19 + .../cpu/sgx/arch.h => include/asm/sgx_arch.h} | 20 + arch/x86/include/asm/vmx.h | 1 + arch/x86/include/uapi/asm/vmx.h | 1 + arch/x86/kernel/cpu/cpuid-deps.c | 3 + arch/x86/kernel/cpu/feat_ctl.c | 63 ++- arch/x86/kernel/cpu/scattered.c | 2 + arch/x86/kernel/cpu/sgx/Makefile | 1 + arch/x86/kernel/cpu/sgx/driver.c | 17 - arch/x86/kernel/cpu/sgx/encl.c | 15 +- arch/x86/kernel/cpu/sgx/encls.h | 29 +- arch/x86/kernel/cpu/sgx/ioctl.c | 23 +- arch/x86/kernel/cpu/sgx/main.c | 67 ++- arch/x86/kernel/cpu/sgx/sgx.h | 4 +- arch/x86/kernel/cpu/sgx/virt.c | 316 ++++++++++++ arch/x86/kernel/cpu/sgx/virt.h | 14 + arch/x86/kvm/Makefile | 2 + arch/x86/kvm/cpuid.c | 89 +++- arch/x86/kvm/cpuid.h | 50 +- arch/x86/kvm/vmx/nested.c | 70 ++- arch/x86/kvm/vmx/nested.h | 5 + arch/x86/kvm/vmx/sgx.c | 462 ++++++++++++++++++ arch/x86/kvm/vmx/sgx.h | 34 ++ arch/x86/kvm/vmx/vmcs12.c | 1 + arch/x86/kvm/vmx/vmcs12.h | 4 +- arch/x86/kvm/vmx/vmx.c | 171 +++++-- arch/x86/kvm/vmx/vmx.h | 27 +- arch/x86/kvm/x86.c | 24 + include/uapi/linux/kvm.h | 1 + tools/testing/selftests/sgx/defines.h | 2 +- 34 files changed, 1432 insertions(+), 147 deletions(-) create mode 100644 arch/x86/include/asm/sgx.h rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx_arch.h} (96%) create mode 100644 arch/x86/kernel/cpu/sgx/virt.c create mode 100644 arch/x86/kernel/cpu/sgx/virt.h create mode 100644 arch/x86/kvm/vmx/sgx.c create mode 100644 arch/x86/kvm/vmx/sgx.hSignature · 1 line
2.29.2
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 02/26] x86/sgx: Remove a warn from sgx_free_epc_page()
In reply to Kai HuangQuoted 1 line · Kai Huang
From: "[email protected]" <[email protected]>
~~~~~~~~~~~~~~~~~~~
Jarkko SakkinenQuoted 6 lines
Remove
SGX_EPC_PAGE_RECLAIMER_TRACKEDcheck and warning. This cannot happen, as enclave pages are freed only at the time whenencl->refcounttriggers, i.e. when both VFS and the page reclaimer have given up on their references.Signed-off-by [email protected] <[email protected]>
~~~~~~~~~~~~~~~~~
Jarkko SakkinenQuoted 1 line
Signed-off-by Kai Huang <[email protected]>
Sorry about this. I was reconfiguring my environment (or actually was moving it to another machine), and forgot to set user.name. I'll send you in private replacements with a legit name.
/Jarkko
Borislav Petkov
maintainerRe: [RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
In reply to Kai Huangarch/x86/kernel/cpu/feat_ctl.c static void init_vmx_capabilities(struct cpuinfo_x86 *c) · line 99
Quoted 9 lines · Kai Huang
arch/x86/kernel/cpu/feat_ctl.c +0
-1@@ -96,7 +96,6 @@ static void init_vmx_capabilities(struct cpuinfo_x86 *c)static void clear_sgx_caps (void ) {setup_clear_cpu_cap (X86_FEATURE_SGX ); -setup_clear_cpu_cap (X86_FEATURE_SGX_LC );
Why is that line being removed here?
Shouldn't this add SGX1 and SGX2 here instead as this function is supposed to, well, *clear* sgx caps on feat_ctl setup failures or "nosgx" cmdline?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerRe: [RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
Quoted 13 lines · Borislav Petkov
arch/x86/kernel/cpu/feat_ctl.c static void init_vmx_capabilities(struct cpuinfo_x86 *c) · line 99
Quoted 9 lines · Kai Huang
arch/x86/kernel/cpu/feat_ctl.c +0
-1@@ -96,7 +96,6 @@ static void init_vmx_capabilities(struct cpuinfo_x86 *c)static void clear_sgx_caps (void ) {setup_clear_cpu_cap (X86_FEATURE_SGX ); -setup_clear_cpu_cap (X86_FEATURE_SGX_LC );Why is that line being removed here?
Quoted 3 lines
Shouldn't this add SGX1 and SGX2 here instead as this function is supposed to, well, *clear* sgx caps on
feat_ctlsetup failures or "nosgx" cmdline?
Doesn't adding making the SGX sub-features depend on X86_FEATURE_SGX have the same net effect? Or am I misreading do_clear_cpu_cap()?
Though if we use the cpuid_deps table, I'd vote to get rid of clear_sgx_caps() and call setup_clear_cpu_cap(X86_FEATURE_SGX) directly. And probably change the existing SGX_LC behavior and drop clear_sgx_caps() in a separate patch instead of squeezing it into this one.
Re: [RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
Quoted 14 lines · Sean Christopherson
Quoted 13 lines · Borislav Petkov
arch/x86/kernel/cpu/feat_ctl.c static void init_vmx_capabilities(struct cpuinfo_x86 *c) · line 99
Quoted 9 lines · Kai Huang
arch/x86/kernel/cpu/feat_ctl.c +0
-1@@ -96,7 +96,6 @@ static void init_vmx_capabilities(struct cpuinfo_x86 *c) static void clear_sgx_caps(void) { setup_clear_cpu_cap(X86_FEATURE_SGX); -setup_clear_cpu_cap (X86_FEATURE_SGX_LC );Why is that line being removed here?
Quoted 6 lines
Quoted 3 lines
Shouldn't this add SGX1 and SGX2 here instead as this function is supposed to, well, *clear* sgx caps on
feat_ctlsetup failures or "nosgx" cmdline?Doesn't adding making the SGX sub-features depend on
X86_FEATURE_SGXhave the same net effect? Or am I misreadingdo_clear_cpu_cap()?
On my testing machine with SGX, SGX_LC, and SGX1. I just double tested that clearing X86_FEATURE_SGX also clears SGX1 and SGX_LC bits.
Quoted 4 lines
Though if we use the
cpuid_depstable, I'd vote to get rid ofclear_sgx_caps()and callsetup_clear_cpu_cap(X86_FEATURE_SGX)directly.
Yes I can do that.
Quoted 3 lines
And probably change the
existingSGX_LCbehavior and dropclear_sgx_caps()in a separate patch instead of squeezing it into this one.
To double confirm, you want:
1) This patch to introduce SGX1, SGX2, and also put SGX1 and SGX2 in to CPUID dependency table;
2) A separate patch to put SGX_LC to CPUID dependency table too, and also git rid of clear_sgx_caps()
Correct?
Btw, in your original patch, both SGX1 and SGX2 depends on SGX, but I changed to make SGX2 depend on SGX1. However I still make SGX_LC depend on SGX, but not SGX1. Does this make sense to you?
Re: [RFC PATCH v2 02/26] x86/sgx: Remove a warn from sgx_free_epc_page()
In reply to Jarkko SakkinenQuoted 11 lines · Jarkko Sakkinen
Quoted 1 line · Kai Huang
From: "[email protected]" <[email protected]>
~~~~~~~~~~~~~~~~~~~
Jarkko SakkinenQuoted 6 lines
Remove
SGX_EPC_PAGE_RECLAIMER_TRACKEDcheck and warning. This cannot happen, as enclave pages are freed only at the time whenencl->refcounttriggers, i.e. when both VFS and the page reclaimer have given up on their references.Signed-off-by [email protected] <[email protected]>
Quoted 8 lines
~~~~~~~~~~~~~~~~~
Jarkko SakkinenQuoted 1 line
Signed-off-by Kai Huang <[email protected]>Sorry about this. I was reconfiguring my environment (or actually was moving it to another machine), and forgot to set user.name. I'll send you in private replacements with a legit name.
Sure. No problem. Thanks.
Quoted 17 lines · Jarkko Sakkinen
Can you send a new version that applies:
$ git pw series apply 416463 Applying: x86/cpufeatures: Add SGX1 and SGX2 sub-features Applying: x86/sgx: Remove a warn from sgx_free_epc_page() Applying: x86/sgx: Wipe out EREMOVE from sgx_free_epc_page() Applying: x86/sgx: Add SGX_CHILD_PRESENT hardware error code Applying: x86/sgx: Introduce virtual EPC for use by KVM guests Applying: x86/cpu/intel: Allow SGX virtualization without Launch Control support Applying: x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled error: sha1 information is lacking or useless (arch/x86/kernel/cpu/sgx/main.c). error: could not build fake ancestor hint: Use 'git am --show-current-patch=diff' to see the failed patch Patch failed at 0007 x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".
Could you let me know which branch should I rebase to? It appears your linux-sgx tree next branch?
And I don't think it is good to resend for this series at this moment, due to: 1) people have already started to comment; 2) The two patches from you don't have your name on your SoB, so has to fix anyway. It would be more appropriate to rebase to your tree in next version, if required. And I can prepare a separate branch for you (after resolving the merge conflict with sgx_init() in your tree) at the same time.
And Boris, Paolo, Sean, Jarkko, and other maintainers,
I'd like to take this chance to ask: when this series is ready to be merged, what is the properly way to merge? This series has x86 non-sgx (cpufeature, feat_ctl) and sgx changes, and it obviously has KVM changes too. So instance, who should be the one to take this series? And which tree and branch should I rebase to in next version?
Thanks for all your feedback.
Quoted 244 lines
Thanks.
/Jarkko
Quoted 238 lines · Kai Huang
--- Disclaimer ---
These patches were originally written by Sean Christopherson while at Intel. Now that Sean has left Intel, I (Kai) have taken over getting them upstream. This series needs more review before it can be merged. It is being posted publicly and under RFC so Sean and others can review it. Maintainers are safe ignoring it for now.
------------------
Hi all,
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Please help to review this series. Also I'd like to hear what is the proper way to merge this series, since it contains change to both x86/SGX and KVM subsystem. Any feedback is highly appreciated. And please let me know if I forgot to CC anyone, or anyone wants to be removed from CC. Thanks in advance!
This series is based against upstream v5.11-rc3. You can also get the code from upstream branch of kvm-sgx repo on github:
https://github.com/intel/kvm-sgx.git upstream
It also requires Qemu changes to create VM with SGX support. You can find Qemu repo here:
https://github.com/intel/qemu-sgx.git next
Please refer to README.md of above qemu-sgx repo for detail on how to create guest with SGX support. At meantime, for your quick reference you can use below command to create SGX guest:
#qemu-system-x86_64 -smp 4 -m 2G -drive file=<your_vm_image>,if=virtio \ -cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,preallocPlease note that the SGX relevant part is:
-cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,preallocAnd you can change other parameters of your qemu command based on your needs.
=========
Changelog:(Changelog here is for global changes. Please see each patch's changelog for changes made to specific patch.)
v1->v2:- Refined this cover letter by addressing comments from Dave and Jarkko.
- The original patch which introduced new X86_FEATURE_SGX1/SGX2 were replaced
by 3 new patches from Sean, following Boris and Sean's discussion.
[RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
[RFC PATCH v2 18/26] KVM: x86: Add support for reverse CPUID lookup of scattered features [RFC PATCH v2 19/26] KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
- The original patch 1
x86/sgx: Split out adding EPC page to free list to separate helper
was replaced with 2 new patches from Jarkko
[RFC PATCH v2 02/26] x86/sgx: Remove a warn fromsgx_free_epc_page()
[RFC PATCH v2 03/26] x86/sgx: Wipe out EREMOVE fromsgx_free_epc_page()addressing Jarkko's comments.
- Moved modifyingsgx_init()to always initializesgx_virt_epc_init()out of
patch
x86/sgx: Introduce virtual EPC for use by KVM guests
to a separate patch:
[RFC PATCH v2 07/26] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled to address Dave's comment that patch ordering can be improved due to before patch "Allow SGX virtualization without Launch Control support", all SGX, including SGX virtualization, is actually disabled when SGX LC is not
present.KVM part patches are not changed comparing to v1 (except changes due to X86_FEATURE_SGX1/2 patches). For changes to each x86 patch, please see changelog in each indudival patch. If no changelog, then no change was made to it.
=========
KVM SGX virtualization Overview- Virtual EPC
SGX enclave memory is special and is reserved specifically for enclave use. In bare-metal SGX enclaves, the kernel allocates enclave pages, copies data into the pages with privileged instructions, then allows the enclave to start. In this scenario, only initialized pages already assigned to an enclave are mapped to userspace.
In virtualized environments, the hypervisor still needs to do the physical enclave page allocation. The guest kernel is responsible for the data copying (among other things). This means that the job of starting an enclave is now split between hypervisor and guest.
This series introduces a new misc device:
/dev/sgx_virt_epc. This device allows the host to map *uninitialized* enclave memory into userspace, which can then be passed into a guest.While it might be *possible* to start a host-side enclave with
/dev/sgx_enclaveand pass its memory into a guest, it would be wasteful and convoluted.Implement the *raw* EPC allocation in the x86 core-SGX subsystem via
/dev/sgx_virt_epcrather than in KVM. Doing so has two major advantages:- Does not require changes to KVM's uAPI, e.g. EPC gets handled as just another memory backend for guests.
- EPC management is wholly contained in the SGX subsystem, e.g. SGX does not have to export any symbols, changes to reclaim flows don't need to be routed through KVM, SGX's dirty laundry doesn't have to get aired out for the world to see, and so on and so forth.
The virtual EPC pages allocated to guests are currently not reclaimable. Reclaiming EPC page used by enclave requires a special reclaim mechanism separate from normal page reclaim, and that mechanism is not supported for virutal EPC pages. Due to the complications of handling reclaim conflicts between guest and host, reclaiming virtual EPC pages is significantly more complex than basic support for SGX virtualization.
- Support SGX virtualization without SGX Flexible Launch Control
SGX hardware supports two "launch control" modes to limit which enclaves can run. In the "locked" mode, the hardware prevents enclaves from running unless they are blessed by a third party. In the unlocked mode, the kernel is in full control of which enclaves can run. The bare-metal SGX code refuses to launch enclaves unless it is in the unlocked mode.
This
sgx_virt_epcdriver does not have such a restriction. This allows guests which are OK with the locked mode to use SGX, even if the host kernel refuses to.- Support exposing SGX2
Due to the same reason above, SGX2 feature detection is added to core SGX code to allow KVM to expose SGX2 to guest, even currently SGX driver doesn't support SGX2, because SGX2 can work just fine in guest w/o any interaction to host SGX driver.
- Restricit SGX guest access to provisioning key
To grant guest being able to fully use SGX, guest needs to be able to access provisioning key. The provisioning key is sensitive, and accessing to it should be restricted. In bare-metal driver, allowing enclave to access provisioning key is restricted by being able to open
/dev/sgx_provision.Add a new
KVM_CAP_SGX_ATTRIBUTEto KVM uAPI to extend above mechanism to KVM guests as well. When userspace hypervisor creates a new VM, the new cap is only added to VM when userspace hypervisior is able to open/dev/sgx_provision, following the same role as in bare-metal driver. KVM then traps ECREATE from guest, and only allows ECREATE with provisioning key bit to run when guest supportsKVM_CAP_SGX_ATTRIBUTE.Kai Huang (2):
x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
x86/sgx: Add helper to updateSGX_LEPUBKEYHASHnMSRsSean Christopherson (22):
x86/cpufeatures: Add SGX1 and SGX2 sub-features
x86/sgx: AddSGX_CHILD_PRESENThardware error code x86/sgx: Introduce virtual EPC for use by KVM guests x86/cpu/intel: Allow SGX virtualization without Launch Control support x86/sgx: Expose SGX architectural definitions to the kernel x86/sgx: Move ENCLS leaf definitions tosgx_arch.hx86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT) x86/sgx: Addencls_faulted()helper
x86/sgx: Add helpers to expose ECREATE and EINIT to KVM x86/sgx: Move provisioning device creation out of SGX driver
KVM: VMX: Convert vcpu_vmx.exit_reason to a union
KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)
KVM: x86: Define new #PF SGX error code bit
KVM: x86: Add support for reverse CPUID lookup of scattered features
KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
KVM: VMX: Add basic handling of VM-Exit from SGX enclave
KVM: VMX: Frame in ENCLS handler for SGX virtualization
KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions
KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs
KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)
KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC
KVM: x86: Add capability to grant VM access to privileged SGX attribute[email protected] (2):
x86/sgx: Remove a warn fromsgx_free_epc_page()x86/sgx: Wipe out EREMOVE fromsgx_free_epc_page()Documentation/virt/kvm/api.rst | 23 + arch/x86/Kconfig | 12 + arch/x86/include/asm/cpufeatures.h | 2 + arch/x86/include/asm/kvm_host.h | 5 + arch/x86/include/asm/sgx.h | 19 + .../cpu/sgx/arch.h => include/asm/sgx_arch.h} | 20 + arch/x86/include/asm/vmx.h | 1 + arch/x86/include/uapi/asm/vmx.h | 1 + arch/x86/kernel/cpu/cpuid-deps.c | 3 + arch/x86/kernel/cpu/feat_ctl.c | 63 ++- arch/x86/kernel/cpu/scattered.c | 2 + arch/x86/kernel/cpu/sgx/Makefile | 1 + arch/x86/kernel/cpu/sgx/driver.c | 17 - arch/x86/kernel/cpu/sgx/encl.c | 15 +- arch/x86/kernel/cpu/sgx/encls.h | 29 +- arch/x86/kernel/cpu/sgx/ioctl.c | 23 +- arch/x86/kernel/cpu/sgx/main.c | 67 ++- arch/x86/kernel/cpu/sgx/sgx.h | 4 +- arch/x86/kernel/cpu/sgx/virt.c | 316 ++++++++++++ arch/x86/kernel/cpu/sgx/virt.h | 14 + arch/x86/kvm/Makefile | 2 + arch/x86/kvm/cpuid.c | 89 +++- arch/x86/kvm/cpuid.h | 50 +- arch/x86/kvm/vmx/nested.c | 70 ++- arch/x86/kvm/vmx/nested.h | 5 + arch/x86/kvm/vmx/sgx.c | 462 ++++++++++++++++++ arch/x86/kvm/vmx/sgx.h | 34 ++ arch/x86/kvm/vmx/vmcs12.c | 1 + arch/x86/kvm/vmx/vmcs12.h | 4 +- arch/x86/kvm/vmx/vmx.c | 171 +++++-- arch/x86/kvm/vmx/vmx.h | 27 +- arch/x86/kvm/x86.c | 24 + include/uapi/linux/kvm.h | 1 + tools/testing/selftests/sgx/defines.h | 2 +- 34 files changed, 1432 insertions(+), 147 deletions(-)create mode 100644
arch/x86/include/asm/sgx.h
renamearch/x86/{kernel/cpu/sgx/arch.h=>include/asm/sgx_arch.h} (96%) create mode 100644arch/x86/kernel/cpu/sgx/virt.c
create mode 100644arch/x86/kernel/cpu/sgx/virt.h
create mode 100644arch/x86/kvm/vmx/sgx.c
create mode 100644arch/x86/kvm/vmx/sgx.hSignature · 1 line
2.29.2
Borislav Petkov
maintainerRe: [RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
In reply to Sean ChristophersonQuoted 2 lines · Sean Christopherson
Doesn't adding making the SGX sub-features depend on
X86_FEATURE_SGXhave the same net effect? Or am I misreadingdo_clear_cpu_cap()?
No, you're correct - I missed that.
Quoted 4 lines
Though if we use the
cpuid_depstable, I'd vote to get rid ofclear_sgx_caps()and callsetup_clear_cpu_cap(X86_FEATURE_SGX)directly. And probably change the existingSGX_LCbehavior and dropclear_sgx_caps()in a separate patch instead of squeezing it into this one.
Yah, sounds good.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteJarkko Sakkinen
maintainerRe: [RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
In reply to Kai HuangQuoted 6 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add SGX1 and SGX2 feature flags, via CPUID.0x12.0x0.EAX, as scattered features. As part of virtualizing SGX, KVM will expose the SGX CPUID leafs to its guest, and to do so correctly needs to query hardware and kernel support for SGX1 and SGX2.
This commit message is missing reasoning behind scattered vs. own word.
Please just document the reasoning, that's all.
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 05/26] x86/sgx: Introduce virtual EPC for use by KVM guests
In reply to Kai HuangQuoted 6 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add a misc device
/dev/sgx_virt_epcto allow userspace to allocate "raw" EPC without an associated enclave. The intended and only known use case for raw EPC allocation is to expose EPC to a KVM guest, hence thevirt_epcmoniker, virt.{c,h} files andX86_SGX_VIRTUALIZATIONKconfig.
Is /dev/sgx_virt_epc something only usable for KVM, or is there any thinkable use outside of the KVM context?
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 08/26] x86/sgx: Expose SGX architectural definitions to the kernel
In reply to Kai HuangQuoted 4 lines · Kai Huang
From: Sean Christopherson <[email protected]>
KVM will use many of the architectural constants and structs to virtualize SGX.
"Expose SGX architectural structures, as ..."
Quoted 57 lines
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx_arch.h} | 0 arch/x86/kernel/cpu/sgx/encl.c | 2 +- arch/x86/kernel/cpu/sgx/sgx.h | 2 +- tools/testing/selftests/sgx/defines.h | 2 +- 4 files changed, 3 insertions(+), 3 deletions(-) rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx_arch.h} (100%)arch/x86/include/asm/sgx_arch.h renamed from arch/x86/kernel/cpu/sgx/arch.h +0
-0arch/x86/kernel/cpu/sgx/encl.c +1
-1@@ -7,7 +7,7 @@#include <linux/shmem_fs.h> #include <linux/suspend.h> #include <linux/sched/mm.h> -#include "arch.h" +#include <asm/sgx_arch.h> #include "encl.h" #include "encls.h" #include "sgx.h" arch/x86/kernel/cpu/sgx/sgx.h +1
-1@@ -8,7 +8,7 @@#include <linux/rwsem.h> #include <linux/types.h> #include <asm/asm.h> -#include "arch.h" +#include <asm/sgx_arch.h> #undef pr_fmt#define pr_fmt (fmt) "sgx: " fmttools/testing/selftests/sgx/defines.h +1
-1@@ -14,7 +14,7 @@#define __aligned (x) __attribute__((__aligned__(x)))#define __packed __attribute__((packed)) -#include "../../../../arch/x86/kernel/cpu/sgx/arch.h" +#include "../../../../arch/x86/include/asm/sgx_arch.h" #include "../../../../arch/x86/include/asm/enclu.h" #include "../../../../arch/x86/include/uapi/asm/sgx.h" Signature · 1 line
2.29.2
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 04/26] x86/sgx: Add SGX_CHILD_PRESENT hardware error code
In reply to Kai HuangQuoted 28 lines · Kai Huang
From: Sean Christopherson <[email protected]>
SGX virtualization requires to allocate "raw" EPC and use it as "virtual EPC" for SGX guest. Unlike EPC used by SGX driver, virtual EPC doesn't track how EPC pages are used in VM, e.g. (de)construction of enclaves, so it cannot guarantee EREMOVE success, e.g. it doesn't have a priori knowledge of which pages are SECS with non-zero child counts.
Add
SGX_CHILD_PRESENTfor use by SGX virtualization to assert EREMOVE failures are expected, but only due toSGX_CHILD_PRESENT.Signed-off-by Sean Christopherson <[email protected]>Acked-by Jarkko Sakkinen <[email protected]>Signed-off-by Kai Huang <[email protected]>---
v1->v2:- Change title to reflect hardware error code.
--- arch/x86/kernel/cpu/sgx/arch.h | 2 ++ 1 file changed, 2 insertions(+)arch/x86/kernel/cpu/sgx/arch.h +0
-0
arch/x86/kernel/cpu/sgx/arch.h line 29
Quoted 4 lines
arch/x86/kernel/cpu/sgx/arch.h +1
-0@@ -26,12 +26,14 @@* enum sgx_return_code - The return code type for ENCLS, ENCLU and ENCLV * %SGX_NOT_TRACKED: Previous ETRACK's shootdown sequence has not * been completed yet. +* %SGX_CHILD_PRESENT Enclave has child pages present in the EPC.
s/Enclave/SECS/
Quoted 14 lines
arch/x86/kernel/cpu/sgx/arch.h +1
-0* %SGX_INVALID_EINITTOKEN: EINITTOKEN is invalid and enclave signer's * public key does not match IA32_SGXLEPUBKEYHASH. * %SGX_UNMASKED_EVENT: An unmasked event, e.g. INTR, was received */ enum sgx_return_code {SGX_NOT_TRACKED =11 , +SGX_CHILD_PRESENT =13 ,SGX_INVALID_EINITTOKEN =16 ,SGX_UNMASKED_EVENT =128 , };Signature · 1 line
2.29.2
/Jarkko
Re: [RFC PATCH v2 09/26] x86/sgx: Move ENCLS leaf definitions to sgx_arch.h
In reply to Kai HuangQuoted 8 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Move the ENCLS leaf definitions to
sgx_arch.hso that they can be used by KVM. And because they're architectural.Signed-off-by Sean Christopherson <[email protected]>Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>
/Jarkko
Quoted 61 lines
--- arch/x86/include/asm/sgx_arch.h | 15 +++++++++++++++ arch/x86/kernel/cpu/sgx/encls.h | 15 --------------- 2 files changed, 15 insertions(+), 15 deletions(-)arch/x86/include/asm/sgx_arch.h +15
-0@@ -22,6 +22,21 @@/* The bitmask for the EPC section type. */ #define SGX_CPUID_EPC_MASK GENMASK(3, 0) +enum sgx_encls_function { +ECREATE =0x00 , +EADD =0x01 , +EINIT =0x02 , +EREMOVE =0x03 , +EDGBRD =0x04 , +EDGBWR =0x05 , +EEXTEND =0x06 , +ELDU =0x08 , +EBLOCK =0x09 , +EPA =0x0A , +EWB =0x0B , +ETRACK =0x0C , +}; +/** * enum sgx_return_code - The return code type for ENCLS, ENCLU and ENCLV * %SGX_NOT_TRACKED: Previous ETRACK's shootdown sequence has not arch/x86/kernel/cpu/sgx/encls.h +0
-15@@ -11,21 +11,6 @@#include <asm/traps.h> #include "sgx.h" -enum sgx_encls_function { -ECREATE =0x00 , -EADD =0x01 , -EINIT =0x02 , -EREMOVE =0x03 , -EDGBRD =0x04 , -EDGBWR =0x05 , -EEXTEND =0x06 , -ELDU =0x08 , -EBLOCK =0x09 , -EPA =0x0A , -EWB =0x0B , -ETRACK =0x0C , -}; -/** * ENCLS_FAULT_FLAG - flag signifying an ENCLS return code is a trapnr * Signature · 1 line
2.29.2
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 11/26] x86/sgx: Add encls_faulted() helper
In reply to Kai HuangQuoted 18 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add a helper to extract the fault indicator from an encoded ENCLS return value. SGX virtualization will also need to detect ENCLS faults.
Signed-off-by Sean Christopherson <[email protected]>Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/encls.h | 14 +++++++++++++- arch/x86/kernel/cpu/sgx/ioctl.c | 2 +- 2 files changed, 14 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/sgx/encls.h +0
-0
arch/x86/kernel/cpu/sgx/encls.h line 48
Quoted 9 lines
arch/x86/kernel/cpu/sgx/encls.h +6
-0@@ -40,6 +40,18 @@ }while (0 ); \ } +/* +* encls_faulted() - Check if an ENCLS leaf faulted given an error code +* @ret the return value of an ENCLS leaf function call +* +* Return: +* %true if @ret indicates a fault, %false otherwise
Follow here the style of commenting as in ioctl.c, for the return value. It has optimal readability both as text, and also when converted to HTML. See sgx_ioc_enclave_add_pages() for an example.
Quoted 35 lines
arch/x86/kernel/cpu/sgx/encls.h +7
-1+*/ +static inline bool encls_faulted (int ret) +{ +return ret &ENCLS_FAULT_FLAG ; +} +/** * encls_failed() - Check if an ENCLS function failed * @ret: the return value of an ENCLS function call @@ -50,7 +62,7 @@*/ static inline bool encls_failed (int ret) { -if (ret &ENCLS_FAULT_FLAG ) +if (encls_faulted (ret))return ENCLS_TRAPNR (ret) !=X86_TRAP_PF ;return !!ret;arch/x86/kernel/cpu/sgx/ioctl.c +1
-1@@ -568,7 +568,7 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct, } } -if (ret &ENCLS_FAULT_FLAG ) { +if (encls_faulted (ret)) {if (encls_failed (ret))ENCLS_WARN (ret,"EINIT" );Signature · 1 line
2.29.2
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Kai HuangQuoted 40 lines · Kai Huang
Add a helper to update
SGX_LEPUBKEYHASHnMSRs. SGX virtualization also needs to update those MSRs based on guest's "virtual"SGX_LEPUBKEYHASHnbefore EINIT from guest.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/ioctl.c | 5 ++--- arch/x86/kernel/cpu/sgx/main.c | 8 ++++++++ arch/x86/kernel/cpu/sgx/sgx.h | 2 ++ 3 files changed, 12 insertions(+), 3 deletions(-)arch/x86/kernel/cpu/sgx/ioctl.c +2
-3@@ -495,7 +495,7 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct, void *token) {u64 mrsigner[4 ]; -int i, j, k; +int i, j;void *addr;int ret;@@ -544,8 +544,7 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,preempt_disable (); -for (k =0 ; k <4 ; k++) -wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 + k, mrsigner[k]); +sgx_update_lepubkeyhash (mrsigner); ret =__einit (sigstruct, token, addr);arch/x86/kernel/cpu/sgx/main.c +0
-0
arch/x86/kernel/cpu/sgx/main.c static bool __init sgx_page_cache_init(void) · line 706
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +7
-0@@ -697,6 +697,14 @@ static bool __init sgx_page_cache_init(void)return true; } +void sgx_update_lepubkeyhash (u64 *lepubkeyhash) +{ +int i; + +for (i =0 ; i <4 ; i++) +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 + i, lepubkeyhash[i]); +}
Missing kdoc.
Quoted 19 lines
arch/x86/kernel/cpu/sgx/main.c +1
-0+static void __init sgx_init (void ) {int ret;arch/x86/kernel/cpu/sgx/sgx.h +2
-0@@ -83,4 +83,6 @@ void sgx_mark_page_reclaimable(struct sgx_epc_page *page);int sgx_unmark_page_reclaimable (struct sgx_epc_page *page);struct sgx_epc_page *sgx_alloc_epc_page (void *owner,bool reclaim); +void sgx_update_lepubkeyhash (u64 *lepubkeyhash); +#endif /* _X86_SGX_H */ Signature · 1 line
2.29.2
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 13/26] x86/sgx: Add helpers to expose ECREATE and EINIT to KVM
In reply to Kai HuangQuoted 6 lines · Kai Huang
From: Sean Christopherson <[email protected]>
The bare-metal kernel must intercept ECREATE to be able to impose policies on guests. When it does this, the bare-metal kernel runs ECREATE against the userspace mapping of the virtualized EPC.
Quoted 8 lines
Provide wrappers around
__ecreate()and__einit()to hide the ugliness of overloading the ENCLS return value to encode multiple error formats in a single int. KVM will trap-and-execute ECREATE and EINIT as part of SGX virtualization, and on an exception, KVM needs the trapnr so that it can inject the correct fault into the guest.Signed-off-by Sean Christopherson <[email protected]>[Kai: Use sgx_update_lepubkeyhash() to update pubkey hash MSRs.]
If you have to go to this, please instead use co-developed-by.
Quoted 72 lines
Signed-off-by Kai Huang <[email protected]>---
v1->v2:- Refined commit msg based on Dave's comment.
- Added comment to explain why to use__uaccess_xxx().--- arch/x86/include/asm/sgx.h | 16 +++++++++ arch/x86/kernel/cpu/sgx/virt.c | 61 ++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+) create mode 100644 arch/x86/include/asm/sgx.harch/x86/include/asm/sgx.h new file +16
-0@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_SGX_H +#define _ASM_X86_SGX_H + +#include <linux/types.h> + +#ifdef CONFIG_X86_SGX_VIRTUALIZATION +struct sgx_pageinfo ; + +int sgx_virt_ecreate (struct sgx_pageinfo *pageinfo,void __user *secs, +int *trapnr); +int sgx_virt_einit (void __user *sigstruct,void __user *token, +void __user *secs,u64 *lepubkeyhash,int *trapnr); +#endif + +#endif /* _ASM_X86_SGX_H */ arch/x86/kernel/cpu/sgx/virt.c +30
-0@@ -253,3 +253,64 @@ int __init sgx_virt_epc_init(void)return misc_register (&sgx_virt_epc_dev); } + +int sgx_virt_ecreate (struct sgx_pageinfo *pageinfo,void __user *secs, +int *trapnr) +{ +int ret; + +/* +* @secs is userspace address, and it's not guaranteed @secs points at +* an actual EPC page. It's also possible to generate a kernel mapping +* to physical EPC page by resolving PFN but using __uaccess_xx() is +* simpler. +*/ +__uaccess_begin (); + ret =__ecreate (pageinfo, (void *)secs); +__uaccess_end (); + +if (encls_faulted (ret)) { + *trapnr =ENCLS_TRAPNR (ret); +return -EFAULT ; + } + +/* ECREATE doesn't return an error code, it faults or succeeds. */ +WARN_ON_ONCE (ret); +return 0 ; +} +EXPORT_SYMBOL_GPL (sgx_virt_ecreate); + +static int __sgx_virt_einit (void __user *sigstruct,void __user *token, +void __user *secs) +{
arch/x86/kernel/cpu/sgx/virt.c int __init sgx_virt_epc_init(void) · line 295
Quoted 10 lines
arch/x86/kernel/cpu/sgx/virt.c +10
-0+int ret; + +__uaccess_begin (); + ret =__einit ((void *)sigstruct, (void *)token, (void *)secs); +__uaccess_end (); +return ret; +} + +int sgx_virt_einit (void __user *sigstruct,void __user *token, +void __user *secs,u64 *lepubkeyhash,int *trapnr)
kdoc missing.
Quoted 25 lines
arch/x86/kernel/cpu/sgx/virt.c +21
-0+{ +int ret; + +if (!boot_cpu_has (X86_FEATURE_SGX_LC )) { + ret =__sgx_virt_einit (sigstruct, token, secs); + }else { +preempt_disable (); + +sgx_update_lepubkeyhash (lepubkeyhash); + + ret =__sgx_virt_einit (sigstruct, token, secs); +preempt_enable (); + } + +if (encls_faulted (ret)) { + *trapnr =ENCLS_TRAPNR (ret); +return -EFAULT ; + } +return ret; +} +EXPORT_SYMBOL_GPL (sgx_virt_einit);Signature · 1 line
2.29.2
/Jarkko
Re: [RFC PATCH v2 10/26] x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT)
In reply to Kai HuangQuoted 9 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Define the ENCLS leafs that are available with SGX2, also referred to as Enclave Dynamic Memory Management (EDMM). The leafs will be used by KVM to conditionally expose SGX2 capabilities to guests.
Signed-off-by Sean Christopherson <[email protected]>Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>
/Jarkko
Quoted 22 lines
--- arch/x86/include/asm/sgx_arch.h | 3 +++ 1 file changed, 3 insertions(+)arch/x86/include/asm/sgx_arch.h +3
-0@@ -35,6 +35,9 @@ enum sgx_encls_function {EPA =0x0A ,EWB =0x0B ,ETRACK =0x0C , +EAUG =0x0D , +EMODPR =0x0E , + EMODT = 0x0F, };/** Signature · 1 line
2.29.2
Re: [RFC PATCH v2 16/26] KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)
In reply to Kai HuangQuoted 13 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Export the
gva_to_gpa()helpers for use by SGX virtualization when executing ENCLS[ECREATE] and ENCLS[EINIT] on behalf of the guest. To execute ECREATE and EINIT, KVM must obtain the GPA of the target Secure Enclave Control Structure (SECS) in order to get its corresponding HVA.Because the SECS must reside in the Enclave Page Cache (EPC), copying the SECS's data to a host-controlled buffer via existing exported helpers is not a viable option as the EPC is not readable or writable by the kernel.
Quoted 6 lines
SGX virtualization will also use
gva_to_gpa()to obtain HVAs for non-EPC pages in order to pass user pointers directly to ECREATE and EINIT, which avoids having to copy pages worth of data into the kernel.Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Kai Huang <[email protected]>
/Jarkko
Quoted 28 lines
--- arch/x86/kvm/x86.c | 2 ++ 1 file changed, 2 insertions(+)arch/x86/kvm/x86.c +2
-0@@ -5891,6 +5891,7 @@ gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva,u32 access = (kvm_x86_ops.get_cpl (vcpu) ==3 ) ?PFERR_USER_MASK :0 ;return vcpu->arch.walk_mmu->gva_to_gpa (vcpu, gva, access, exception); } +EXPORT_SYMBOL_GPL (kvm_mmu_gva_to_gpa_read);gpa_t kvm_mmu_gva_to_gpa_fetch (struct kvm_vcpu *vcpu,gva_t gva,struct x86_exception *exception)@@ -5907,6 +5908,7 @@ gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, access |=PFERR_WRITE_MASK ;return vcpu->arch.walk_mmu->gva_to_gpa (vcpu, gva, access, exception); } +EXPORT_SYMBOL_GPL (kvm_mmu_gva_to_gpa_write);/* uses this to access any guest's mapped memory without checking CPL */ gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva,Signature · 1 line
2.29.2
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 15/26] KVM: VMX: Convert vcpu_vmx.exit_reason to a union
In reply to Kai HuangQuoted 35 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Convert vcpu_vmx.exit_reason from a u32 to a union (of size u32). The full
VM_EXIT_REASONfield is comprised of a 16-bit basic exit reason in bits 15:0, and single-bit modifiers in bits 31:16.Historically, KVM has only had to worry about handling the "failed VM-Entry" modifier, which could only be set in very specific flows and required dedicated handling. I.e. manually stripping the
FAILED_VMENTRYbit was a somewhat viable approach. But even with only a single bit to worry about, KVM has had several bugs related to comparing a basic exit reason against the full exit reason store invcpu_vmx.Upcoming Intel features, e.g. SGX, will add new modifier bits that can be set on more or less any VM-Exit, as opposed to the significantly more restricted
FAILED_VMENTRY, i.e. correctly handling everything in one-off flows isn't scalable. Tracking exit reason in a union forces code to explicitly choose between consuming the full exit reason and the basic exit, and is a convenient way to document and access the modifiers.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kvm/vmx/nested.c | 42 +++++++++++++++--------- arch/x86/kvm/vmx/vmx.c | 68 ++++++++++++++++++++------------------- arch/x86/kvm/vmx/vmx.h | 25 +++++++++++++- 3 files changed, 86 insertions(+), 49 deletions(-)arch/x86/kvm/vmx/nested.c +0
-0
arch/x86/kvm/vmx/nested.c enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, · line 3318
Quoted 9 lines
arch/x86/kvm/vmx/nested.c +5
-1@@ -3311,7 +3311,11 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);enum vm_entry_failure_code entry_failure_code;bool evaluate_pending_interrupts; -u32 exit_reason, failed_index; +u32 failed_index; +union vmx_exit_reason exit_reason = { + .basic =-1 , + .failed_vmentry =1 , + };
Instead, put this declaration to the correct place, following the reverse christmas tree ordering:
union vmx_exit_reason exit_reason = {};And after declarations:
exit_reason.basic = -1;
exit_reason.failed_vmentry = 1;More pleasing for the eye.
/Jarkko
Sean Christopherson
maintainerRe: [RFC PATCH v2 15/26] KVM: VMX: Convert vcpu_vmx.exit_reason to a union
Quoted 49 lines · Jarkko Sakkinen
Quoted 35 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Convert vcpu_vmx.exit_reason from a u32 to a union (of size u32). The full
VM_EXIT_REASONfield is comprised of a 16-bit basic exit reason in bits 15:0, and single-bit modifiers in bits 31:16.Historically, KVM has only had to worry about handling the "failed VM-Entry" modifier, which could only be set in very specific flows and required dedicated handling. I.e. manually stripping the
FAILED_VMENTRYbit was a somewhat viable approach. But even with only a single bit to worry about, KVM has had several bugs related to comparing a basic exit reason against the full exit reason store invcpu_vmx.Upcoming Intel features, e.g. SGX, will add new modifier bits that can be set on more or less any VM-Exit, as opposed to the significantly more restricted
FAILED_VMENTRY, i.e. correctly handling everything in one-off flows isn't scalable. Tracking exit reason in a union forces code to explicitly choose between consuming the full exit reason and the basic exit, and is a convenient way to document and access the modifiers.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kvm/vmx/nested.c | 42 +++++++++++++++--------- arch/x86/kvm/vmx/vmx.c | 68 ++++++++++++++++++++------------------- arch/x86/kvm/vmx/vmx.h | 25 +++++++++++++- 3 files changed, 86 insertions(+), 49 deletions(-)arch/x86/kvm/vmx/nested.c +0
-0arch/x86/kvm/vmx/nested.c enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, · line 3318
Quoted 9 lines
arch/x86/kvm/vmx/nested.c +5
-1@@ -3311,7 +3311,11 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);enum vm_entry_failure_code entry_failure_code;bool evaluate_pending_interrupts; -u32 exit_reason, failed_index; +u32 failed_index; +union vmx_exit_reason exit_reason = { + .basic =-1 , + .failed_vmentry =1 , + };Instead, put this declaration to the correct place, following the reverse christmas tree ordering:
Quoted 8 lines
union vmx_exit_reason exit_reason = {};And after declarations:
exit_reason.basic = -1; exit_reason.failed_vmentry = 1;More pleasing for the eye.
I disagree (obviously, since I wrote the patch). Initializing the fields to their respective values is a critical, but subtle, aspect of this code. Making the code stand out via explicit initialization is a good thing, and we really don't want any possibility of code touching exit_reason before it is initialized.
Paolo Bonzini
maintainerQuoted 10 lines · Sean Christopherson
Quoted 4 lines · Kai Huang
I'd like to take this chance to ask: when this series is ready to be merged, what is the properly way to merge? This series has x86 non-sgx (cpufeature,
feat_ctl) and sgx changes, and it obviously has KVM changes too. So instance, who should be the one to take this series? And which tree and branch should I rebase to in next version?The path of least resistance is likely to get acks for the x86 and sgx changes, and let Paolo take it through the KVM tree. The KVM changes are much more likely to have non-trivial conflicts, e.g. making
exit_reasona union touches a ton of code; getting and carrying acked-by for those will be tough sledding.
Yes, the best way is to get a topic branch from Thomas or Borislav.
Paolo
Sean Christopherson
maintainerQuoted 4 lines · Kai Huang
I'd like to take this chance to ask: when this series is ready to be merged, what is the properly way to merge? This series has x86 non-sgx (cpufeature,
feat_ctl) and sgx changes, and it obviously has KVM changes too. So instance, who should be the one to take this series? And which tree and branch should I rebase to in next version?
The path of least resistance is likely to get acks for the x86 and sgx changes, and let Paolo take it through the KVM tree. The KVM changes are much more likely to have non-trivial conflicts, e.g. making exit_reason a union touches a ton of code; getting and carrying acked-by for those will be tough sledding.
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 14/26] x86/sgx: Move provisioning device creation out of SGX driver
In reply to Kai HuangQuoted 149 lines · Kai Huang
From: Sean Christopherson <[email protected]>
And extract
sgx_set_attribute()out ofsgx_ioc_enclave_provision()and export it as symbol for KVM to use.Provisioning key is sensitive. SGX driver only allows to create enclave which can access provisioning key when enclave creator has permission to open
/dev/sgx_provision. It should apply to VM as well, as provisioning key is platform specific, thus unrestricted VM can also potentially compromise provisioning key.Move provisioning device creation out of
sgx_drv_init()tosgx_init()as preparation for adding SGX virtualization support, so that even SGX driver is not enabled due to flexible launch control is not available, SGX virtualization can still be enabled, and use it to restrict VM's capability of being able to access provisioning key.Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/include/asm/sgx.h | 3 +++ arch/x86/kernel/cpu/sgx/driver.c | 17 ------------ arch/x86/kernel/cpu/sgx/ioctl.c | 16 ++---------- arch/x86/kernel/cpu/sgx/main.c | 45 +++++++++++++++++++++++++++++++- 4 files changed, 49 insertions(+), 32 deletions(-)arch/x86/include/asm/sgx.h +3
-0@@ -4,6 +4,9 @@#include <linux/types.h> +int sgx_set_attribute (unsigned long *allowed_attributes, +unsigned int attribute_fd); +#ifdef CONFIG_X86_SGX_VIRTUALIZATION struct sgx_pageinfo ;arch/x86/kernel/cpu/sgx/driver.c +0
-17@@ -133,10 +133,6 @@ static const struct file_operations sgx_encl_fops = { .get_unmapped_area = sgx_get_unmapped_area, }; -const struct file_operations sgx_provision_fops = { - .owner =THIS_MODULE , -}; -static struct miscdevice sgx_dev_enclave = { .minor =MISC_DYNAMIC_MINOR , .name ="sgx_enclave" ,@@ -144,13 +140,6 @@ static struct miscdevice sgx_dev_enclave = { .fops = &sgx_encl_fops, }; -static struct miscdevice sgx_dev_provision = { - .minor =MISC_DYNAMIC_MINOR , - .name ="sgx_provision" , - .nodename ="sgx_provision" , - .fops = &sgx_provision_fops, -}; -int __init sgx_drv_init (void ) {unsigned int eax, ebx, ecx, edx;@@ -184,11 +173,5 @@ int __init sgx_drv_init(void)if (ret)return ret; - ret =misc_register (&sgx_dev_provision); -if (ret) { -misc_deregister (&sgx_dev_enclave); -return ret; - } -return 0 ; }arch/x86/kernel/cpu/sgx/ioctl.c +2
-14@@ -2,6 +2,7 @@/* Copyright(c) 2016-20 Intel Corporation. */ #include <asm/mman.h> +#include <asm/sgx.h> #include <linux/mman.h> #include <linux/delay.h> #include <linux/file.h> @@ -664,24 +665,11 @@ static long sgx_ioc_enclave_init(struct sgx_encl *encl, void __user *arg)static long sgx_ioc_enclave_provision (struct sgx_encl *encl,void __user *arg) {struct sgx_enclave_provision params; -struct file *file;if (copy_from_user (¶ms, arg,sizeof (params)))return -EFAULT ; - file =fget (params.fd); -if (!file) -return -EINVAL ; - -if (file->f_op != &sgx_provision_fops) { -fput (file); -return -EINVAL ; - } - - encl->attributes_mask |=SGX_ATTR_PROVISIONKEY ; - -fput (file); -return 0 ; +return sgx_set_attribute (&encl->attributes_mask, params.fd); }long sgx_ioctl (struct file *filep,unsigned int cmd,unsigned long arg)arch/x86/kernel/cpu/sgx/main.c +7
-0@@ -1,14 +1,18 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright(c) 2016-20 Intel Corporation. */ +#include <linux/file.h> #include <linux/freezer.h> #include <linux/highmem.h> #include <linux/kthread.h> +#include <linux/miscdevice.h> #include <linux/pagemap.h> #include <linux/ratelimit.h> #include <linux/sched/mm.h> #include <linux/sched/signal.h> #include <linux/slab.h> +#include <asm/sgx_arch.h> +#include <asm/sgx.h> #include "driver.h" #include "encl.h" #include "encls.h" @@ -705,6 +709,38 @@ void sgx_update_lepubkeyhash(u64 *lepubkeyhash)wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 + i, lepubkeyhash[i]); } +const struct file_operations sgx_provision_fops = { + .owner =THIS_MODULE , +};
arch/x86/kernel/cpu/sgx/main.c void sgx_update_lepubkeyhash(u64 *lepubkeyhash) · line 724
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +10
-0+ +static struct miscdevice sgx_dev_provision = { + .minor =MISC_DYNAMIC_MINOR , + .name ="sgx_provision" , + .nodename ="sgx_provision" , + .fops = &sgx_provision_fops, +}; + +int sgx_set_attribute (unsigned long *allowed_attributes, +unsigned int attribute_fd)
kdoc
Quoted 48 lines
arch/x86/kernel/cpu/sgx/main.c +27
-1+{ +struct file *file; + + file =fget (attribute_fd); +if (!file) +return -EINVAL ; + +if (file->f_op != &sgx_provision_fops) { +fput (file); +return -EINVAL ; + } + + *allowed_attributes |=SGX_ATTR_PROVISIONKEY ; + +fput (file); +return 0 ; +} +EXPORT_SYMBOL_GPL (sgx_set_attribute); +static void __init sgx_init (void ) {int ret;@@ -719,13 +755,20 @@ static void __init sgx_init(void)if (!sgx_page_reclaimer_init ()) goto err_page_cache; + ret =misc_register (&sgx_dev_provision); +if (ret) + goto err_kthread; +/* Success if the native *or* virtual EPC driver initialized cleanly. */ ret = !!sgx_drv_init () & !!sgx_virt_epc_init ();if (ret) - goto err_kthread; + goto err_provision;return ; +err_provision: +misc_deregister (&sgx_dev_provision); + err_kthread:kthread_stop (ksgxd_tsk);Signature · 1 line
2.29.2
/Jarkko
Sean Christopherson
maintainerRe: [RFC PATCH v2 05/26] x86/sgx: Introduce virtual EPC for use by KVM guests
In reply to Jarkko SakkinenQuoted 10 lines · Jarkko Sakkinen
Quoted 6 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add a misc device
/dev/sgx_virt_epcto allow userspace to allocate "raw" EPC without an associated enclave. The intended and only known use case for raw EPC allocation is to expose EPC to a KVM guest, hence thevirt_epcmoniker, virt.{c,h} files andX86_SGX_VIRTUALIZATIONKconfig.Is
/dev/sgx_virt_epcsomething only usable for KVM, or is there any thinkable use outside of the KVM context?
I can't think of a sane use case without KVM (or an out-of-tree hypervisor). Doing anything useful with EPC requires ENCLS, which means being able to run CPL0 code.
Re: [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Jarkko SakkinenQuoted 8 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/main.c static bool __init sgx_page_cache_init(void) · line 706
Quoted 7 lines
arch/x86/kernel/cpu/sgx/main.c +7
-0+void sgx_update_lepubkeyhash (u64 *lepubkeyhash) +{ +int i; + +for (i =0 ; i <4 ; i++) +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 + i, lepubkeyhash[i]); +}Missing kdoc.
I dunno... kdoc is nice, but I'm not sure its verbosity is useful here, even if this function is called from more than one .c file.
I'd be happy with a single-line comment, personally.
Re: [RFC PATCH v2 06/26] x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Kai Huangarch/x86/kernel/cpu/feat_ctl.c void init_ia32_feat_ctl(struct cpuinfo_x86 *c) · line 144
Quoted 12 lines · Kai Huang
arch/x86/kernel/cpu/feat_ctl.c +9
-3- enable_sgx =cpu_has (c,X86_FEATURE_SGX ) && -cpu_has (c,X86_FEATURE_SGX_LC ) && -IS_ENABLED (CONFIG_X86_SGX ); + enable_sgx_driver =cpu_has (c,X86_FEATURE_SGX ) && +cpu_has (c,X86_FEATURE_SGX1 ) && +IS_ENABLED (CONFIG_X86_SGX ) && +cpu_has (c,X86_FEATURE_SGX_LC ); + enable_sgx_virt =cpu_has (c,X86_FEATURE_SGX ) && +cpu_has (c,X86_FEATURE_SGX1 ) && +IS_ENABLED (CONFIG_X86_SGX ) && +IS_ENABLED (CONFIG_X86_SGX_VIRTUALIZATION ) && + enable_vmx;
Would it be too much to ask that the SGX/SGX1 checks not be duplicated?
Perhaps:
enable_sgx_any = cpu_feature_enabled(CONFIG_X86_SGX) && cpu_feature_enabled(CONFIG_X86_SGX1); enable_sgx_driver = enable_sgx_any && cpu_has(c, X86_FEATURE_SGX_LC); enable_sgx_virt = enable_sgx_any && enable_vmx && IS_ENABLED(CONFIG_X86_SGX_VIRTUALIZATION)
BTW, CONFIG_X86_SGX_VIRTUALIZATION is a pretty porky name. Maybe just CONFIG_X86_SGX_VIRT?
Re: [RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
In reply to Jarkko SakkinenQuoted 11 lines · On Wed, 20 Jan 2021 13:50:12 +0200 Jarkko Sakkinen wrote:
Quoted 6 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add SGX1 and SGX2 feature flags, via CPUID.0x12.0x0.EAX, as scattered features. As part of virtualizing SGX, KVM will expose the SGX CPUID leafs to its guest, and to do so correctly needs to query hardware and kernel support for SGX1 and SGX2.
This commit message is missing reasoning behind scattered vs. own word.
Please just document the reasoning, that's all.
OK. Will do. How about:
"Add SGX1 and SGX2 feature flags, via CPUID.0x12.0x0.EAX, as scattered features, since adding a new leaf for only two bits would be wasteful."
?
Quoted 2 lines
/Jarkko
Re: [RFC PATCH v2 14/26] x86/sgx: Move provisioning device creation out of SGX driver
In reply to Jarkko SakkinenQuoted 17 lines
Quoted 5 lines
arch/x86/kernel/cpu/sgx/main.c +3
-0+const struct file_operations sgx_provision_fops = { + .owner =THIS_MODULE , +};arch/x86/kernel/cpu/sgx/main.c void sgx_update_lepubkeyhash(u64 *lepubkeyhash) · line 724
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +10
-0+ +static struct miscdevice sgx_dev_provision = { + .minor =MISC_DYNAMIC_MINOR , + .name ="sgx_provision" , + .nodename ="sgx_provision" , + .fops = &sgx_provision_fops, +}; + +int sgx_set_attribute (unsigned long *allowed_attributes, +unsigned int attribute_fd)kdoc
Will do.
Quoted 20 lines
Quoted 20 lines
arch/x86/kernel/cpu/sgx/main.c +19
-0+{ +struct file *file; + + file =fget (attribute_fd); +if (!file) +return -EINVAL ; + +if (file->f_op != &sgx_provision_fops) { +fput (file); +return -EINVAL ; + } + + *allowed_attributes |=SGX_ATTR_PROVISIONKEY ; + +fput (file); +return 0 ; +} +EXPORT_SYMBOL_GPL (sgx_set_attribute); +
Re: [RFC PATCH v2 13/26] x86/sgx: Add helpers to expose ECREATE and EINIT to KVM
In reply to Jarkko SakkinenQuoted 13 lines · On Wed, 20 Jan 2021 14:04:36 +0200 Jarkko Sakkinen wrote:
Quoted 12 lines · Kai Huang
From: Sean Christopherson <[email protected]>
The bare-metal kernel must intercept ECREATE to be able to impose policies on guests. When it does this, the bare-metal kernel runs ECREATE against the userspace mapping of the virtualized EPC.
Provide wrappers around
__ecreate()and__einit()to hide the ugliness of overloading the ENCLS return value to encode multiple error formats in a single int. KVM will trap-and-execute ECREATE and EINIT as part of SGX virtualization, and on an exception, KVM needs the trapnr so that it can inject the correct fault into the guest.
Quoted 4 lines
Quoted 2 lines
Signed-off-by Sean Christopherson <[email protected]>[Kai: Use sgx_update_lepubkeyhash() to update pubkey hash MSRs.]If you have to go to this, please instead use co-developed-by.
I'll remove it. I don't want to use co-developed-by here.
Quoted 77 lines
Quoted 77 lines
Signed-off-by Kai Huang <[email protected]>---
v1->v2:- Refined commit msg based on Dave's comment.
- Added comment to explain why to use__uaccess_xxx().--- arch/x86/include/asm/sgx.h | 16 +++++++++ arch/x86/kernel/cpu/sgx/virt.c | 61 ++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+) create mode 100644 arch/x86/include/asm/sgx.harch/x86/include/asm/sgx.h new file +16
-0@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_SGX_H +#define _ASM_X86_SGX_H + +#include <linux/types.h> + +#ifdef CONFIG_X86_SGX_VIRTUALIZATION +struct sgx_pageinfo ; + +int sgx_virt_ecreate (struct sgx_pageinfo *pageinfo,void __user *secs, +int *trapnr); +int sgx_virt_einit (void __user *sigstruct,void __user *token, +void __user *secs,u64 *lepubkeyhash,int *trapnr); +#endif + +#endif /* _ASM_X86_SGX_H */ arch/x86/kernel/cpu/sgx/virt.c +34
-0@@ -253,3 +253,64 @@ int __init sgx_virt_epc_init(void)return misc_register (&sgx_virt_epc_dev); } + +int sgx_virt_ecreate (struct sgx_pageinfo *pageinfo,void __user *secs, +int *trapnr) +{ +int ret; + +/* +* @secs is userspace address, and it's not guaranteed @secs points at +* an actual EPC page. It's also possible to generate a kernel mapping +* to physical EPC page by resolving PFN but using __uaccess_xx() is +* simpler. +*/ +__uaccess_begin (); + ret =__ecreate (pageinfo, (void *)secs); +__uaccess_end (); + +if (encls_faulted (ret)) { + *trapnr =ENCLS_TRAPNR (ret); +return -EFAULT ; + } + +/* ECREATE doesn't return an error code, it faults or succeeds. */ +WARN_ON_ONCE (ret); +return 0 ; +} +EXPORT_SYMBOL_GPL (sgx_virt_ecreate); + +static int __sgx_virt_einit (void __user *sigstruct,void __user *token, +void __user *secs) +{ +int ret; + +__uaccess_begin (); + ret =__einit ((void *)sigstruct, (void *)token, (void *)secs);
Quoted 8 lines
arch/x86/kernel/cpu/sgx/virt.c int __init sgx_virt_epc_init(void) · line 295
Quoted 6 lines
arch/x86/kernel/cpu/sgx/virt.c +6
-0+__uaccess_end (); +return ret; +} + +int sgx_virt_einit (void __user *sigstruct,void __user *token, +void __user *secs,u64 *lepubkeyhash,int *trapnr)kdoc missing.
OK. Will add for both sgx_virt_create() and sgx_virt_init().
Quoted 28 lines
Quoted 26 lines
arch/x86/kernel/cpu/sgx/virt.c +21
-0+{ +int ret; + +if (!boot_cpu_has (X86_FEATURE_SGX_LC )) { + ret =__sgx_virt_einit (sigstruct, token, secs); + }else { +preempt_disable (); + +sgx_update_lepubkeyhash (lepubkeyhash); + + ret =__sgx_virt_einit (sigstruct, token, secs); +preempt_enable (); + } + +if (encls_faulted (ret)) { + *trapnr =ENCLS_TRAPNR (ret); +return -EFAULT ; + } +return ret; +} +EXPORT_SYMBOL_GPL (sgx_virt_einit);Signature · 1 line
2.29.2/Jarkko
Re: [RFC PATCH v2 06/26] x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Sean ChristophersonQuoted 11 lines · Sean Christopherson
Quoted 2 lines · Dave Hansen
BTW,
CONFIG_X86_SGX_VIRTUALIZATIONis a pretty porky name. Maybe justCONFIG_X86_SGX_VIRT?Mmm, bacon. I used the full "virtualization" to avoid any possible confusion with virtual memory. The existing
sgx_get_epc_virt_addr()in particular gave me pause.I agree it's long and not consistent since other code in this series uses "virt". My thinking was that most shortand versions, e.g.
virt_epc, would be used only in contexts that are already fairly obvious to be KVM/virtualization related, whereas the porcineKconfigwould help establish that context.
Not a big deal either way. I agree that "virt" can be confusing.
Considering that:
+config X86_SGX_VIRTUALIZATION
+ depends on ... KVM_INTEL
Calling it X86_SGX_KVM doesn't seem horrible either.
Re: [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Dave HansenQuoted 10 lines · On Wed, 20 Jan 2021 10:36:09 -0800 Dave Hansen wrote:
Quoted 8 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/main.c static bool __init sgx_page_cache_init(void) · line 706
Quoted 7 lines
arch/x86/kernel/cpu/sgx/main.c +7
-0+void sgx_update_lepubkeyhash (u64 *lepubkeyhash) +{ +int i; + +for (i =0 ; i <4 ; i++) +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 + i, lepubkeyhash[i]); +}Missing kdoc.
Quoted 5 lines
I dunno... kdoc is nice, but I'm not sure its verbosity is useful here, even if this function is called from more than one .c file.
I'd be happy with a single-line comment, personally.
I actually feel the function name already explains what the function does clearly, therefore I don't think even comment is needed. To be honest I don't know how to rephrase here. Perhaps:
/* Update SGX LEPUBKEYHASH MSRs of the platform. */
?
Sean Christopherson
maintainerRe: [RFC PATCH v2 06/26] x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Dave HansenQuoted 2 lines · Dave Hansen
BTW,
CONFIG_X86_SGX_VIRTUALIZATIONis a pretty porky name. Maybe justCONFIG_X86_SGX_VIRT?
Mmm, bacon. I used the full "virtualization" to avoid any possible confusion with virtual memory. The existing sgx_get_epc_virt_addr() in particular gave me pause.
I agree it's long and not consistent since other code in this series uses "virt". My thinking was that most shortand versions, e.g. virt_epc, would be used only in contexts that are already fairly obvious to be KVM/virtualization related, whereas the porcine Kconfig would help establish that context.
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 15/26] KVM: VMX: Convert vcpu_vmx.exit_reason to a union
In reply to Sean ChristophersonQuoted 57 lines · Sean Christopherson
Quoted 56 lines · Jarkko Sakkinen
Quoted 35 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Convert vcpu_vmx.exit_reason from a u32 to a union (of size u32). The full
VM_EXIT_REASONfield is comprised of a 16-bit basic exit reason in bits 15:0, and single-bit modifiers in bits 31:16.Historically, KVM has only had to worry about handling the "failed VM-Entry" modifier, which could only be set in very specific flows and required dedicated handling. I.e. manually stripping the
FAILED_VMENTRYbit was a somewhat viable approach. But even with only a single bit to worry about, KVM has had several bugs related to comparing a basic exit reason against the full exit reason store invcpu_vmx.Upcoming Intel features, e.g. SGX, will add new modifier bits that can be set on more or less any VM-Exit, as opposed to the significantly more restricted
FAILED_VMENTRY, i.e. correctly handling everything in one-off flows isn't scalable. Tracking exit reason in a union forces code to explicitly choose between consuming the full exit reason and the basic exit, and is a convenient way to document and access the modifiers.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kvm/vmx/nested.c | 42 +++++++++++++++--------- arch/x86/kvm/vmx/vmx.c | 68 ++++++++++++++++++++------------------- arch/x86/kvm/vmx/vmx.h | 25 +++++++++++++- 3 files changed, 86 insertions(+), 49 deletions(-)arch/x86/kvm/vmx/nested.c +0
-0arch/x86/kvm/vmx/nested.c enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, · line 3318
Quoted 9 lines
arch/x86/kvm/vmx/nested.c +5
-1@@ -3311,7 +3311,11 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);enum vm_entry_failure_code entry_failure_code;bool evaluate_pending_interrupts; -u32 exit_reason, failed_index; +u32 failed_index; +union vmx_exit_reason exit_reason = { + .basic =-1 , + .failed_vmentry =1 , + };Instead, put this declaration to the correct place, following the reverse christmas tree ordering:
union vmx_exit_reason exit_reason = {};And after declarations:
exit_reason.basic = -1; exit_reason.failed_vmentry = 1;
Quoted 6 lines
Quoted 1 line
More pleasing for the eye.
I disagree (obviously, since I wrote the patch). Initializing the fields to their respective values is a critical, but subtle, aspect of this code. Making the code stand out via explicit initialization is a good thing, and we really don't want any possibility of code touching
exit_reasonbefore it is initialized.
The struct does get initialized to zero, and fields get initialized to their respective values. What is the critical aspect that is gone? I'm not following.
/Jarkko
Re: [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Dave HansenQuoted 15 lines · On Wed, 20 Jan 2021 15:50:31 -0800 Dave Hansen wrote:
Quoted 5 lines · Kai Huang
I actually feel the function name already explains what the function does clearly, therefore I don't think even comment is needed. To be honest I don't know how to rephrase here. Perhaps:
/* Update SGX LEPUBKEYHASH MSRs of the platform. */
Whee! I'm gonna write me a function comment!
/*
* A Launch Enclave (LE) must be signed with a public key
* that matches this SHA256 hash. Usually overwrites Intel's
* default signing key.
*/
Quoted 3 lines
So, this isn't a one-liner. *But*, it tells us what "le" means, what "pubkey" means and implies that there need to be 4x64-bits worth of MSR writes to get to a SHA256 hash.
In current linux driver implementation, LE is effectively abandoned, because the initialization of any enclave doesn't take a valid TOKEN, making initializing enclave requires hash of enclave's signer equal to the hash in SGX_LEPUBKEYHASH MSRs.
I written the function name based on SDM's description, to reflect the fact that we are updating the SGX_LEPUBKEYHASH MSRs, but nothing more.
So perhaps below?
/*
* Update the SGX_LEPUBKEYHASH MSRs to the values specified by caller. *
* EINITTOKEN is not used in enclave initialization, which requires
* hash of enclave's signer must match values in SGX_LEPUBKEYHASH MSRs
* to make EINIT be successful.
*/
It also tells what it's usually doing
Quoted 1 line
here: overwriting Intel's blasted hash.
Technically, only initial value is intel's pubkey hash. This function overwrites whatever pubkey hash that used to sign previous enclave.
Quoted 2 lines
It sure beats the entirely uncommented for loop that we've got today.
Agreed, although to me it seems the comment is a little bit out of the scope of this function itself, but is more about the logic of the caller.
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
In reply to Kai HuangQuoted 9 lines · Kai Huang
Quoted 8 lines · On Wed, 20 Jan 2021 13:50:12 +0200 Jarkko Sakkinen wrote:
Quoted 6 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add SGX1 and SGX2 feature flags, via CPUID.0x12.0x0.EAX, as scattered features. As part of virtualizing SGX, KVM will expose the SGX CPUID leafs to its guest, and to do so correctly needs to query hardware and kernel support for SGX1 and SGX2.
Quoted 8 lines
Quoted 3 lines
This commit message is missing reasoning behind scattered vs. own word.
Please just document the reasoning, that's all.
OK. Will do. How about:
"Add SGX1 and SGX2 feature flags, via CPUID.0x12.0x0.EAX, as scattered features, since adding a new leaf for only two bits would be wasteful."
For *me*, that would be sufficient.
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 11/26] x86/sgx: Add encls_faulted() helper
In reply to Kai HuangQuoted 34 lines · Kai Huang
Quoted 22 lines · On Wed, 20 Jan 2021 14:03:08 +0200 Jarkko Sakkinen wrote:
Quoted 21 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add a helper to extract the fault indicator from an encoded ENCLS return value. SGX virtualization will also need to detect ENCLS faults.
Signed-off-by Sean Christopherson <[email protected]>Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/encls.h | 14 +++++++++++++- arch/x86/kernel/cpu/sgx/ioctl.c | 2 +- 2 files changed, 14 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/sgx/encls.h +0
-0@@ -40,6 +40,18 @@ }while (0 ); \ }Quoted 10 lines
arch/x86/kernel/cpu/sgx/encls.h line 48
Quoted 6 lines
arch/x86/kernel/cpu/sgx/encls.h +6
-0+/* +* encls_faulted() - Check if an ENCLS leaf faulted given an error code +* @ret the return value of an ENCLS leaf function call +* +* Return: +* %true if @ret indicates a fault, %false otherwise Follow here the style of commenting as in
ioctl.c, for the return value. It has optimal readability both as text, and also when converted to HTML. Seesgx_ioc_enclave_add_pages()for an example.
Quoted 5 lines
You mean something like below?
Return:
- %true: @ret indicates a fault.
- %false: @ret indicates no fault.
Yeah, with '\t' indentation. I'd remove also '%'. Also '@ret' is redudant.
To put this all together:
* Return:
* - true: ENCLS leaf faulted.
* - false: Otherwise.
I tried various ways and this was the best way to document return values that i've found. It's easy to read as plain text, and also has the benefit that return values get nicely lined up in htmldocs.
I've been even considering a patch for
https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt
The only advice given ATM is: "Take a look around the source tree for examples."
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Dave HansenQuoted 10 lines · Dave Hansen
Quoted 8 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/main.c static bool __init sgx_page_cache_init(void) · line 706
Quoted 7 lines
arch/x86/kernel/cpu/sgx/main.c +7
-0+void sgx_update_lepubkeyhash (u64 *lepubkeyhash) +{ +int i; + +for (i =0 ; i <4 ; i++) +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 + i, lepubkeyhash[i]); +}Missing kdoc.
Quoted 4 lines
I dunno... kdoc is nice, but I'm not sure its verbosity is useful here, even if this function is called from more than one .c file.
I'd be happy with a single-line comment, personally.
WFM.
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 06/26] x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Dave HansenQuoted 27 lines · Dave Hansen
arch/x86/kernel/cpu/feat_ctl.c void init_ia32_feat_ctl(struct cpuinfo_x86 *c) · line 144
Quoted 12 lines · Kai Huang
arch/x86/kernel/cpu/feat_ctl.c +9
-3- enable_sgx =cpu_has (c,X86_FEATURE_SGX ) && -cpu_has (c,X86_FEATURE_SGX_LC ) && -IS_ENABLED (CONFIG_X86_SGX ); + enable_sgx_driver =cpu_has (c,X86_FEATURE_SGX ) && +cpu_has (c,X86_FEATURE_SGX1 ) && +IS_ENABLED (CONFIG_X86_SGX ) && +cpu_has (c,X86_FEATURE_SGX_LC ); + enable_sgx_virt =cpu_has (c,X86_FEATURE_SGX ) && +cpu_has (c,X86_FEATURE_SGX1 ) && +IS_ENABLED (CONFIG_X86_SGX ) && +IS_ENABLED (CONFIG_X86_SGX_VIRTUALIZATION ) && + enable_vmx;Would it be too much to ask that the SGX/SGX1 checks not be duplicated?
Perhaps:enable_sgx_any = cpu_feature_enabled(CONFIG_X86_SGX) && cpu_feature_enabled(CONFIG_X86_SGX1); enable_sgx_driver = enable_sgx_any && cpu_has(c, X86_FEATURE_SGX_LC); enable_sgx_virt = enable_sgx_any && enable_vmx && IS_ENABLED(CONFIG_X86_SGX_VIRTUALIZATION)
Quoted 2 lines
BTW,
CONFIG_X86_SGX_VIRTUALIZATIONis a pretty porky name. Maybe justCONFIG_X86_SGX_VIRT?
If my /dev/sgx_vepc naming gets acceptance, then IMHO the best name ought to be CONFIG_X86_VEPC.
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 06/26] x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Dave HansenQuoted 20 lines · Dave Hansen
Quoted 11 lines · Sean Christopherson
Quoted 2 lines · Dave Hansen
BTW,
CONFIG_X86_SGX_VIRTUALIZATIONis a pretty porky name. Maybe justCONFIG_X86_SGX_VIRT?Mmm, bacon. I used the full "virtualization" to avoid any possible confusion with virtual memory. The existing
sgx_get_epc_virt_addr()in particular gave me pause.I agree it's long and not consistent since other code in this series uses "virt". My thinking was that most shortand versions, e.g.
virt_epc, would be used only in contexts that are already fairly obvious to be KVM/virtualization related, whereas the porcineKconfigwould help establish that context.Not a big deal either way. I agree that "virt" can be confusing.
Considering that:
(quoted diff) +2
-0+config X86_SGX_VIRTUALIZATION + depends on ... KVM_INTEL
Quoted 1 line
Calling it
X86_SGX_KVMdoesn't seem horrible either.
This is something that I could cope just as well as with my proposal as this is KVM tied feature.
/Jarkko
Re: [RFC PATCH v2 11/26] x86/sgx: Add encls_faulted() helper
In reply to Jarkko SakkinenQuoted 53 lines · On Thu, 21 Jan 2021 03:08:18 +0200 Jarkko Sakkinen wrote:
Quoted 34 lines · Kai Huang
Quoted 22 lines · On Wed, 20 Jan 2021 14:03:08 +0200 Jarkko Sakkinen wrote:
Quoted 21 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add a helper to extract the fault indicator from an encoded ENCLS return value. SGX virtualization will also need to detect ENCLS faults.
Signed-off-by Sean Christopherson <[email protected]>Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/encls.h | 14 +++++++++++++- arch/x86/kernel/cpu/sgx/ioctl.c | 2 +- 2 files changed, 14 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/sgx/encls.h +0
-0@@ -40,6 +40,18 @@ }while (0 ); \ }Quoted 10 lines
arch/x86/kernel/cpu/sgx/encls.h line 48
Quoted 6 lines
arch/x86/kernel/cpu/sgx/encls.h +6
-0+/* +* encls_faulted() - Check if an ENCLS leaf faulted given an error code +* @ret the return value of an ENCLS leaf function call +* +* Return: +* %true if @ret indicates a fault, %false otherwise Follow here the style of commenting as in
ioctl.c, for the return value. It has optimal readability both as text, and also when converted to HTML. Seesgx_ioc_enclave_add_pages()for an example.Quoted 5 lines
You mean something like below?
Return:
- %true: @ret indicates a fault.
- %false: @ret indicates no fault.Yeah, with '\t' indentation. I'd remove also '%'. Also '@ret' is redudant.
To put this all together:
* Return:
* - true: ENCLS leaf faulted.
* - false: Otherwise.I tried various ways and this was the best way to document return values that i've found. It's easy to read as plain text, and also has the benefit that return values get nicely lined up in htmldocs.
Quoted 6 lines
I've been even considering a patch for
https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt
The only advice given ATM is: "Take a look around the source tree for examples."
OK. Will change to yours. Thanks for the info.
Quoted 2 lines
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Kai HuangQuoted 17 lines · Kai Huang
Quoted 10 lines · On Wed, 20 Jan 2021 10:36:09 -0800 Dave Hansen wrote:
Quoted 8 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/main.c static bool __init sgx_page_cache_init(void) · line 706
Quoted 7 lines
arch/x86/kernel/cpu/sgx/main.c +7
-0+void sgx_update_lepubkeyhash (u64 *lepubkeyhash) +{ +int i; + +for (i =0 ; i <4 ; i++) +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 + i, lepubkeyhash[i]); +}Missing kdoc.
Quoted 5 lines
I dunno... kdoc is nice, but I'm not sure its verbosity is useful here, even if this function is called from more than one .c file.
I'd be happy with a single-line comment, personally.
Quoted 7 lines
I actually feel the function name already explains what the function does clearly, therefore I don't think even comment is needed. To be honest I don't know how to rephrase here. Perhaps:
/* Update SGX LEPUBKEYHASH MSRs of the platform. */
?
WFM, thanks.
/Jarkko
Re: [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Kai HuangQuoted 8 lines · Kai Huang
/*
* Update theSGX_LEPUBKEYHASHMSRs to the values specified by caller. *
* EINITTOKEN is not used in enclave initialization, which requires
* hash of enclave's signer must match values inSGX_LEPUBKEYHASHMSRs
* to make EINIT be successful.
*/
I'm grumpy, but I hate it.
I'll stop the bike shedding for now, though.
Re: [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Kai HuangQuoted 30 lines · On Thu, 21 Jan 2021 14:06:38 +1300 Kai Huang wrote:
Quoted 15 lines · On Wed, 20 Jan 2021 15:50:31 -0800 Dave Hansen wrote:
Quoted 5 lines · Kai Huang
I actually feel the function name already explains what the function does clearly, therefore I don't think even comment is needed. To be honest I don't know how to rephrase here. Perhaps:
/* Update SGX LEPUBKEYHASH MSRs of the platform. */
Whee! I'm gonna write me a function comment!
/*
* A Launch Enclave (LE) must be signed with a public key
* that matches this SHA256 hash. Usually overwrites Intel's
* default signing key.
*/Quoted 3 lines
So, this isn't a one-liner. *But*, it tells us what "le" means, what "pubkey" means and implies that there need to be 4x64-bits worth of MSR writes to get to a SHA256 hash.
In current linux driver implementation, LE is effectively abandoned, because the initialization of any enclave doesn't take a valid TOKEN, making initializing enclave requires hash of enclave's signer equal to the hash in
SGX_LEPUBKEYHASHMSRs.I written the function name based on SDM's description, to reflect the fact that we are updating the
SGX_LEPUBKEYHASHMSRs, but nothing more.So perhaps below?
Quoted 8 lines
/*
* Update theSGX_LEPUBKEYHASHMSRs to the values specified by caller. *
* EINITTOKEN is not used in enclave initialization, which requires
* hash of enclave's signer must match values inSGX_LEPUBKEYHASHMSRs
* to make EINIT be successful.
*/
Actually I take it back. This is only valid for bare-metal driver. For KVM guest, it should be:
/*
* Update the SGX_LEPUBKEYHASH MSRs according to guest's *virtual*
* SGX_LEPUBKEYHASH MSRs values, to make EINIT from guest consistent
* with hardware behavior.
*/
So like I said below, the comment is actually more reasonable for the logic of caller of this function.
Makes sense?
Quoted 12 lines
It also tells what it's usually doing
Quoted 1 line
here: overwriting Intel's blasted hash.
Technically, only initial value is intel's pubkey hash. This function overwrites whatever pubkey hash that used to sign previous enclave.
Quoted 2 lines
It sure beats the entirely uncommented for loop that we've got today.
Agreed, although to me it seems the comment is a little bit out of the scope of this function itself, but is more about the logic of the caller.
Jarkko Sakkinen
maintainerQuoted 24 lines · Kai Huang
Quoted 23 lines · On Thu, 21 Jan 2021 01:43:41 +0200 Jarkko Sakkinen wrote:
Quoted 21 lines · Kai Huang
Quoted 17 lines · Jarkko Sakkinen
Can you send a new version that applies:
$ git pw series apply 416463 Applying: x86/cpufeatures: Add SGX1 and SGX2 sub-features Applying: x86/sgx: Remove a warn from sgx_free_epc_page() Applying: x86/sgx: Wipe out EREMOVE from sgx_free_epc_page() Applying: x86/sgx: Add SGX_CHILD_PRESENT hardware error code Applying: x86/sgx: Introduce virtual EPC for use by KVM guests Applying: x86/cpu/intel: Allow SGX virtualization without Launch Control support Applying: x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled error: sha1 information is lacking or useless (arch/x86/kernel/cpu/sgx/main.c). error: could not build fake ancestor hint: Use 'git am --show-current-patch=diff' to see the failed patch Patch failed at 0007 x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".Could you let me know which branch should I rebase to? It appears your linux-sgx tree next branch?
Quoted 7 lines
Quoted 5 lines
I tried my tree first, which was actually out-of-sync, so I rebase it. You can find it from
MAINTAINERSfailed.After sending this email I rebased it to tip/x86/sgx, which also failed. That tree failed with a merge conflict.
This series is based on v5.11-rc3, as mentioned in cover letter below:
Please use tip/x86/sgx as base instead.
/Jarkko
Quoted 1 line
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 05/26] x86/sgx: Introduce virtual EPC for use by KVM guests
In reply to Jarkko SakkinenQuoted 22 lines · Jarkko Sakkinen
Quoted 9 lines · Sean Christopherson
Quoted 8 lines · Jarkko Sakkinen
Quoted 6 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add a misc device
/dev/sgx_virt_epcto allow userspace to allocate "raw" EPC without an associated enclave. The intended and only known use case for raw EPC allocation is to expose EPC to a KVM guest, hence thevirt_epcmoniker, virt.{c,h} files andX86_SGX_VIRTUALIZATIONKconfig.Quoted 6 lines
Quoted 2 lines
Is
/dev/sgx_virt_epcsomething only usable for KVM, or is there any thinkable use outside of the KVM context?I can't think of a sane use case without KVM (or an out-of-tree hypervisor). Doing anything useful with EPC requires ENCLS, which means being able to run CPL0 code.
OK, thanks for elaboration.
To bring more context to my thinking to call this to
sgx_vepcis that it's kind of "standard" in other device names I've such as:
Quoted 8 lines
-
vhost_net
- vhci
- vtpm
- vcs*
- The list goes on..Since "virtual something" is always abbreviated just with a 'v', wouldn't it make sense to just follow along?
And even kVm :-)
/Jarkko
Quoted 6 lines · On Wed, 20 Jan 2021 17:39:13 +0100 Paolo Bonzini wrote:
Quoted 5 lines · Sean Christopherson
Quoted 4 lines · Kai Huang
I'd like to take this chance to ask: when this series is ready to be merged, what is the properly way to merge? This series has x86 non-sgx (cpufeature,
feat_ctl) and sgx changes, and it obviously has KVM changes too. So instance, who should be the one to take this series? And which tree and branch should I rebase to in next version?
Quoted 7 lines
Quoted 5 lines
The path of least resistance is likely to get acks for the x86 and sgx changes, and let Paolo take it through the KVM tree. The KVM changes are much more likely to have non-trivial conflicts, e.g. making
exit_reasona union touches a ton of code; getting and carrying acked-by for those will be tough sledding.Yes, the best way is to get a topic branch from Thomas or Borislav.
Thanks Paolo. I'll rebase against tip/x86/sgx for future versions.
Quoted 3 lines
Paolo
Quoted 28 lines · On Thu, 21 Jan 2021 03:16:19 +0200 Jarkko Sakkinen wrote:
Quoted 27 lines · Kai Huang
Quoted 26 lines · On Thu, 21 Jan 2021 01:43:41 +0200 Jarkko Sakkinen wrote:
Quoted 19 lines · Kai Huang
Quoted 17 lines · Jarkko Sakkinen
Can you send a new version that applies:
$ git pw series apply 416463 Applying: x86/cpufeatures: Add SGX1 and SGX2 sub-features Applying: x86/sgx: Remove a warn from sgx_free_epc_page() Applying: x86/sgx: Wipe out EREMOVE from sgx_free_epc_page() Applying: x86/sgx: Add SGX_CHILD_PRESENT hardware error code Applying: x86/sgx: Introduce virtual EPC for use by KVM guests Applying: x86/cpu/intel: Allow SGX virtualization without Launch Control support Applying: x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled error: sha1 information is lacking or useless (arch/x86/kernel/cpu/sgx/main.c). error: could not build fake ancestor hint: Use 'git am --show-current-patch=diff' to see the failed patch Patch failed at 0007 x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".Quoted 2 lines
Could you let me know which branch should I rebase to? It appears your linux-sgx tree next branch?
I tried my tree first, which was actually out-of-sync, so I rebase it. You can find it from
MAINTAINERSfailed.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
After sending this email I rebased it to tip/x86/sgx, which also failed. That tree failed with a merge conflict.
This series is based on v5.11-rc3, as mentioned in cover letter below:
Please use tip/x86/sgx as base instead.
When I wrote this series, the tip/x86/sgx was still v5.10-rc4. I felt I should rebase to some newer code, so I chose upstream v5.11-rc3.
I just checked the latest tip/x86/sgx, and it has updated to v5.11-rc3, so yes I will rebase to it for next version.
Quoted 3 lines
/Jarkko
Quoted 1 line
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 05/26] x86/sgx: Introduce virtual EPC for use by KVM guests
In reply to Sean ChristophersonQuoted 9 lines · Sean Christopherson
Quoted 8 lines · Jarkko Sakkinen
Quoted 6 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add a misc device
/dev/sgx_virt_epcto allow userspace to allocate "raw" EPC without an associated enclave. The intended and only known use case for raw EPC allocation is to expose EPC to a KVM guest, hence thevirt_epcmoniker, virt.{c,h} files andX86_SGX_VIRTUALIZATIONKconfig.
Quoted 6 lines
Quoted 2 lines
Is
/dev/sgx_virt_epcsomething only usable for KVM, or is there any thinkable use outside of the KVM context?I can't think of a sane use case without KVM (or an out-of-tree hypervisor). Doing anything useful with EPC requires ENCLS, which means being able to run CPL0 code.
OK, thanks for elaboration.
To bring more context to my thinking to call this to sgx_vepc is that it's kind of "standard" in other device names I've such as:
- vhost_net
- vhci
- vtpm
- vcs*
- The list goes on..
Since "virtual something" is always abbreviated just with a 'v', wouldn't it make sense to just follow along?
/Jarkko
Re: [RFC PATCH v2 06/26] x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Dave HansenQuoted 17 lines · On Wed, 20 Jan 2021 15:27:27 -0800 Dave Hansen wrote:
Quoted 11 lines · Sean Christopherson
Quoted 2 lines · Dave Hansen
BTW,
CONFIG_X86_SGX_VIRTUALIZATIONis a pretty porky name. Maybe justCONFIG_X86_SGX_VIRT?Mmm, bacon. I used the full "virtualization" to avoid any possible confusion with virtual memory. The existing
sgx_get_epc_virt_addr()in particular gave me pause.I agree it's long and not consistent since other code in this series uses "virt". My thinking was that most shortand versions, e.g.
virt_epc, would be used only in contexts that are already fairly obvious to be KVM/virtualization related, whereas the porcineKconfigwould help establish that context.Not a big deal either way. I agree that "virt" can be confusing.
Considering that:
Review
Quoted 2 lines
(quoted diff) +2
-0+config X86_SGX_VIRTUALIZATION + depends on ... KVM_INTEL
It is already in patch 3: Introduce virtual EPC for use by KVM guests:
+config X86_SGX_VIRTUALIZATION
+ bool "Software Guard eXtensions (SGX) Virtualization"
+ depends on X86_SGX && KVM_INTEL
Quoted 2 lines
Calling it
X86_SGX_KVMdoesn't seem horrible either.
Re: [RFC PATCH v2 06/26] x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Dave HansenQuoted 17 lines · On Wed, 20 Jan 2021 13:02:15 -0800 Dave Hansen wrote:
arch/x86/kernel/cpu/feat_ctl.c void init_ia32_feat_ctl(struct cpuinfo_x86 *c) · line 144
Quoted 12 lines · Kai Huang
arch/x86/kernel/cpu/feat_ctl.c +9
-3- enable_sgx =cpu_has (c,X86_FEATURE_SGX ) && -cpu_has (c,X86_FEATURE_SGX_LC ) && -IS_ENABLED (CONFIG_X86_SGX ); + enable_sgx_driver =cpu_has (c,X86_FEATURE_SGX ) && +cpu_has (c,X86_FEATURE_SGX1 ) && +IS_ENABLED (CONFIG_X86_SGX ) && +cpu_has (c,X86_FEATURE_SGX_LC ); + enable_sgx_virt =cpu_has (c,X86_FEATURE_SGX ) && +cpu_has (c,X86_FEATURE_SGX1 ) && +IS_ENABLED (CONFIG_X86_SGX ) && +IS_ENABLED (CONFIG_X86_SGX_VIRTUALIZATION ) && + enable_vmx;Would it be too much to ask that the SGX/SGX1 checks not be duplicated?
Perhaps:
Quoted 10 lines
enable_sgx_any = cpu_feature_enabled(CONFIG_X86_SGX) && cpu_feature_enabled(CONFIG_X86_SGX1); enable_sgx_driver = enable_sgx_any && cpu_has(c, X86_FEATURE_SGX_LC); enable_sgx_virt = enable_sgx_any && enable_vmx && IS_ENABLED(CONFIG_X86_SGX_VIRTUALIZATION)
I am happy to do it. Thanks.
Re: [RFC PATCH v2 08/26] x86/sgx: Expose SGX architectural definitions to the kernel
In reply to Jarkko SakkinenQuoted 7 lines · On Wed, 20 Jan 2021 13:58:48 +0200 Jarkko Sakkinen wrote:
Quoted 4 lines · Kai Huang
From: Sean Christopherson <[email protected]>
KVM will use many of the architectural constants and structs to virtualize SGX.
"Expose SGX architectural structures, as ..."
Will do.
Quoted 20 lines · On Thu, 21 Jan 2021 01:43:41 +0200 Jarkko Sakkinen wrote:
Quoted 19 lines · Kai Huang
Quoted 17 lines · Jarkko Sakkinen
Can you send a new version that applies:
$ git pw series apply 416463 Applying: x86/cpufeatures: Add SGX1 and SGX2 sub-features Applying: x86/sgx: Remove a warn from sgx_free_epc_page() Applying: x86/sgx: Wipe out EREMOVE from sgx_free_epc_page() Applying: x86/sgx: Add SGX_CHILD_PRESENT hardware error code Applying: x86/sgx: Introduce virtual EPC for use by KVM guests Applying: x86/cpu/intel: Allow SGX virtualization without Launch Control support Applying: x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled error: sha1 information is lacking or useless (arch/x86/kernel/cpu/sgx/main.c). error: could not build fake ancestor hint: Use 'git am --show-current-patch=diff' to see the failed patch Patch failed at 0007 x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".
Quoted 8 lines
Quoted 2 lines
Could you let me know which branch should I rebase to? It appears your linux-sgx tree next branch?
I tried my tree first, which was actually out-of-sync, so I rebase it. You can find it from
MAINTAINERSfailed.After sending this email I rebased it to tip/x86/sgx, which also failed. That tree failed with a merge conflict.
This series is based on v5.11-rc3, as mentioned in cover letter below:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
This series is based against upstream v5.11-rc3. You can also get the code from upstream branch of kvm-sgx repo on github:
https://github.com/intel/kvm-sgx.git upstream
Re: [RFC PATCH v2 06/26] x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Kai HuangReview
Quoted 11 lines · Kai Huang
Review
Quoted 6 lines
Not a big deal either way. I agree that "virt" can be confusing.
Considering that:
(quoted diff) +2
-0+config X86_SGX_VIRTUALIZATION + depends on ... KVM_INTELIt is already in patch 3: Introduce virtual EPC for use by KVM guests:
(quoted diff) +3
-0+config X86_SGX_VIRTUALIZATION + bool "Software Guard eXtensions (SGX) Virtualization" + depends on X86_SGX && KVM_INTEL
Right, I'm suggesting that patch 3 should call it:
X86_SGX_KVM
instead of:
X86_SGX_VIRTUALIZATION
Re: [RFC PATCH v2 06/26] x86/cpu/intel: Allow SGX virtualization without Launch Control support
Quoted 15 lines · On Wed, 20 Jan 2021 15:51:44 -0800 Dave Hansen wrote:
Review
Quoted 11 lines · Kai Huang
Review
Quoted 6 lines
Not a big deal either way. I agree that "virt" can be confusing.
Considering that:
(quoted diff) +2
-0+config X86_SGX_VIRTUALIZATION + depends on ... KVM_INTELIt is already in patch 3: Introduce virtual EPC for use by KVM guests:
(quoted diff) +3
-0+config X86_SGX_VIRTUALIZATION + bool "Software Guard eXtensions (SGX) Virtualization" + depends on X86_SGX && KVM_INTELRight, I'm suggesting that patch 3 should call it:
Quoted 5 lines
X86_SGX_KVMinstead of:
X86_SGX_VIRTUALIZATION
In case we want to change to X86_SGX_KVM, is it more reasonable to put it to arch/x86/kvm/Kconfig (maybe change to X86_KVM_SGX)?
Jarkko also mentioned X86_SGX_VEPC, in which case still putting it to arch/x86/Kconfig looks a better fit.
Sean, Paolo,
Do you have comment here?
Re: [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Dave HansenQuoted 10 lines · On Wed, 20 Jan 2021 17:15:35 -0800 Dave Hansen wrote:
Quoted 8 lines · Kai Huang
/*
* Update theSGX_LEPUBKEYHASHMSRs to the values specified by caller. *
* EINITTOKEN is not used in enclave initialization, which requires
* hash of enclave's signer must match values inSGX_LEPUBKEYHASHMSRs
* to make EINIT be successful.
*/
Quoted 3 lines
I'm grumpy, but I hate it.
I'll stop the bike shedding for now, though.
Jarkko and Dave,
I'll change to use below:
/*
* Update the SGX_LEPUBKEYHASH MSRs to the values specified by caller.
* Bare-metal driver requires to update them to hash of enclave's signer
* before EINIT. KVM needs to update them to guest's virtual MSR values
* before doing EINIT from guest.
*/
Please let me know if are not OK with this.
Re: [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Kai HuangQuoted 5 lines · Kai Huang
I actually feel the function name already explains what the function does clearly, therefore I don't think even comment is needed. To be honest I don't know how to rephrase here. Perhaps:
/* Update SGX LEPUBKEYHASH MSRs of the platform. */
Whee! I'm gonna write me a function comment!
/*
* A Launch Enclave (LE) must be signed with a public key
* that matches this SHA256 hash. Usually overwrites Intel's
* default signing key.
*/
So, this isn't a one-liner. *But*, it tells us what "le" means, what "pubkey" means and implies that there need to be 4x64-bits worth of MSR writes to get to a SHA256 hash. It also tells what it's usually doing
here: overwriting Intel's blasted hash.
It sure beats the entirely uncommented for loop that we've got today.
Jarkko Sakkinen
maintainerQuoted 19 lines · Kai Huang
Quoted 17 lines · Jarkko Sakkinen
Can you send a new version that applies:
$ git pw series apply 416463 Applying: x86/cpufeatures: Add SGX1 and SGX2 sub-features Applying: x86/sgx: Remove a warn from sgx_free_epc_page() Applying: x86/sgx: Wipe out EREMOVE from sgx_free_epc_page() Applying: x86/sgx: Add SGX_CHILD_PRESENT hardware error code Applying: x86/sgx: Introduce virtual EPC for use by KVM guests Applying: x86/cpu/intel: Allow SGX virtualization without Launch Control support Applying: x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled error: sha1 information is lacking or useless (arch/x86/kernel/cpu/sgx/main.c). error: could not build fake ancestor hint: Use 'git am --show-current-patch=diff' to see the failed patch Patch failed at 0007 x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".
Quoted 2 lines
Could you let me know which branch should I rebase to? It appears your linux-sgx tree next branch?
I tried my tree first, which was actually out-of-sync, so I rebase it. You can find it from MAINTAINERS failed.
After sending this email I rebased it to tip/x86/sgx, which also failed. That tree failed with a merge conflict.
/Jarkko
Quoted 263 lines
And I don't think it is good to resend for this series at this moment, due to: 1) people have already started to comment; 2) The two patches from you don't have your name on your SoB, so has to fix anyway. It would be more appropriate to rebase to your tree in next version, if required. And I can prepare a separate branch for you (after resolving the merge conflict with
sgx_init()in your tree) at the same time.And Boris, Paolo, Sean, Jarkko, and other maintainers,
I'd like to take this chance to ask: when this series is ready to be merged, what is the properly way to merge? This series has x86 non-sgx (cpufeature,
feat_ctl) and sgx changes, and it obviously has KVM changes too. So instance, who should be the one to take this series? And which tree and branch should I rebase to in next version?Thanks for all your feedback.
Quoted 244 lines
Thanks.
/Jarkko
Quoted 238 lines · Kai Huang
--- Disclaimer ---
These patches were originally written by Sean Christopherson while at Intel. Now that Sean has left Intel, I (Kai) have taken over getting them upstream. This series needs more review before it can be merged. It is being posted publicly and under RFC so Sean and others can review it. Maintainers are safe ignoring it for now.
------------------
Hi all,
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Please help to review this series. Also I'd like to hear what is the proper way to merge this series, since it contains change to both x86/SGX and KVM subsystem. Any feedback is highly appreciated. And please let me know if I forgot to CC anyone, or anyone wants to be removed from CC. Thanks in advance!
This series is based against upstream v5.11-rc3. You can also get the code from upstream branch of kvm-sgx repo on github:
https://github.com/intel/kvm-sgx.git upstream
It also requires Qemu changes to create VM with SGX support. You can find Qemu repo here:
https://github.com/intel/qemu-sgx.git next
Please refer to README.md of above qemu-sgx repo for detail on how to create guest with SGX support. At meantime, for your quick reference you can use below command to create SGX guest:
#qemu-system-x86_64 -smp 4 -m 2G -drive file=<your_vm_image>,if=virtio \ -cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,preallocPlease note that the SGX relevant part is:
-cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,preallocAnd you can change other parameters of your qemu command based on your needs.
=========
Changelog:(Changelog here is for global changes. Please see each patch's changelog for changes made to specific patch.)
v1->v2:- Refined this cover letter by addressing comments from Dave and Jarkko.
- The original patch which introduced new X86_FEATURE_SGX1/SGX2 were replaced
by 3 new patches from Sean, following Boris and Sean's discussion.
[RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
[RFC PATCH v2 18/26] KVM: x86: Add support for reverse CPUID lookup of scattered features [RFC PATCH v2 19/26] KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
- The original patch 1
x86/sgx: Split out adding EPC page to free list to separate helper
was replaced with 2 new patches from Jarkko
[RFC PATCH v2 02/26] x86/sgx: Remove a warn fromsgx_free_epc_page()
[RFC PATCH v2 03/26] x86/sgx: Wipe out EREMOVE fromsgx_free_epc_page()addressing Jarkko's comments.
- Moved modifyingsgx_init()to always initializesgx_virt_epc_init()out of
patch
x86/sgx: Introduce virtual EPC for use by KVM guests
to a separate patch:
[RFC PATCH v2 07/26] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled to address Dave's comment that patch ordering can be improved due to before patch "Allow SGX virtualization without Launch Control support", all SGX, including SGX virtualization, is actually disabled when SGX LC is not
present.KVM part patches are not changed comparing to v1 (except changes due to X86_FEATURE_SGX1/2 patches). For changes to each x86 patch, please see changelog in each indudival patch. If no changelog, then no change was made to it.
=========
KVM SGX virtualization Overview- Virtual EPC
SGX enclave memory is special and is reserved specifically for enclave use. In bare-metal SGX enclaves, the kernel allocates enclave pages, copies data into the pages with privileged instructions, then allows the enclave to start. In this scenario, only initialized pages already assigned to an enclave are mapped to userspace.
In virtualized environments, the hypervisor still needs to do the physical enclave page allocation. The guest kernel is responsible for the data copying (among other things). This means that the job of starting an enclave is now split between hypervisor and guest.
This series introduces a new misc device:
/dev/sgx_virt_epc. This device allows the host to map *uninitialized* enclave memory into userspace, which can then be passed into a guest.While it might be *possible* to start a host-side enclave with
/dev/sgx_enclaveand pass its memory into a guest, it would be wasteful and convoluted.Implement the *raw* EPC allocation in the x86 core-SGX subsystem via
/dev/sgx_virt_epcrather than in KVM. Doing so has two major advantages:- Does not require changes to KVM's uAPI, e.g. EPC gets handled as just another memory backend for guests.
- EPC management is wholly contained in the SGX subsystem, e.g. SGX does not have to export any symbols, changes to reclaim flows don't need to be routed through KVM, SGX's dirty laundry doesn't have to get aired out for the world to see, and so on and so forth.
The virtual EPC pages allocated to guests are currently not reclaimable. Reclaiming EPC page used by enclave requires a special reclaim mechanism separate from normal page reclaim, and that mechanism is not supported for virutal EPC pages. Due to the complications of handling reclaim conflicts between guest and host, reclaiming virtual EPC pages is significantly more complex than basic support for SGX virtualization.
- Support SGX virtualization without SGX Flexible Launch Control
SGX hardware supports two "launch control" modes to limit which enclaves can run. In the "locked" mode, the hardware prevents enclaves from running unless they are blessed by a third party. In the unlocked mode, the kernel is in full control of which enclaves can run. The bare-metal SGX code refuses to launch enclaves unless it is in the unlocked mode.
This
sgx_virt_epcdriver does not have such a restriction. This allows guests which are OK with the locked mode to use SGX, even if the host kernel refuses to.- Support exposing SGX2
Due to the same reason above, SGX2 feature detection is added to core SGX code to allow KVM to expose SGX2 to guest, even currently SGX driver doesn't support SGX2, because SGX2 can work just fine in guest w/o any interaction to host SGX driver.
- Restricit SGX guest access to provisioning key
To grant guest being able to fully use SGX, guest needs to be able to access provisioning key. The provisioning key is sensitive, and accessing to it should be restricted. In bare-metal driver, allowing enclave to access provisioning key is restricted by being able to open
/dev/sgx_provision.Add a new
KVM_CAP_SGX_ATTRIBUTEto KVM uAPI to extend above mechanism to KVM guests as well. When userspace hypervisor creates a new VM, the new cap is only added to VM when userspace hypervisior is able to open/dev/sgx_provision, following the same role as in bare-metal driver. KVM then traps ECREATE from guest, and only allows ECREATE with provisioning key bit to run when guest supportsKVM_CAP_SGX_ATTRIBUTE.Kai Huang (2):
x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
x86/sgx: Add helper to updateSGX_LEPUBKEYHASHnMSRsSean Christopherson (22):
x86/cpufeatures: Add SGX1 and SGX2 sub-features
x86/sgx: AddSGX_CHILD_PRESENThardware error code x86/sgx: Introduce virtual EPC for use by KVM guests x86/cpu/intel: Allow SGX virtualization without Launch Control support x86/sgx: Expose SGX architectural definitions to the kernel x86/sgx: Move ENCLS leaf definitions tosgx_arch.hx86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT) x86/sgx: Addencls_faulted()helper
x86/sgx: Add helpers to expose ECREATE and EINIT to KVM x86/sgx: Move provisioning device creation out of SGX driver
KVM: VMX: Convert vcpu_vmx.exit_reason to a union
KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)
KVM: x86: Define new #PF SGX error code bit
KVM: x86: Add support for reverse CPUID lookup of scattered features
KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
KVM: VMX: Add basic handling of VM-Exit from SGX enclave
KVM: VMX: Frame in ENCLS handler for SGX virtualization
KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions
KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs
KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)
KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC
KVM: x86: Add capability to grant VM access to privileged SGX attribute[email protected] (2):
x86/sgx: Remove a warn fromsgx_free_epc_page()x86/sgx: Wipe out EREMOVE fromsgx_free_epc_page()Documentation/virt/kvm/api.rst | 23 + arch/x86/Kconfig | 12 + arch/x86/include/asm/cpufeatures.h | 2 + arch/x86/include/asm/kvm_host.h | 5 + arch/x86/include/asm/sgx.h | 19 + .../cpu/sgx/arch.h => include/asm/sgx_arch.h} | 20 + arch/x86/include/asm/vmx.h | 1 + arch/x86/include/uapi/asm/vmx.h | 1 + arch/x86/kernel/cpu/cpuid-deps.c | 3 + arch/x86/kernel/cpu/feat_ctl.c | 63 ++- arch/x86/kernel/cpu/scattered.c | 2 + arch/x86/kernel/cpu/sgx/Makefile | 1 + arch/x86/kernel/cpu/sgx/driver.c | 17 - arch/x86/kernel/cpu/sgx/encl.c | 15 +- arch/x86/kernel/cpu/sgx/encls.h | 29 +- arch/x86/kernel/cpu/sgx/ioctl.c | 23 +- arch/x86/kernel/cpu/sgx/main.c | 67 ++- arch/x86/kernel/cpu/sgx/sgx.h | 4 +- arch/x86/kernel/cpu/sgx/virt.c | 316 ++++++++++++ arch/x86/kernel/cpu/sgx/virt.h | 14 + arch/x86/kvm/Makefile | 2 + arch/x86/kvm/cpuid.c | 89 +++- arch/x86/kvm/cpuid.h | 50 +- arch/x86/kvm/vmx/nested.c | 70 ++- arch/x86/kvm/vmx/nested.h | 5 + arch/x86/kvm/vmx/sgx.c | 462 ++++++++++++++++++ arch/x86/kvm/vmx/sgx.h | 34 ++ arch/x86/kvm/vmx/vmcs12.c | 1 + arch/x86/kvm/vmx/vmcs12.h | 4 +- arch/x86/kvm/vmx/vmx.c | 171 +++++-- arch/x86/kvm/vmx/vmx.h | 27 +- arch/x86/kvm/x86.c | 24 + include/uapi/linux/kvm.h | 1 + tools/testing/selftests/sgx/defines.h | 2 +- 34 files changed, 1432 insertions(+), 147 deletions(-)create mode 100644
arch/x86/include/asm/sgx.h
renamearch/x86/{kernel/cpu/sgx/arch.h=>include/asm/sgx_arch.h} (96%) create mode 100644arch/x86/kernel/cpu/sgx/virt.c
create mode 100644arch/x86/kernel/cpu/sgx/virt.h
create mode 100644arch/x86/kvm/vmx/sgx.c
create mode 100644arch/x86/kvm/vmx/sgx.hSignature · 1 line
2.29.2
Re: [RFC PATCH v2 11/26] x86/sgx: Add encls_faulted() helper
In reply to Jarkko SakkinenQuoted 22 lines · On Wed, 20 Jan 2021 14:03:08 +0200 Jarkko Sakkinen wrote:
Quoted 21 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add a helper to extract the fault indicator from an encoded ENCLS return value. SGX virtualization will also need to detect ENCLS faults.
Signed-off-by Sean Christopherson <[email protected]>Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/encls.h | 14 +++++++++++++- arch/x86/kernel/cpu/sgx/ioctl.c | 2 +- 2 files changed, 14 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/sgx/encls.h +0
-0@@ -40,6 +40,18 @@ }while (0 ); \ }
Quoted 10 lines
arch/x86/kernel/cpu/sgx/encls.h line 48
Quoted 6 lines
arch/x86/kernel/cpu/sgx/encls.h +6
-0+/* +* encls_faulted() - Check if an ENCLS leaf faulted given an error code +* @ret the return value of an ENCLS leaf function call +* +* Return: +* %true if @ret indicates a fault, %false otherwise Follow here the style of commenting as in
ioctl.c, for the return value. It has optimal readability both as text, and also when converted to HTML. Seesgx_ioc_enclave_add_pages()for an example.
You mean something like below?
Return:
- %true: @ret indicates a fault.
- %false: @ret indicates no fault.
Jarkko Sakkinen
maintainerQuoted 34 lines · Kai Huang
Quoted 33 lines · On Thu, 21 Jan 2021 03:16:19 +0200 Jarkko Sakkinen wrote:
Quoted 31 lines · Kai Huang
Quoted 20 lines · On Thu, 21 Jan 2021 01:43:41 +0200 Jarkko Sakkinen wrote:
Quoted 19 lines · Kai Huang
Quoted 17 lines · Jarkko Sakkinen
Can you send a new version that applies:
$ git pw series apply 416463 Applying: x86/cpufeatures: Add SGX1 and SGX2 sub-features Applying: x86/sgx: Remove a warn from sgx_free_epc_page() Applying: x86/sgx: Wipe out EREMOVE from sgx_free_epc_page() Applying: x86/sgx: Add SGX_CHILD_PRESENT hardware error code Applying: x86/sgx: Introduce virtual EPC for use by KVM guests Applying: x86/cpu/intel: Allow SGX virtualization without Launch Control support Applying: x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled error: sha1 information is lacking or useless (arch/x86/kernel/cpu/sgx/main.c). error: could not build fake ancestor hint: Use 'git am --show-current-patch=diff' to see the failed patch Patch failed at 0007 x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".Quoted 8 lines
Quoted 2 lines
Could you let me know which branch should I rebase to? It appears your linux-sgx tree next branch?
I tried my tree first, which was actually out-of-sync, so I rebase it. You can find it from
MAINTAINERSfailed.After sending this email I rebased it to tip/x86/sgx, which also failed. That tree failed with a merge conflict.
This series is based on v5.11-rc3, as mentioned in cover letter below:
Quoted 7 lines
Quoted 1 line
Please use tip/x86/sgx as base instead.
When I wrote this series, the tip/x86/sgx was still v5.10-rc4. I felt I should rebase to some newer code, so I chose upstream v5.11-rc3.
I just checked the latest tip/x86/sgx, and it has updated to v5.11-rc3, so yes I will rebase to it for next version.
I think we did a good review round anyway, and I've used this time to learn to use Graphene, i.e. no time wasted :-)
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 06/26] x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Kai HuangQuoted 20 lines · Kai Huang
Quoted 19 lines · On Wed, 20 Jan 2021 15:51:44 -0800 Dave Hansen wrote:
Review
Quoted 11 lines · Kai Huang
Review
Quoted 6 lines
Not a big deal either way. I agree that "virt" can be confusing.
Considering that:
(quoted diff) +2
-0+config X86_SGX_VIRTUALIZATION + depends on ... KVM_INTELIt is already in patch 3: Introduce virtual EPC for use by KVM guests:
(quoted diff) +3
-0+config X86_SGX_VIRTUALIZATION + bool "Software Guard eXtensions (SGX) Virtualization" + depends on X86_SGX && KVM_INTELRight, I'm suggesting that patch 3 should call it:
X86_SGX_KVMinstead of:
Quoted 7 lines
Quoted 1 line
X86_SGX_VIRTUALIZATIONIn case we want to change to
X86_SGX_KVM, is it more reasonable to put it toarch/x86/kvm/Kconfig(maybe change toX86_KVM_SGX)?Jarkko also mentioned
X86_SGX_VEPC, in which case still putting it toarch/x86/Kconfiglooks a better fit.
I disagree with myself on that now :-) I think the other suggestions are better. I'm only pursuing 'vepc' for the device name because it follows the pattern used in the other devices.
Quoted 4 lines
Sean, Paolo,
Do you have comment here?
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Kai HuangQuoted 19 lines · Kai Huang
Quoted 10 lines · On Wed, 20 Jan 2021 17:15:35 -0800 Dave Hansen wrote:
Quoted 8 lines · Kai Huang
/*
* Update theSGX_LEPUBKEYHASHMSRs to the values specified by caller. *
* EINITTOKEN is not used in enclave initialization, which requires
* hash of enclave's signer must match values inSGX_LEPUBKEYHASHMSRs
* to make EINIT be successful.
*/Quoted 3 lines
I'm grumpy, but I hate it.
I'll stop the bike shedding for now, though.
Jarkko and Dave,
I'll change to use below:
Quoted 8 lines
/*
* Update theSGX_LEPUBKEYHASHMSRs to the values specified by caller.
* Bare-metal driver requires to update them to hash of enclave's signer
* before EINIT. KVM needs to update them to guest's virtual MSR values
* before doing EINIT from guest.
*/Please let me know if are not OK with this.
I am.
/Jarkko
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 11/26] x86/sgx: Add encls_faulted() helper
In reply to Kai HuangQuoted 54 lines · Kai Huang
Quoted 53 lines · On Thu, 21 Jan 2021 03:08:18 +0200 Jarkko Sakkinen wrote:
Quoted 34 lines · Kai Huang
Quoted 22 lines · On Wed, 20 Jan 2021 14:03:08 +0200 Jarkko Sakkinen wrote:
Quoted 21 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Add a helper to extract the fault indicator from an encoded ENCLS return value. SGX virtualization will also need to detect ENCLS faults.
Signed-off-by Sean Christopherson <[email protected]>Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/encls.h | 14 +++++++++++++- arch/x86/kernel/cpu/sgx/ioctl.c | 2 +- 2 files changed, 14 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/sgx/encls.h +0
-0@@ -40,6 +40,18 @@ }while (0 ); \ }Quoted 10 lines
arch/x86/kernel/cpu/sgx/encls.h line 48
Quoted 6 lines
arch/x86/kernel/cpu/sgx/encls.h +6
-0+/* +* encls_faulted() - Check if an ENCLS leaf faulted given an error code +* @ret the return value of an ENCLS leaf function call +* +* Return: +* %true if @ret indicates a fault, %false otherwise Follow here the style of commenting as in
ioctl.c, for the return value. It has optimal readability both as text, and also when converted to HTML. Seesgx_ioc_enclave_add_pages()for an example.Quoted 5 lines
You mean something like below?
Return:
- %true: @ret indicates a fault.
- %false: @ret indicates no fault.Yeah, with '\t' indentation. I'd remove also '%'. Also '@ret' is redudant.
To put this all together:
* Return:
* - true: ENCLS leaf faulted.
* - false: Otherwise.I tried various ways and this was the best way to document return values that i've found. It's easy to read as plain text, and also has the benefit that return values get nicely lined up in htmldocs.
Quoted 8 lines
Quoted 6 lines
I've been even considering a patch for
https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt
The only advice given ATM is: "Take a look around the source tree for examples."
OK. Will change to yours. Thanks for the info.
There's also two examples of patterns that *do not* work in that file :-)
I think I'll post a patch for that file.
/Jarkko
Sean Christopherson
maintainerRe: [RFC PATCH v2 15/26] KVM: VMX: Convert vcpu_vmx.exit_reason to a union
In reply to Jarkko SakkinenQuoted 36 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 33 lines · Jarkko Sakkinen
Quoted 11 lines · Kai Huang
--- arch/x86/kvm/vmx/nested.c | 42 +++++++++++++++--------- arch/x86/kvm/vmx/vmx.c | 68 ++++++++++++++++++++------------------- arch/x86/kvm/vmx/vmx.h | 25 +++++++++++++- 3 files changed, 86 insertions(+), 49 deletions(-)arch/x86/kvm/vmx/nested.c +0
-0arch/x86/kvm/vmx/nested.c enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, · line 3318
Quoted 9 lines
arch/x86/kvm/vmx/nested.c +5
-1@@ -3311,7 +3311,11 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);enum vm_entry_failure_code entry_failure_code;bool evaluate_pending_interrupts; -u32 exit_reason, failed_index; +u32 failed_index; +union vmx_exit_reason exit_reason = { + .basic =-1 , + .failed_vmentry =1 , + };Instead, put this declaration to the correct place, following the reverse christmas tree ordering:
union vmx_exit_reason exit_reason = {};And after declarations:
exit_reason.basic = -1; exit_reason.failed_vmentry = 1;More pleasing for the eye.
Quoted 8 lines
Quoted 4 lines
I disagree (obviously, since I wrote the patch). Initializing the fields to their respective values is a critical, but subtle, aspect of this code. Making the code stand out via explicit initialization is a good thing, and we really don't want any possibility of code touching
exit_reasonbefore it is initialized.The struct does get initialized to zero, and fields get initialized to their respective values. What is the critical aspect that is gone? I'm not following.
Setting exit_reason.failed_vmentry from time zero. This code should never synthesize a nested VM-Exit with failed_vmentry=0. There have been bugs around the exit_reason in this code in the past, I strongly prefer to not have any window where exit_reason has the "wrong" value.
Jarkko Sakkinen
maintainerRe: [RFC PATCH v2 15/26] KVM: VMX: Convert vcpu_vmx.exit_reason to a union
Quoted 42 lines · Sean Christopherson
Quoted 41 lines · Jarkko Sakkinen
Quoted 39 lines · Sean Christopherson
Quoted 25 lines · Jarkko Sakkinen
Quoted 11 lines · Kai Huang
--- arch/x86/kvm/vmx/nested.c | 42 +++++++++++++++--------- arch/x86/kvm/vmx/vmx.c | 68 ++++++++++++++++++++------------------- arch/x86/kvm/vmx/vmx.h | 25 +++++++++++++- 3 files changed, 86 insertions(+), 49 deletions(-)arch/x86/kvm/vmx/nested.c +0
-0arch/x86/kvm/vmx/nested.c enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, · line 3318
Quoted 9 lines
arch/x86/kvm/vmx/nested.c +5
-1@@ -3311,7 +3311,11 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);enum vm_entry_failure_code entry_failure_code;bool evaluate_pending_interrupts; -u32 exit_reason, failed_index; +u32 failed_index; +union vmx_exit_reason exit_reason = { + .basic =-1 , + .failed_vmentry =1 , + };Instead, put this declaration to the correct place, following the reverse christmas tree ordering:
Quoted 8 lines
union vmx_exit_reason exit_reason = {};And after declarations:
exit_reason.basic = -1; exit_reason.failed_vmentry = 1;More pleasing for the eye.
I disagree (obviously, since I wrote the patch). Initializing the fields to their respective values is a critical, but subtle, aspect of this code. Making the code stand out via explicit initialization is a good thing, and we really don't want any possibility of code touching
exit_reasonbefore it is initialized.
Quoted 8 lines
Quoted 3 lines
The struct does get initialized to zero, and fields get initialized to their respective values. What is the critical aspect that is gone? I'm not following.
Setting exit_reason.failed_vmentry from time zero. This code should never synthesize a nested VM-Exit with failed_vmentry=0. There have been bugs around the
exit_reasonin this code in the past, I strongly prefer to not have any window whereexit_reasonhas the "wrong" value.
OK, I see. Thanks for the explanation.
/Jarkko