KVM SGX virtualization support
Subsystems (6)
v2 cover KVM SGX virtualization support
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
This series is based against latest upstream kernel master branch.
You can also get the code from upstream branch of kvm-sgx repo on github:
https://github.com/intel/kvm-sgx.git upstream
It also requires Qemu changes to create VM with SGX support. You can find Qemu repo here:
https://github.com/intel/qemu-sgx.git upstream
Please refer to README.md of above qemu-sgx repo for detail on how to create guest with SGX support. At meantime, for your quick reference you can use below command to create SGX guest:
#qemu-system-x86_64 -smp 4 -m 2G -drive file=<your_vm_image>,if=virtio \ -cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,prealloc
Please note that the SGX relevant part is:
-cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,prealloc
And you can change other parameters of your qemu command based on your needs.
=========
Changelog:
(Changelog here is for global changes. Please see each patch's changelog for
changes made to specific patch.)
v1->v2:
- No big change in design, structural of patch series, etc.
- Addressed Boris's comments regarding to suppressing both SGX1 and SGX2 in /proc/cpuinfo, and improvement in feat_ctl.c when enabling SGX (patch 2 and 6).
- Addressed Sean's comments for both x86 part patches and KVM patches (patch 3, 5, 9, 12, 19, 21).
- Addressed Dave's comments in RFC v6 series (patch 13).
RFC->v1:
- Refined patch (x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()) to print error msg that EPC page is leaked when EREMOVE failed, requested by Dave.
- Changelog history of all RFC series is removed in both this cover letter and each individual patch, since majority of x86 part patches already got Acked-by from Dave and Jarkko. And the changelogs are not quite useful from my perspective.
=========
KVM SGX virtualization Overview
- Virtual EPC
SGX enclave memory is special and is reserved specifically for enclave use. In bare-metal SGX enclaves, the kernel allocates enclave pages, copies data into the pages with privileged instructions, then allows the enclave to start. In this scenario, only initialized pages already assigned to an enclave are mapped to userspace.
In virtualized environments, the hypervisor still needs to do the physical enclave page allocation. The guest kernel is responsible for the data copying (among other things). This means that the job of starting an enclave is now split between hypervisor and guest.
This series introduces a new misc device: /dev/sgx_vepc. This device allows the host to map *uninitialized* enclave memory into userspace, which can then be passed into a guest.
While it might be *possible* to start a host-side enclave with /dev/sgx_enclave and pass its memory into a guest, it would be wasteful and convoluted.
Implement the *raw* EPC allocation in the x86 core-SGX subsystem via /dev/sgx_vepc rather than in KVM. Doing so has two major advantages:
- Does not require changes to KVM's uAPI, e.g. EPC gets handled as just another memory backend for guests.
- EPC management is wholly contained in the SGX subsystem, e.g. SGX does not have to export any symbols, changes to reclaim flows don't need to be routed through KVM, SGX's dirty laundry doesn't have to get aired out for the world to see, and so on and so forth.
The virtual EPC pages allocated to guests are currently not reclaimable. Reclaiming EPC page used by enclave requires a special reclaim mechanism separate from normal page reclaim, and that mechanism is not supported for virutal EPC pages. Due to the complications of handling reclaim conflicts between guest and host, reclaiming virtual EPC pages is significantly more complex than basic support for SGX virtualization.
- Support SGX virtualization without SGX Flexible Launch Control
SGX hardware supports two "launch control" modes to limit which enclaves can run. In the "locked" mode, the hardware prevents enclaves from running unless they are blessed by a third party. In the unlocked mode, the kernel is in full control of which enclaves can run. The bare-metal SGX code refuses to launch enclaves unless it is in the unlocked mode.
This sgx_virt_epc driver does not have such a restriction. This allows guests which are OK with the locked mode to use SGX, even if the host kernel refuses to.
- Support exposing SGX2
Due to the same reason above, SGX2 feature detection is added to core SGX code to allow KVM to expose SGX2 to guest, even currently SGX driver doesn't support SGX2, because SGX2 can work just fine in guest w/o any interaction to host SGX driver.
- Restricit SGX guest access to provisioning key
To grant guest being able to fully use SGX, guest needs to be able to access provisioning key. The provisioning key is sensitive, and accessing to it should be restricted. In bare-metal driver, allowing enclave to access provisioning key is restricted by being able to open /dev/sgx_provision.
Add a new KVM_CAP_SGX_ATTRIBUTE to KVM uAPI to extend above mechanism to KVM guests as well. When userspace hypervisor creates a new VM, the new cap is only added to VM when userspace hypervisior is able to open /dev/sgx_provision, following the same role as in bare-metal driver. KVM then traps ECREATE from guest, and only allows ECREATE with provisioning key bit to run when guest supports KVM_CAP_SGX_ATTRIBUTE.
Jarkko Sakkinen (1):
x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
Kai Huang (3):
x86/cpufeatures: Make SGX_LC feature bit depend on SGX bit x86/sgx: Initialize virtual EPC driver even when SGX driver is
disabled
x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
Sean Christopherson (21):
x86/cpufeatures: Add SGX1 and SGX2 sub-features
x86/sgx: Add SGX_CHILD_PRESENT hardware error code x86/sgx: Introduce virtual EPC for use by KVM guests x86/cpu/intel: Allow SGX virtualization without Launch Control support x86/sgx: Expose SGX architectural definitions to the kernel x86/sgx: Move ENCLS leaf definitions to sgx.h
x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT) x86/sgx: Add encls_faulted() helper
x86/sgx: Add helpers to expose ECREATE and EINIT to KVM x86/sgx: Move provisioning device creation out of SGX driver
KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)
KVM: x86: Define new #PF SGX error code bit
KVM: x86: Add support for reverse CPUID lookup of scattered features
KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
KVM: VMX: Add basic handling of VM-Exit from SGX enclave
KVM: VMX: Frame in ENCLS handler for SGX virtualization
KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions
KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs
KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)
KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC
KVM: x86: Add capability to grant VM access to privileged SGX
attribute
Documentation/virt/kvm/api.rst | 23 +
arch/x86/Kconfig | 12 +
arch/x86/include/asm/cpufeatures.h | 2 +
arch/x86/include/asm/kvm_host.h | 5 +
.../cpu/sgx/arch.h => include/asm/sgx.h} | 50 +-
arch/x86/include/asm/vmx.h | 1 +
arch/x86/include/uapi/asm/vmx.h | 1 +
arch/x86/kernel/cpu/cpuid-deps.c | 3 +
arch/x86/kernel/cpu/feat_ctl.c | 71 ++-
arch/x86/kernel/cpu/scattered.c | 2 +
arch/x86/kernel/cpu/sgx/Makefile | 1 +
arch/x86/kernel/cpu/sgx/driver.c | 17 -
arch/x86/kernel/cpu/sgx/encl.c | 29 +-
arch/x86/kernel/cpu/sgx/encls.h | 30 +-
arch/x86/kernel/cpu/sgx/ioctl.c | 23 +-
arch/x86/kernel/cpu/sgx/main.c | 94 +++-
arch/x86/kernel/cpu/sgx/sgx.h | 13 +-
arch/x86/kernel/cpu/sgx/virt.c | 370 ++++++++++++++
arch/x86/kvm/Makefile | 2 +
arch/x86/kvm/cpuid.c | 89 +++-
arch/x86/kvm/cpuid.h | 50 +-
arch/x86/kvm/vmx/nested.c | 28 +-
arch/x86/kvm/vmx/nested.h | 5 +
arch/x86/kvm/vmx/sgx.c | 481 ++++++++++++++++++
arch/x86/kvm/vmx/sgx.h | 34 ++
arch/x86/kvm/vmx/vmcs12.c | 1 +
arch/x86/kvm/vmx/vmcs12.h | 4 +-
arch/x86/kvm/vmx/vmx.c | 109 +++-
arch/x86/kvm/vmx/vmx.h | 2 +
arch/x86/kvm/x86.c | 23 +
include/uapi/linux/kvm.h | 1 +
tools/testing/selftests/sgx/defines.h | 2 +-
32 files changed, 1460 insertions(+), 118 deletions(-)
rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx.h} (89%)
create mode 100644 arch/x86/kernel/cpu/sgx/virt.c
create mode 100644 arch/x86/kvm/vmx/sgx.c
create mode 100644 arch/x86/kvm/vmx/sgx.hSignature · 1 line
2.29.2
PATCH v2 1/25 x86/cpufeatures: Make SGX_LC feature bit depend on SGX bit
Move SGX_LC feature bit to CPUID dependency table to make clearing all SGX feature bits easier. Also remove clear_sgx_caps() since it is just a wrapper of setup_clear_cpu_cap(X86_FEATURE_SGX) now.
--- arch/x86/kernel/cpu/cpuid-deps.c | 1 + arch/x86/kernel/cpu/feat_ctl.c | 12 +++--------- 2 files changed, 4 insertions(+), 9 deletions(-)
arch/x86/kernel/cpu/cpuid-deps.c +1 -0
@@ -72,6 +72,7 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_AVX512_FP16, X86_FEATURE_AVX512BW }, { X86_FEATURE_ENQCMD, X86_FEATURE_XSAVES }, { X86_FEATURE_PER_THREAD_MBA, X86_FEATURE_MBA }, + { X86_FEATURE_SGX_LC, X86_FEATURE_SGX }, {} };
arch/x86/kernel/cpu/feat_ctl.c +3 -9
@@ -93,15 +93,9 @@ static void init_vmx_capabilities(struct cpuinfo_x86 *c) }#endif /* CONFIG_X86_VMX_FEATURE_NAMES */ -static void clear_sgx_caps (void ) -{ -setup_clear_cpu_cap (X86_FEATURE_SGX ); -setup_clear_cpu_cap (X86_FEATURE_SGX_LC ); -} -static int __init nosgx (char *str) { -clear_sgx_caps (); +setup_clear_cpu_cap (X86_FEATURE_SGX );return 0 ; }
@@ -116,7 +110,7 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c)if (rdmsrl_safe (MSR_IA32_FEAT_CTL , &msr)) {clear_cpu_cap (c,X86_FEATURE_VMX ); -clear_sgx_caps (); +clear_cpu_cap (c,X86_FEATURE_SGX );return ; }
@@ -177,6 +171,6 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c) !(msr & FEAT_CTL_SGX_LC_ENABLED) || !enable_sgx) {if (enable_sgx)pr_err_once ("SGX disabled by BIOS\n" ); -clear_sgx_caps (); +clear_cpu_cap (c,X86_FEATURE_SGX ); } }
Signature · 1 line
2.29.2
PATCH v2 2/25 x86/cpufeatures: Add SGX1 and SGX2 sub-features
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add SGX1 and SGX2 feature flags, via CPUID.0x12.0x0.EAX, as scattered features, since adding a new leaf for only two bits would be wasteful. As part of virtualizing SGX, KVM will expose the SGX CPUID leafs to its guest, and to do so correctly needs to query hardware and kernel support for SGX1 and SGX2.
Suppress both SGX1 and SGX2 from /proc/cpuinfo. SGX1 basically means SGX, and for SGX2 there is no concrete use case of using it in /proc/cpuinfo.
---v1->v2:
- Changed to hide both SGX1 and SGX2 from /proc/cpuinfo, since no concrete use case, per Boris.
- Refined commit msg to explain why to hide SGX1 and SGX2 in /proc/cpuinfo.
--- arch/x86/include/asm/cpufeatures.h | 2 ++ arch/x86/kernel/cpu/cpuid-deps.c | 2 ++ arch/x86/kernel/cpu/scattered.c | 2 ++ 3 files changed, 6 insertions(+)
arch/x86/include/asm/cpufeatures.h +2 -0
@@ -290,6 +290,8 @@#define X86_FEATURE_FENCE_SWAPGS_KERNEL (11*32+ 5)/* "" LFENCE in kernel entry SWAPGS path */ #define X86_FEATURE_SPLIT_LOCK_DETECT (11*32+ 6)/* #AC for split lock */ #define X86_FEATURE_PER_THREAD_MBA (11*32+ 7)/* "" Per-thread Memory Bandwidth Allocation */ +#define X86_FEATURE_SGX1 (11*32+ 8)/* "" Basic SGX */ +#define X86_FEATURE_SGX2 (11*32+ 9)/* "" SGX Enclave Dynamic Memory Management (EDMM) */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4)/* AVX VNNI instructions */
arch/x86/kernel/cpu/cpuid-deps.c +2 -0
@@ -73,6 +73,8 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_ENQCMD, X86_FEATURE_XSAVES }, { X86_FEATURE_PER_THREAD_MBA, X86_FEATURE_MBA }, { X86_FEATURE_SGX_LC, X86_FEATURE_SGX }, + { X86_FEATURE_SGX1, X86_FEATURE_SGX }, + { X86_FEATURE_SGX2, X86_FEATURE_SGX1 }, {} };
arch/x86/kernel/cpu/scattered.c +2 -0
@@ -36,6 +36,8 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_CDP_L2, CPUID_ECX, 2, 0x00000010, 2 }, { X86_FEATURE_MBA, CPUID_EBX, 3, 0x00000010, 0 }, { X86_FEATURE_PER_THREAD_MBA, CPUID_ECX, 0, 0x00000010, 3 }, + { X86_FEATURE_SGX1, CPUID_EAX, 0, 0x00000012, 0 }, + { X86_FEATURE_SGX2, CPUID_EAX, 1, 0x00000012, 0 }, { X86_FEATURE_HW_PSTATE, CPUID_EDX, 7, 0x80000007, 0 }, { X86_FEATURE_CPB, CPUID_EDX, 9, 0x80000007, 0 }, { X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 },
Signature · 1 line
2.29.2
PATCH v2 3/25 x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Kai HuangFrom: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original sgx_free_epc_page() as sgx_encl_free_epc_page() to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.
---v1->v2:
- Merge original WARN() and pr_err_once() into one single WARN(), suggested by Sean.
--- arch/x86/kernel/cpu/sgx/encl.c | 27 ++++++++++++++++++++++++--- arch/x86/kernel/cpu/sgx/main.c | 12 ++++-------- 2 files changed, 28 insertions(+), 11 deletions(-)
arch/x86/kernel/cpu/sgx/encl.c +24 -3
@@ -381,6 +381,27 @@ const struct vm_operations_struct sgx_vm_ops = { .access = sgx_vma_access, }; +static void sgx_encl_free_epc_page (struct sgx_epc_page *epc_page) +{ +int ret; + +WARN_ON_ONCE (epc_page->flags &SGX_EPC_PAGE_RECLAIMER_TRACKED ); + +/* +* Give a message to remind EPC page is leaked when EREMOVE fails, +* and requires machine reboot to get leaked pages back. This can +* be improved in future by adding stats of leaked pages, etc. +*/ +#define EREMOVE_ERROR_MESSAGE \ + "EREMOVE returned %d (0x%x). EPC page leaked. Reboot required to retrieve leaked pages." + ret =__eremove (sgx_get_epc_virt_addr (epc_page)); +if (WARN_ONCE (ret,EREMOVE_ERROR_MESSAGE , ret, ret)) +return ; +#undef EREMOVE_ERROR_MESSAGE + +sgx_free_epc_page (epc_page); +} +/** * sgx_encl_release - Destroy an enclave instance * @kref: address of a kref inside &sgx_encl
@@ -404,7 +425,7 @@ void sgx_encl_release(struct kref *ref)if (sgx_unmark_page_reclaimable (entry->epc_page))continue ; -sgx_free_epc_page (entry->epc_page); +sgx_encl_free_epc_page (entry->epc_page); encl->secs_child_cnt--; entry->epc_page =NULL ; }
@@ -415,7 +436,7 @@ void sgx_encl_release(struct kref *ref)xa_destroy (&encl->page_array);if (!encl->secs_child_cnt && encl->secs.epc_page) { -sgx_free_epc_page (encl->secs.epc_page); +sgx_encl_free_epc_page (entry->epc_page); encl->secs.epc_page =NULL ; }
@@ -423,7 +444,7 @@ void sgx_encl_release(struct kref *ref) va_page = list_first_entry(&encl->va_pages, struct sgx_va_page, list);list_del (&va_page->list); -sgx_free_epc_page (va_page->epc_page); +sgx_encl_free_epc_page (entry->epc_page);kfree (va_page); }
arch/x86/kernel/cpu/sgx/main.c +4 -8
@@ -598,18 +598,14 @@ struct sgx_epc_page *sgx_alloc_epc_page(void *owner, bool reclaim)* sgx_free_epc_page() - Free an EPC page * @page: an EPC page * -* Call EREMOVE for an EPC page and insert it back to the list of free pages. +* Put the EPC page back to the list of free pages. It's the caller's +* responsibility to make sure that the page is in uninitialized state. In other +* words, do EREMOVE, EWB or whatever operation is necessary before calling +* this function. */ void sgx_free_epc_page (struct sgx_epc_page *page) {struct sgx_epc_section *section = &sgx_epc_sections[page->section]; -int ret; - -WARN_ON_ONCE (page->flags &SGX_EPC_PAGE_RECLAIMER_TRACKED ); - - ret =__eremove (sgx_get_epc_virt_addr (page)); -if (WARN_ONCE (ret,"EREMOVE returned %d (0x%x)" , ret, ret)) -return ;spin_lock (§ion->lock);list_add_tail (&page->list, §ion->page_list);
Signature · 1 line
2.29.2
PATCH v2 4/25 x86/sgx: Add SGX_CHILD_PRESENT hardware error code
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
SGX driver can accurately track how enclave pages are used. This enables SECS to be specifically targeted and EREMOVE'd only after all child pages have been EREMOVE'd. This ensures that SGX driver will never encounter SGX_CHILD_PRESENT in normal operation.
Virtual EPC is different. The host does not track how EPC pages are used by the guest, so it cannot guarantee EREMOVE success. It might, for instance, encounter a SECS with a non-zero child count.
Add a definition of SGX_CHILD_PRESENT. It will be used exclusively by the SGX virtualization driver to handle recoverable EREMOVE errors when saniziting EPC pages after they are freed.
--- arch/x86/kernel/cpu/sgx/arch.h | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kernel/cpu/sgx/arch.h +2 -0
@@ -26,12 +26,14 @@* enum sgx_return_code - The return code type for ENCLS, ENCLU and ENCLV * %SGX_NOT_TRACKED: Previous ETRACK's shootdown sequence has not * been completed yet. +* %SGX_CHILD_PRESENT SECS has child pages present in the EPC. * %SGX_INVALID_EINITTOKEN: EINITTOKEN is invalid and enclave signer's * public key does not match IA32_SGXLEPUBKEYHASH. * %SGX_UNMASKED_EVENT: An unmasked event, e.g. INTR, was received */ enum sgx_return_code {SGX_NOT_TRACKED =11 , +SGX_CHILD_PRESENT =13 ,SGX_INVALID_EINITTOKEN =16 ,SGX_UNMASKED_EVENT =128 , };
Signature · 1 line
2.29.2
PATCH v2 5/25 x86/sgx: Introduce virtual EPC for use by KVM guests
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add a misc device /dev/sgx_vepc to allow userspace to allocate "raw" EPC without an associated enclave. The intended and only known use case for raw EPC allocation is to expose EPC to a KVM guest, hence the 'vepc' moniker, virt.{c,h} files and X86_SGX_KVM Kconfig.
SGX driver uses misc device /dev/sgx_enclave to support userspace to create enclave. Each file descriptor from opening /dev/sgx_enclave represents an enclave. Unlike SGX driver, KVM doesn't control how guest uses EPC, therefore EPC allocated to KVM guest is not associated to an enclave, and /dev/sgx_enclave is not suitable for allocating EPC for KVM guest.
Having separate device nodes for SGX driver and KVM virtual EPC also allows separate permission control for running host SGX enclaves and KVM SGX guests.
To use /dev/sgx_vepc to allocate a virtual EPC instance with particular size, the userspace hypervisor opens /dev/sgx_vepc, and uses mmap() with the intended size to get an address range of virtual EPC. Then it may use the address range to create one KVM memory slot as virtual EPC for guest.
Implement the "raw" EPC allocation in the x86 core-SGX subsystem via /dev/sgx_vepc rather than in KVM. Doing so has two major advantages:
- Does not require changes to KVM's uAPI, e.g. EPC gets handled as just another memory backend for guests.
- EPC management is wholly contained in the SGX subsystem, e.g. SGX does not have to export any symbols, changes to reclaim flows don't need to be routed through KVM, SGX's dirty laundry doesn't have to get aired out for the world to see, and so on and so forth.
The virtual EPC pages allocated to guests are currently not reclaimable. Reclaiming EPC page used by enclave requires a special reclaim mechanism separate from normal page reclaim, and that mechanism is not supported for virutal EPC pages. Due to the complications of handling reclaim conflicts between guest and host, reclaiming virtual EPC pages is significantly more complex than basic support for SGX virtualization.
---v1->v2:
- Addressed one typo in comments for last round of zapping zombie_secs_pages, pointed out by Sean.
--- arch/x86/Kconfig | 12 ++ arch/x86/kernel/cpu/sgx/Makefile | 1 + arch/x86/kernel/cpu/sgx/sgx.h | 9 ++ arch/x86/kernel/cpu/sgx/virt.c | 260 +++++++++++++++++++++++++++++++ 4 files changed, 282 insertions(+) create mode 100644 arch/x86/kernel/cpu/sgx/virt.c
arch/x86/Kconfig +12 -0
@@ -1941,6 +1941,18 @@ config X86_SGX If unsure, say N. +config X86_SGX_KVM + bool "Software Guard eXtensions (SGX) Virtualization" + depends on X86_SGX && KVM_INTEL + help + + Enables KVM guests to create SGX enclaves. + + This includes support to expose "raw" unreclaimable enclave memory to + guests via a device node, e.g. /dev/sgx_vepc. + + If unsure, say N. + config EFI bool "EFI runtime service support" depends on ACPI
arch/x86/kernel/cpu/sgx/Makefile +1 -0
@@ -3,3 +3,4 @@ obj-y += \ encl.o \ ioctl.o \ main.o +obj-$(CONFIG_X86_SGX_KVM) += virt.o
arch/x86/kernel/cpu/sgx/sgx.h +9 -0
@@ -83,4 +83,13 @@ void sgx_mark_page_reclaimable(struct sgx_epc_page *page);int sgx_unmark_page_reclaimable (struct sgx_epc_page *page);struct sgx_epc_page *sgx_alloc_epc_page (void *owner,bool reclaim); +#ifdef CONFIG_X86_SGX_KVM +int __init sgx_vepc_init (void ); +#else +static inline int __init sgx_vepc_init (void ) +{ +return -ENODEV ; +} +#endif +#endif /* _X86_SGX_H */
arch/x86/kernel/cpu/sgx/virt.c new file +260 -0
260 lines not on this page · raw message
Signature · 1 line
2.29.2
PATCH v2 6/25 x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
The kernel will currently disable all SGX support if the hardware does not support launch control. Make it more permissive to allow SGX virtualization on systems without Launch Control support. This will allow KVM to expose SGX to guests that have less-strict requirements on the availability of flexible launch control.
Improve error message to distinguish between three cases. There are two cases where SGX support is completely disabled:
1) SGX has been disabled completely by the BIOS
2) SGX LC is locked by the BIOS. Bare-metal support is disabled because of LC unavailability. SGX virtualization is unavailable (because of Kconfig).
One where it is partially available:
3) SGX LC is locked by the BIOS. Bare-metal support is disabled because of LC unavailability. SGX virtualization is supported.
---v1->v2:
- Changed order of declaration of variables so that they are in reverse-christmas tree pattern.
- Removed enable_sgx_any, suggested by Boris.
--- arch/x86/kernel/cpu/feat_ctl.c | 59 +++++++++++++++++++++++++--------- 1 file changed, 44 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/feat_ctl.c +44 -15
@@ -104,8 +104,9 @@ early_param("nosgx", nosgx);void init_ia32_feat_ctl (struct cpuinfo_x86 *c) { +bool enable_sgx_kvm = false, enable_sgx_driver = false;bool tboot =tboot_enabled (); -bool enable_sgx; +bool enable_vmx;u64 msr;if (rdmsrl_safe (MSR_IA32_FEAT_CTL , &msr)) {
@@ -114,13 +115,19 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c)return ; } -/* -* Enable SGX if and only if the kernel supports SGX and Launch Control -* is supported, i.e. disable SGX if the LE hash MSRs can't be written. -*/ - enable_sgx =cpu_has (c,X86_FEATURE_SGX ) && -cpu_has (c,X86_FEATURE_SGX_LC ) && -IS_ENABLED (CONFIG_X86_SGX ); + enable_vmx =cpu_has (c,X86_FEATURE_VMX ) && +IS_ENABLED (CONFIG_KVM_INTEL ); + +if (cpu_has (c,X86_FEATURE_SGX ) &&IS_ENABLED (CONFIG_X86_SGX )) { +/* +* Separate out SGX driver enabling from KVM. This allows KVM +* guests to use SGX even if the kernel SGX driver refuses to +* use it. This happens if flexible Faunch Control is not +* available. +*/ + enable_sgx_driver =cpu_has (c,X86_FEATURE_SGX_LC ); + enable_sgx_kvm = enable_vmx &&IS_ENABLED (CONFIG_X86_SGX_KVM ); + }if (msr &FEAT_CTL_LOCKED ) goto update_caps;
@@ -136,15 +143,18 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c)* i.e. KVM is enabled, to avoid unnecessarily adding an attack vector * for the kernel, e.g. using VMX to hide malicious code. */ -if (cpu_has (c,X86_FEATURE_VMX ) &&IS_ENABLED (CONFIG_KVM_INTEL )) { +if (enable_vmx) { msr |=FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX ;if (tboot) msr |=FEAT_CTL_VMX_ENABLED_INSIDE_SMX ; } -if (enable_sgx) - msr |=FEAT_CTL_SGX_ENABLED |FEAT_CTL_SGX_LC_ENABLED ; +if (enable_sgx_kvm || enable_sgx_driver) { + msr |=FEAT_CTL_SGX_ENABLED ; +if (enable_sgx_driver) + msr |=FEAT_CTL_SGX_LC_ENABLED ; + }wrmsrl (MSR_IA32_FEAT_CTL , msr);
@@ -167,10 +177,29 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c) } update_sgx: -if (!(msr &FEAT_CTL_SGX_ENABLED ) || - !(msr &FEAT_CTL_SGX_LC_ENABLED ) || !enable_sgx) { -if (enable_sgx) -pr_err_once ("SGX disabled by BIOS\n" ); +if (!(msr &FEAT_CTL_SGX_ENABLED )) { +if (enable_sgx_kvm || enable_sgx_driver) +pr_err_once ("SGX disabled by BIOS.\n" );clear_cpu_cap (c,X86_FEATURE_SGX ); +return ; + } + +/* +* VMX feature bit may be cleared due to being disabled in BIOS, +* in which case SGX virtualization cannot be supported either. +*/ +if (!cpu_has (c,X86_FEATURE_VMX ) && enable_sgx_kvm) { +pr_err_once ("SGX virtualization disabled due to lack of VMX.\n" ); + enable_sgx_kvm =0 ; + } + +if (!(msr &FEAT_CTL_SGX_LC_ENABLED ) && enable_sgx_driver) { +if (!enable_sgx_kvm) { +pr_err_once ("SGX Launch Control is locked. Disable SGX.\n" ); +clear_cpu_cap (c,X86_FEATURE_SGX ); + }else { +pr_err_once ("SGX Launch Control is locked. Support SGX virtualization only.\n" ); +clear_cpu_cap (c,X86_FEATURE_SGX_LC ); + } } }
Signature · 1 line
2.29.2
PATCH v2 7/25 x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Kai HuangModify sgx_init() to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.
--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/main.c +9 -1
@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; } - ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();if (ret) goto err_kthread;
Signature · 1 line
2.29.2
PATCH v2 8/25 x86/sgx: Expose SGX architectural definitions to the kernel
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Expose SGX architectural structures, as KVM will use many of the architectural constants and structs to virtualize SGX.
Name the new header file as asm/sgx.h, rather than asm/sgx_arch.h, to have single header to provide SGX facilities to share with other kernel componments.
---
.../cpu/sgx/arch.h => include/asm/sgx.h} | 20 ++++++++++++++-----
arch/x86/kernel/cpu/sgx/encl.c | 2 +-
arch/x86/kernel/cpu/sgx/sgx.h | 2 +-
tools/testing/selftests/sgx/defines.h | 2 +-
4 files changed, 18 insertions(+), 8 deletions(-)
rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx.h} (95%)arch/x86/include/asm/sgx.h renamed from arch/x86/kernel/cpu/sgx/arch.h +15 -5
@@ -2,15 +2,20 @@/** * Copyright(c) 2016-20 Intel Corporation. * -* Contains data structures defined by the SGX architecture. Data structures -* defined by the Linux software stack should not be placed here. +* Intel Software Guard Extensions (SGX) support. */ -#ifndef _ASM_X86_SGX_ARCH_H -#define _ASM_X86_SGX_ARCH_H +#ifndef _ASM_X86_SGX_H +#define _ASM_X86_SGX_H#include <linux/bits.h> #include <linux/types.h> +/* +* This file contains both data structures defined by SGX architecture and Linux +* defined software data structures and functions. The two should not be mixed +* together for better readibility. The architectural definitions come first. +*/ +/* The SGX specific CPUID function. */ #define SGX_CPUID 0x12/* EPC enumeration. */
@@ -337,4 +342,9 @@ struct sgx_sigstruct {#define SGX_LAUNCH_TOKEN_SIZE 304 -#endif /* _ASM_X86_SGX_ARCH_H */ +/* +* Do not put any hardware-defined SGX structure representations below this +* line! +*/ + +#endif /* _ASM_X86_SGX_H */
arch/x86/kernel/cpu/sgx/encl.c +1 -1
@@ -7,7 +7,7 @@#include <linux/shmem_fs.h> #include <linux/suspend.h> #include <linux/sched/mm.h> -#include "arch.h" +#include <asm/sgx.h> #include "encl.h" #include "encls.h" #include "sgx.h"
arch/x86/kernel/cpu/sgx/sgx.h +1 -1
@@ -8,7 +8,7 @@#include <linux/rwsem.h> #include <linux/types.h> #include <asm/asm.h> -#include "arch.h" +#include <asm/sgx.h> #undef pr_fmt#define pr_fmt (fmt) "sgx: " fmt
tools/testing/selftests/sgx/defines.h +1 -1
@@ -14,7 +14,7 @@#define __aligned (x) __attribute__((__aligned__(x)))#define __packed __attribute__((packed)) -#include "../../../../arch/x86/kernel/cpu/sgx/arch.h" +#include "../../../../arch/x86/include/asm/sgx.h" #include "../../../../arch/x86/include/asm/enclu.h" #include "../../../../arch/x86/include/uapi/asm/sgx.h"
Signature · 1 line
2.29.2
PATCH v2 9/25 x86/sgx: Move ENCLS leaf definitions to sgx.h
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Move the ENCLS leaf definitions to sgx.h so that they can be used by KVM.
---v1->v2:
- Removed "And because they're architectural." in commit msg, per Sean.
--- arch/x86/include/asm/sgx.h | 15 +++++++++++++++ arch/x86/kernel/cpu/sgx/encls.h | 15 --------------- 2 files changed, 15 insertions(+), 15 deletions(-)
arch/x86/include/asm/sgx.h +15 -0
@@ -27,6 +27,21 @@/* The bitmask for the EPC section type. */ #define SGX_CPUID_EPC_MASK GENMASK(3, 0) +enum sgx_encls_function { +ECREATE =0x00 , +EADD =0x01 , +EINIT =0x02 , +EREMOVE =0x03 , +EDGBRD =0x04 , +EDGBWR =0x05 , +EEXTEND =0x06 , +ELDU =0x08 , +EBLOCK =0x09 , +EPA =0x0A , +EWB =0x0B , +ETRACK =0x0C , +}; +/** * enum sgx_return_code - The return code type for ENCLS, ENCLU and ENCLV * %SGX_NOT_TRACKED: Previous ETRACK's shootdown sequence has not
arch/x86/kernel/cpu/sgx/encls.h +0 -15
@@ -11,21 +11,6 @@#include <asm/traps.h> #include "sgx.h" -enum sgx_encls_function { -ECREATE =0x00 , -EADD =0x01 , -EINIT =0x02 , -EREMOVE =0x03 , -EDGBRD =0x04 , -EDGBWR =0x05 , -EEXTEND =0x06 , -ELDU =0x08 , -EBLOCK =0x09 , -EPA =0x0A , -EWB =0x0B , -ETRACK =0x0C , -}; -/** * ENCLS_FAULT_FLAG - flag signifying an ENCLS return code is a trapnr *
Signature · 1 line
2.29.2
PATCH v2 10/25 x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT)
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Define the ENCLS leafs that are available with SGX2, also referred to as Enclave Dynamic Memory Management (EDMM). The leafs will be used by KVM to conditionally expose SGX2 capabilities to guests.
--- arch/x86/include/asm/sgx.h | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/include/asm/sgx.h +3 -0
@@ -40,6 +40,9 @@ enum sgx_encls_function {EPA =0x0A ,EWB =0x0B ,ETRACK =0x0C , +EAUG =0x0D , +EMODPR =0x0E , + EMODT = 0x0F, };/**
Signature · 1 line
2.29.2
PATCH v2 12/25 x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
In reply to Kai HuangAdd a helper to update SGX_LEPUBKEYHASHn MSRs. SGX virtualization also needs to update those MSRs based on guest's "virtual" SGX_LEPUBKEYHASHn before EINIT from guest.
---v1->v2:
- Added WARN_ON_ONCE(preemptible()) check, per Sean.
--- arch/x86/kernel/cpu/sgx/ioctl.c | 5 ++--- arch/x86/kernel/cpu/sgx/main.c | 17 +++++++++++++++++ arch/x86/kernel/cpu/sgx/sgx.h | 2 ++ 3 files changed, 21 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/sgx/ioctl.c +2 -3
@@ -495,7 +495,7 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct, void *token) {u64 mrsigner[4 ]; -int i, j, k; +int i, j;void *addr;int ret;
@@ -544,8 +544,7 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,preempt_disable (); -for (k =0 ; k <4 ; k++) -wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 + k, mrsigner[k]); +sgx_update_lepubkeyhash (mrsigner); ret =__einit (sigstruct, token, addr);
arch/x86/kernel/cpu/sgx/main.c +17 -0
@@ -696,6 +696,23 @@ static bool __init sgx_page_cache_init(void)return true; } + +/* +* Update the SGX_LEPUBKEYHASH MSRs to the values specified by caller. +* Bare-metal driver requires to update them to hash of enclave's signer +* before EINIT. KVM needs to update them to guest's virtual MSR values +* before doing EINIT from guest. +*/ +void sgx_update_lepubkeyhash (u64 *lepubkeyhash) +{ +int i; + +WARN_ON_ONCE (preemptible ()); + +for (i =0 ; i <4 ; i++) +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 + i, lepubkeyhash[i]); +} +static int __init sgx_init (void ) {int ret;
arch/x86/kernel/cpu/sgx/sgx.h +2 -0
@@ -92,4 +92,6 @@ static inline int __init sgx_vepc_init(void) }#endif +void sgx_update_lepubkeyhash (u64 *lepubkeyhash); +#endif /* _X86_SGX_H */
Signature · 1 line
2.29.2
PATCH v2 13/25 x86/sgx: Add helpers to expose ECREATE and EINIT to KVM
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
The host kernel must intercept ECREATE to impose policies on guests, and intercept EINIT to be able to write guest's virtual SGX_LEPUBKEYHASH MSR values to hardware before running guest's EINIT so it can run correctly according to hardware behavior.
Provide wrappers around __ecreate() and __einit() to hide the ugliness of overloading the ENCLS return value to encode multiple error formats in a single int. KVM will trap-and-execute ECREATE and EINIT as part of SGX virtualization, and reflect ENCLS execution result to guest by setting up guest's GPRs, or on an exception, injecting the correct fault based on return value of __ecreate() and __einit().
Use host userspace addresses (provided by KVM based on guest physical address of ENCLS parameters) to execute ENCLS/EINIT when possible. Accesses to both EPC and memory originating from ENCLS are subject to segmentation and paging mechanisms. It's also possible to generate kernel mappings for ENCLS parameters by resolving PFN but using __uaccess_xx() is simpler.
---v1->v2:
- Tried to address Dave's comments:
- Refined comments around @secs in sgx_virt_ecreate().
- Refined commit msg to explain why to use userspace address for ENCLS, instead of generating kernel mapping.
- Added access_ok() on userspace addresses, and give WARN() if check fails.
--- arch/x86/include/asm/sgx.h | 7 +++ arch/x86/kernel/cpu/sgx/virt.c | 110 +++++++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+)
arch/x86/include/asm/sgx.h +7 -0
@@ -365,4 +365,11 @@ struct sgx_sigstruct {* line! */ +#ifdef CONFIG_X86_SGX_KVM +int sgx_virt_ecreate (struct sgx_pageinfo *pageinfo,void __user *secs, +int *trapnr); +int sgx_virt_einit (void __user *sigstruct,void __user *token, +void __user *secs,u64 *lepubkeyhash,int *trapnr); +#endif +#endif /* _ASM_X86_SGX_H */
arch/x86/kernel/cpu/sgx/virt.c +110 -0
113 lines not on this page · raw message
Signature · 1 line
2.29.2
PATCH v2 11/25 x86/sgx: Add encls_faulted() helper
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add a helper to extract the fault indicator from an encoded ENCLS return value. SGX virtualization will also need to detect ENCLS faults.
--- arch/x86/kernel/cpu/sgx/encls.h | 15 ++++++++++++++- arch/x86/kernel/cpu/sgx/ioctl.c | 2 +- 2 files changed, 15 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/sgx/encls.h +14 -1
@@ -40,6 +40,19 @@ }while (0 ); \ } +/* +* encls_faulted() - Check if an ENCLS leaf faulted given an error code +* @ret the return value of an ENCLS leaf function call +* +* Return: +* - true: ENCLS leaf faulted. +* - false: Otherwise. +*/ +static inline bool encls_faulted (int ret) +{ +return ret &ENCLS_FAULT_FLAG ; +} +/** * encls_failed() - Check if an ENCLS function failed * @ret: the return value of an ENCLS function call
@@ -50,7 +63,7 @@*/ static inline bool encls_failed (int ret) { -if (ret &ENCLS_FAULT_FLAG ) +if (encls_faulted (ret))return ENCLS_TRAPNR (ret) !=X86_TRAP_PF ;return !!ret;
arch/x86/kernel/cpu/sgx/ioctl.c +1 -1
@@ -568,7 +568,7 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct, } } -if (ret &ENCLS_FAULT_FLAG ) { +if (encls_faulted (ret)) {if (encls_failed (ret))ENCLS_WARN (ret,"EINIT" );
Signature · 1 line
2.29.2
PATCH v2 14/25 x86/sgx: Move provisioning device creation out of SGX driver
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
And extract sgx_set_attribute() out of sgx_ioc_enclave_provision() and export it as symbol for KVM to use.
Provisioning key is sensitive. SGX driver only allows to create enclave which can access provisioning key when enclave creator has permission to open /dev/sgx_provision. It should apply to VM as well, as provisioning key is platform specific, thus unrestricted VM can also potentially compromise provisioning key.
Move provisioning device creation out of sgx_drv_init() to sgx_init() as preparation for adding SGX virtualization support, so that even SGX driver is not enabled due to flexible launch control is not available, SGX virtualization can still be enabled, and use it to restrict VM's capability of being able to access provisioning key.
--- arch/x86/include/asm/sgx.h | 3 ++ arch/x86/kernel/cpu/sgx/driver.c | 17 ---------- arch/x86/kernel/cpu/sgx/ioctl.c | 16 ++------- arch/x86/kernel/cpu/sgx/main.c | 57 +++++++++++++++++++++++++++++++- 4 files changed, 61 insertions(+), 32 deletions(-)
arch/x86/include/asm/sgx.h +3 -0
@@ -372,4 +372,7 @@ int sgx_virt_einit(void __user *sigstruct, void __user *token, void __user *secs, u64 *lepubkeyhash, int *trapnr);#endif +int sgx_set_attribute (unsigned long *allowed_attributes, +unsigned int attribute_fd); +#endif /* _ASM_X86_SGX_H */
arch/x86/kernel/cpu/sgx/driver.c +0 -17
@@ -136,10 +136,6 @@ static const struct file_operations sgx_encl_fops = { .get_unmapped_area = sgx_get_unmapped_area, }; -const struct file_operations sgx_provision_fops = { - .owner =THIS_MODULE , -}; -static struct miscdevice sgx_dev_enclave = { .minor =MISC_DYNAMIC_MINOR , .name ="sgx_enclave" ,
@@ -147,13 +143,6 @@ static struct miscdevice sgx_dev_enclave = { .fops = &sgx_encl_fops, }; -static struct miscdevice sgx_dev_provision = { - .minor =MISC_DYNAMIC_MINOR , - .name ="sgx_provision" , - .nodename ="sgx_provision" , - .fops = &sgx_provision_fops, -}; -int __init sgx_drv_init (void ) {unsigned int eax, ebx, ecx, edx;
@@ -187,11 +176,5 @@ int __init sgx_drv_init(void)if (ret)return ret; - ret =misc_register (&sgx_dev_provision); -if (ret) { -misc_deregister (&sgx_dev_enclave); -return ret; - } -return 0 ; }
arch/x86/kernel/cpu/sgx/ioctl.c +2 -14
@@ -2,6 +2,7 @@/* Copyright(c) 2016-20 Intel Corporation. */ #include <asm/mman.h> +#include <asm/sgx.h> #include <linux/mman.h> #include <linux/delay.h> #include <linux/file.h>
@@ -664,24 +665,11 @@ static long sgx_ioc_enclave_init(struct sgx_encl *encl, void __user *arg)static long sgx_ioc_enclave_provision (struct sgx_encl *encl,void __user *arg) {struct sgx_enclave_provision params; -struct file *file;if (copy_from_user (¶ms, arg,sizeof (params)))return -EFAULT ; - file =fget (params.fd); -if (!file) -return -EINVAL ; - -if (file->f_op != &sgx_provision_fops) { -fput (file); -return -EINVAL ; - } - - encl->attributes_mask |=SGX_ATTR_PROVISIONKEY ; - -fput (file); -return 0 ; +return sgx_set_attribute (&encl->attributes_mask, params.fd); }long sgx_ioctl (struct file *filep,unsigned int cmd,unsigned long arg)
arch/x86/kernel/cpu/sgx/main.c +56 -1
92 lines not on this page · raw message
Signature · 1 line
2.29.2
PATCH v2 15/25 KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Export the gva_to_gpa() helpers for use by SGX virtualization when executing ENCLS[ECREATE] and ENCLS[EINIT] on behalf of the guest. To execute ECREATE and EINIT, KVM must obtain the GPA of the target Secure Enclave Control Structure (SECS) in order to get its corresponding HVA.
Because the SECS must reside in the Enclave Page Cache (EPC), copying the SECS's data to a host-controlled buffer via existing exported helpers is not a viable option as the EPC is not readable or writable by the kernel.
SGX virtualization will also use gva_to_gpa() to obtain HVAs for non-EPC pages in order to pass user pointers directly to ECREATE and EINIT, which avoids having to copy pages worth of data into the kernel.
--- arch/x86/kvm/x86.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kvm/x86.c +2 -0
@@ -5959,6 +5959,7 @@ gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva,u32 access = (static_call (kvm_x86_get_cpl)(vcpu) ==3 ) ?PFERR_USER_MASK :0 ;return vcpu->arch.walk_mmu->gva_to_gpa (vcpu, gva, access, exception); } +EXPORT_SYMBOL_GPL (kvm_mmu_gva_to_gpa_read);gpa_t kvm_mmu_gva_to_gpa_fetch (struct kvm_vcpu *vcpu,gva_t gva,struct x86_exception *exception)
@@ -5975,6 +5976,7 @@ gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, access |=PFERR_WRITE_MASK ;return vcpu->arch.walk_mmu->gva_to_gpa (vcpu, gva, access, exception); } +EXPORT_SYMBOL_GPL (kvm_mmu_gva_to_gpa_write);/* uses this to access any guest's mapped memory without checking CPL */ gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva,
Signature · 1 line
2.29.2
PATCH v2 17/25 KVM: x86: Add support for reverse CPUID lookup of scattered features
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Introduce a scheme that allows KVM's CPUID magic to support features that are scattered in the kernel's feature words. To advertise and/or query guest support for CPUID-based features, KVM requires the bit number of an X86_FEATURE_* to match the bit number in its associated CPUID entry. For scattered features, this does not hold true.
Add a framework to allow defining KVM-only words, stored in kvm_cpu_caps after the shared kernel caps, that can be used to gather the scattered feature bits by translating X86_FEATURE_* flags into their KVM-defined feature.
Note, because reverse_cpuid_check() effectively forces kvm_cpu_caps lookups to be resolved at compile time, there is no runtime cost for translating from kernel-defined to kvm-defined features.
More details here: https://lkml.kernel.org/r/X/[email protected]
--- arch/x86/kvm/cpuid.c | 32 +++++++++++++++++++++++++++----- arch/x86/kvm/cpuid.h | 39 ++++++++++++++++++++++++++++++++++----- 2 files changed, 61 insertions(+), 10 deletions(-)
arch/x86/kvm/cpuid.c +27 -5
@@ -28,7 +28,7 @@* Unlike "struct cpuinfo_x86.x86_capability", kvm_cpu_caps doesn't need to be * aligned to sizeof(unsigned long) because it's not accessed via bitops. */ -u32 kvm_cpu_caps[NCAPINTS ]__read_mostly ; +u32 kvm_cpu_caps[NR_KVM_CPU_CAPS ]__read_mostly ;EXPORT_SYMBOL_GPL (kvm_cpu_caps);static u32 xstate_required_size (u64 xstate_bv,bool compacted)
@@ -53,6 +53,7 @@ static u32 xstate_required_size(u64 xstate_bv, bool compacted) }#define F feature_bit +#define SF (name) (boot_cpu_has(X86_FEATURE_##name) ? F(name) : 0)static inline struct kvm_cpuid_entry2 *cpuid_entry2_find (struct kvm_cpuid_entry2 *entries,int nent,u32 function,u32 index)
@@ -347,13 +348,13 @@ int kvm_vcpu_ioctl_get_cpuid2(struct kvm_vcpu *vcpu,return r; } -static __always_inline void kvm_cpu_cap_mask(enum cpuid_leafs leaf, u32 mask) +/* Mask kvm_cpu_caps for @leaf with the raw CPUID capabilities of this CPU. */ +static __always_inline void __kvm_cpu_cap_mask(enum cpuid_leafs leaf) {const struct cpuid_reg cpuid =x86_feature_cpuid (leaf *32 );struct kvm_cpuid_entry2 entry;reverse_cpuid_check (leaf); - kvm_cpu_caps[leaf] &= mask;cpuid_count (cpuid.function, cpuid.index, &entry.eax, &entry.ebx, &entry.ecx, &entry.edx);
@@ -361,6 +362,26 @@ static __always_inline void kvm_cpu_cap_mask(enum cpuid_leafs leaf, u32 mask) kvm_cpu_caps[leaf] &= *__cpuid_entry_get_reg (&entry, cpuid.reg); } +static __always_inline void kvm_cpu_cap_mask(enum cpuid_leafs leaf, u32 mask) +{ +/* Use the "init" variant for scattered leafs. */ +BUILD_BUG_ON (leaf >=NCAPINTS ); + + kvm_cpu_caps[leaf] &= mask; + +__kvm_cpu_cap_mask (leaf); +} + +static __always_inline void kvm_cpu_cap_init(enum cpuid_leafs leaf, u32 mask) +{ +/* Use the "mask" variant for hardwared-defined leafs. */ +BUILD_BUG_ON (leaf <NCAPINTS ); + + kvm_cpu_caps[leaf] = mask; + +__kvm_cpu_cap_mask (leaf); +} +void kvm_set_cpu_caps (void ) {unsigned int f_nx =is_efer_nx () ?F (NX ) :0 ;
@@ -371,12 +392,13 @@ void kvm_set_cpu_caps(void)unsigned int f_gbpages =0 ;unsigned int f_lm =0 ;#endif +memset (kvm_cpu_caps,0 ,sizeof (kvm_cpu_caps)); -BUILD_BUG_ON (sizeof (kvm_cpu_caps) > +BUILD_BUG_ON (sizeof (kvm_cpu_caps) - (NKVMCAPINTS *sizeof (*kvm_cpu_caps)) >sizeof (boot_cpu_data.x86_capability));memcpy (&kvm_cpu_caps, &boot_cpu_data.x86_capability, -sizeof (kvm_cpu_caps)); +sizeof (kvm_cpu_caps) - (NKVMCAPINTS *sizeof (*kvm_cpu_caps))); kvm_cpu_cap_mask(CPUID_1_ECX,/*
arch/x86/kvm/cpuid.h +34 -5
@@ -7,7 +7,20 @@#include <asm/processor.h> #include <uapi/asm/kvm_para.h> -extern u32 kvm_cpu_caps[NCAPINTS ]__read_mostly ; +/* +* Hardware-defined CPUID leafs that are scattered in the kernel, but need to +* be directly used by KVM. Note, these word values conflict with the kernel's +* "bug" caps, but KVM doesn't use those. +*/ +enum kvm_only_cpuid_leafs { +NR_KVM_CPU_CAPS =NCAPINTS , + +NKVMCAPINTS =NR_KVM_CPU_CAPS -NCAPINTS , +}; + +#define X86_KVM_FEATURE (w, f) ((w)*32 + (f)) + +extern u32 kvm_cpu_caps[NR_KVM_CPU_CAPS ]__read_mostly ;void kvm_set_cpu_caps (void );void kvm_update_cpuid_runtime (struct kvm_vcpu *vcpu);
@@ -100,6 +113,20 @@ static __always_inline void reverse_cpuid_check(unsigned int x86_leaf)BUILD_BUG_ON (reverse_cpuid[x86_leaf].function ==0 ); } +/* +* Translate feature bits that are scattered in the kernel's cpufeatures word +* into KVM feature words that align with hardware's definitions. +*/ +static __always_inline u32 __feature_translate(int x86_feature) +{ +return x86_feature; +} + +static __always_inline u32 __feature_leaf(int x86_feature) +{ +return __feature_translate (x86_feature) /32 ; +} +/* * Retrieve the bit mask from an X86_FEATURE_* definition. Features contain * the hardware defined bit number (stored in bits 4:0) and a software defined
@@ -108,6 +135,8 @@ static __always_inline void reverse_cpuid_check(unsigned int x86_leaf)*/ static __always_inline u32 __feature_bit(int x86_feature) { + x86_feature =__feature_translate (x86_feature); +reverse_cpuid_check (x86_feature /32 );return 1 << (x86_feature &31 ); }
@@ -116,7 +145,7 @@ static __always_inline u32 __feature_bit(int x86_feature) static __always_inline struct cpuid_reg x86_feature_cpuid(unsigned int x86_feature) { -unsigned int x86_leaf = x86_feature /32 ; +unsigned int x86_leaf =__feature_leaf (x86_feature);reverse_cpuid_check (x86_leaf);return reverse_cpuid[x86_leaf];
@@ -308,7 +337,7 @@ static inline bool cpuid_fault_enabled(struct kvm_vcpu *vcpu) static __always_inline void kvm_cpu_cap_clear(unsigned int x86_feature) { -unsigned int x86_leaf = x86_feature /32 ; +unsigned int x86_leaf =__feature_leaf (x86_feature);reverse_cpuid_check (x86_leaf); kvm_cpu_caps[x86_leaf] &= ~__feature_bit (x86_feature);
@@ -316,7 +345,7 @@ static __always_inline void kvm_cpu_cap_clear(unsigned int x86_feature) static __always_inline void kvm_cpu_cap_set(unsigned int x86_feature) { -unsigned int x86_leaf = x86_feature /32 ; +unsigned int x86_leaf =__feature_leaf (x86_feature);reverse_cpuid_check (x86_leaf); kvm_cpu_caps[x86_leaf] |=__feature_bit (x86_feature);
@@ -324,7 +353,7 @@ static __always_inline void kvm_cpu_cap_set(unsigned int x86_feature) static __always_inline u32 kvm_cpu_cap_get(unsigned int x86_feature) { -unsigned int x86_leaf = x86_feature /32 ; +unsigned int x86_leaf =__feature_leaf (x86_feature);reverse_cpuid_check (x86_leaf);return kvm_cpu_caps[x86_leaf] &__feature_bit (x86_feature);
Signature · 1 line
2.29.2
PATCH v2 19/25 KVM: VMX: Add basic handling of VM-Exit from SGX enclave
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add support for handling VM-Exits that originate from a guest SGX enclave. In SGX, an "enclave" is a new CPL3-only execution environment, wherein the CPU and memory state is protected by hardware to make the state inaccesible to code running outside of the enclave. When exiting an enclave due to an asynchronous event (from the perspective of the enclave), e.g. exceptions, interrupts, and VM-Exits, the enclave's state is automatically saved and scrubbed (the CPU loads synthetic state), and then reloaded when re-entering the enclave. E.g. after an instruction based VM-Exit from an enclave, vmcs.GUEST_RIP will not contain the RIP of the enclave instruction that trigered VM-Exit, but will instead point to a RIP in the enclave's untrusted runtime (the guest userspace code that coordinates entry/exit to/from the enclave).
To help a VMM recognize and handle exits from enclaves, SGX adds bits to existing VMCS fields, VM_EXIT_REASON.VMX_EXIT_REASON_FROM_ENCLAVE and GUEST_INTERRUPTIBILITY_INFO.GUEST_INTR_STATE_ENCLAVE_INTR. Define the new architectural bits, and add a boolean to struct vcpu_vmx to cache VMX_EXIT_REASON_FROM_ENCLAVE. Clear the bit in exit_reason so that checks against exit_reason do not need to account for SGX, e.g. "if (exit_reason == EXIT_REASON_EXCEPTION_NMI)" continues to work.
KVM is a largely a passive observer of the new bits, e.g. KVM needs to account for the bits when propagating information to a nested VMM, but otherwise doesn't need to act differently for the majority of VM-Exits from enclaves.
The one scenario that is directly impacted is emulation, which is for all intents and purposes impossible[1] since KVM does not have access to the RIP or instruction stream that triggered the VM-Exit. The inability to emulate is a non-issue for KVM, as most instructions that might trigger VM-Exit unconditionally #UD in an enclave (before the VM-Exit check. For the few instruction that conditionally #UD, KVM either never sets the exiting control, e.g. PAUSE_EXITING[2], or sets it if and only if the feature is not exposed to the guest in order to inject a #UD, e.g. RDRAND_EXITING.
But, because it is still possible for a guest to trigger emulation, e.g. MMIO, inject a #UD if KVM ever attempts emulation after a VM-Exit from an enclave. This is architecturally accurate for instruction VM-Exits, and for MMIO it's the least bad choice, e.g. it's preferable to killing the VM. In practice, only broken or particularly stupid guests should ever encounter this behavior.
Add a WARN in skip_emulated_instruction to detect any attempt to modify the guest's RIP during an SGX enclave VM-Exit as all such flows should either be unreachable or must handle exits from enclaves before getting to skip_emulated_instruction.
[1] Impossible for all practical purposes. Not truly impossible since KVM could implement some form of para-virtualization scheme.
[2] PAUSE_LOOP_EXITING only affects CPL0 and enclaves exist only at CPL3, so we also don't need to worry about that interaction.
---v1->v2:
- Add comment to explain why injecting #UD when emulating guest instruction due to VM-EXIT from enclave, per Sean.
--- arch/x86/include/asm/vmx.h | 1 + arch/x86/include/uapi/asm/vmx.h | 1 + arch/x86/kvm/vmx/nested.c | 2 ++ arch/x86/kvm/vmx/vmx.c | 45 +++++++++++++++++++++++++++++++-- 4 files changed, 47 insertions(+), 2 deletions(-)
arch/x86/include/asm/vmx.h +1 -0
@@ -373,6 +373,7 @@ enum vmcs_field {#define GUEST_INTR_STATE_MOV_SS 0x00000002#define GUEST_INTR_STATE_SMI 0x00000004#define GUEST_INTR_STATE_NMI 0x00000008 +#define GUEST_INTR_STATE_ENCLAVE_INTR 0x00000010/* GUEST_ACTIVITY_STATE flags */ #define GUEST_ACTIVITY_ACTIVE 0
arch/x86/include/uapi/asm/vmx.h +1 -0
@@ -27,6 +27,7 @@#define VMX_EXIT_REASONS_FAILED_VMENTRY 0x80000000 +#define VMX_EXIT_REASONS_SGX_ENCLAVE_MODE 0x08000000#define EXIT_REASON_EXCEPTION_NMI 0#define EXIT_REASON_EXTERNAL_INTERRUPT 1
arch/x86/kvm/vmx/nested.c +2 -0
@@ -4105,6 +4105,8 @@ static void prepare_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, {/* update exit information fields: */ vmcs12->vm_exit_reason = vm_exit_reason; +if (to_vmx (vcpu)->exit_reason.enclave_mode) + vmcs12->vm_exit_reason |=VMX_EXIT_REASONS_SGX_ENCLAVE_MODE ; vmcs12->exit_qualification = exit_qualification; vmcs12->vm_exit_intr_info = exit_intr_info;
arch/x86/kvm/vmx/vmx.c +43 -2
76 lines not on this page · raw message
Signature · 1 line
2.29.2
PATCH v2 16/25 KVM: x86: Define new #PF SGX error code bit
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Page faults that are signaled by the SGX Enclave Page Cache Map (EPCM), as opposed to the traditional IA32/EPT page tables, set an SGX bit in the error code to indicate that the #PF was induced by SGX. KVM will need to emulate this behavior as part of its trap-and-execute scheme for virtualizing SGX Launch Control, e.g. to inject SGX-induced #PFs if EINIT faults in the host, and to support live migration.
--- arch/x86/include/asm/kvm_host.h | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -227,6 +227,7 @@ enum x86_intercept_stage;#define PFERR_RSVD_BIT 3#define PFERR_FETCH_BIT 4#define PFERR_PK_BIT 5 +#define PFERR_SGX_BIT 15#define PFERR_GUEST_FINAL_BIT 32#define PFERR_GUEST_PAGE_BIT 33
@@ -236,6 +237,7 @@ enum x86_intercept_stage;#define PFERR_RSVD_MASK (1U << PFERR_RSVD_BIT)#define PFERR_FETCH_MASK (1U << PFERR_FETCH_BIT)#define PFERR_PK_MASK (1U << PFERR_PK_BIT) +#define PFERR_SGX_MASK (1U << PFERR_SGX_BIT)#define PFERR_GUEST_FINAL_MASK (1ULL << PFERR_GUEST_FINAL_BIT)#define PFERR_GUEST_PAGE_MASK (1ULL << PFERR_GUEST_PAGE_BIT)
Signature · 1 line
2.29.2
PATCH v2 18/25 KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Define a new KVM-only feature word for advertising and querying SGX sub-features in CPUID.0x12.0x0.EAX. Because SGX1 and SGX2 are scattered in the kernel's feature word, they need to be translated so that the bit numbers match those of hardware.
--- arch/x86/kvm/cpuid.h | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
arch/x86/kvm/cpuid.h +12 -1
@@ -13,13 +13,18 @@* "bug" caps, but KVM doesn't use those. */ enum kvm_only_cpuid_leafs { -NR_KVM_CPU_CAPS =NCAPINTS , +CPUID_12_EAX =NCAPINTS , +NR_KVM_CPU_CAPS ,NKVMCAPINTS =NR_KVM_CPU_CAPS -NCAPINTS , };#define X86_KVM_FEATURE (w, f) ((w)*32 + (f)) +/* Intel-defined SGX sub-features, CPUID level 0x12 (EAX). */ +#define __X86_FEATURE_SGX1 X86_KVM_FEATURE(CPUID_12_EAX, 0) +#define __X86_FEATURE_SGX2 X86_KVM_FEATURE(CPUID_12_EAX, 1) +extern u32 kvm_cpu_caps[NR_KVM_CPU_CAPS ]__read_mostly ;void kvm_set_cpu_caps (void );
@@ -93,6 +98,7 @@ static const struct cpuid_reg reverse_cpuid[] = { [CPUID_8000_0007_EBX] = {0x80000007, 0, CPUID_EBX}, [CPUID_7_EDX] = { 7, 0, CPUID_EDX}, [CPUID_7_1_EAX] = { 7, 1, CPUID_EAX}, + [CPUID_12_EAX] = {0x00000012, 0, CPUID_EAX}, };/*
@@ -119,6 +125,11 @@ static __always_inline void reverse_cpuid_check(unsigned int x86_leaf)*/ static __always_inline u32 __feature_translate(int x86_feature) { +if (x86_feature ==X86_FEATURE_SGX1 ) +return __X86_FEATURE_SGX1; +else if (x86_feature ==X86_FEATURE_SGX2 ) +return __X86_FEATURE_SGX2; +return x86_feature; }
Signature · 1 line
2.29.2
PATCH v2 20/25 KVM: VMX: Frame in ENCLS handler for SGX virtualization
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Introduce sgx.c and sgx.h, along with the framework for handling ENCLS VM-Exits. Add a bool, enable_sgx, that will eventually be wired up to a module param to control whether or not SGX virtualization is enabled at runtime.
--- arch/x86/kvm/Makefile | 2 ++ arch/x86/kvm/vmx/sgx.c | 50 ++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/sgx.h | 15 +++++++++++++ arch/x86/kvm/vmx/vmx.c | 9 +++++--- 4 files changed, 73 insertions(+), 3 deletions(-) create mode 100644 arch/x86/kvm/vmx/sgx.c create mode 100644 arch/x86/kvm/vmx/sgx.h
arch/x86/kvm/Makefile +2 -0
@@ -23,6 +23,8 @@ kvm-$(CONFIG_KVM_XEN) += xen.o kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \ vmx/evmcs.o vmx/nested.o vmx/posted_intr.o +kvm-intel-$(CONFIG_X86_SGX_KVM) += vmx/sgx.o + kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o svm/sev.o obj-$(CONFIG_KVM) += kvm.o
arch/x86/kvm/vmx/sgx.c new file +50 -0
50 lines not on this page · raw message
arch/x86/kvm/vmx/sgx.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_SGX_H +#define __KVM_X86_SGX_H + +#include <linux/kvm_host.h> + +#ifdef CONFIG_X86_SGX_KVM +extern bool __read_mostly enable_sgx; + +int handle_encls (struct kvm_vcpu *vcpu); +#else +#define enable_sgx 0 +#endif + +#endif /* __KVM_X86_SGX_H */
arch/x86/kvm/vmx/vmx.c +6 -3
@@ -57,6 +57,7 @@#include "mmu.h" #include "nested.h" #include "pmu.h" +#include "sgx.h" #include "trace.h" #include "vmcs.h" #include "vmcs12.h"
@@ -5673,16 +5674,18 @@ static int handle_vmx_instruction(struct kvm_vcpu *vcpu)return 1 ; } +#ifndef CONFIG_X86_SGX_KVM static int handle_encls (struct kvm_vcpu *vcpu) {/* -* SGX virtualization is not yet supported. There is no software -* enable bit for SGX, so we have to trap ENCLS and inject a #UD -* to prevent the guest from executing ENCLS. +* SGX virtualization is disabled. There is no software enable bit for +* SGX, so KVM intercepts all ENCLS leafs and injects a #UD to prevent +* the guest from executing ENCLS (when SGX is supported by hardware). */ kvm_queue_exception (vcpu,UD_VECTOR );return 1 ; } +#endif /* CONFIG_X86_SGX_KVM */ static int handle_bus_lock_vmexit (struct kvm_vcpu *vcpu) {
Signature · 1 line
2.29.2
PATCH v2 21/25 KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add an ECREATE handler that will be used to intercept ECREATE for the purpose of enforcing and enclave's MISCSELECT, ATTRIBUTES and XFRM, i.e. to allow userspace to restrict SGX features via CPUID. ECREATE will be intercepted when any of the aforementioned masks diverges from hardware in order to enforce the desired CPUID model, i.e. inject #GP if the guest attempts to set a bit that hasn't been enumerated as allowed-1 in CPUID.
Note, access to the PROVISIONKEY is not yet supported.
---
v1-v2:
- Fixed 'contents' leak problem pointed out by Sean.
--- arch/x86/include/asm/kvm_host.h | 3 + arch/x86/kvm/vmx/sgx.c | 263 ++++++++++++++++++++++++++++++++ 2 files changed, 266 insertions(+)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -1038,6 +1038,9 @@ struct kvm_arch {bool bus_lock_detection_enabled; +/* Guest can access the SGX PROVISIONKEY. */ +bool sgx_provisioning_allowed; +struct kvm_pmu_event_filter *pmu_event_filter;struct task_struct *nx_lpage_recovery_thread;
arch/x86/kvm/vmx/sgx.c +263 -0
275 lines not on this page · raw message
Signature · 1 line
2.29.2
PATCH v2 22/25 KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Emulate the four Launch Enclave public key hash MSRs (LE hash MSRs) that exist on CPUs that support SGX Launch Control (LC). SGX LC modifies the behavior of ENCLS[EINIT] to use the LE hash MSRs when verifying the key used to sign an enclave. On CPUs without LC support, the LE hash is hardwired into the CPU to an Intel controlled key (the Intel key is also the reset value of the LE hash MSRs). Track the guest's desired hash so that a future patch can stuff the hash into the hardware MSRs when executing EINIT on behalf of the guest, when those MSRs are writable in host.
Note, KVM allows writes to the LE hash MSRs if IA32_FEATURE_CONTROL is unlocked. This is technically not architectural behavior, but it's roughly equivalent to the arch behavior of the MSRs being writable prior to activating SGX[1]. Emulating SGX activation is feasible, but adds no tangible benefits and would just create extra work for KVM and guest firmware.
[1] SGX related bits in IA32_FEATURE_CONTROL cannot be set until SGX is activated, e.g. by firmware. SGX activation is triggered by setting bit 0 in MSR 0x7a. Until SGX is activated, the LE hash MSRs are writable, e.g. to allow firmware to lock down the LE root key with a non-Intel value.
--- arch/x86/kvm/vmx/sgx.c | 35 +++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/sgx.h | 6 ++++++ arch/x86/kvm/vmx/vmx.c | 20 ++++++++++++++++++++ arch/x86/kvm/vmx/vmx.h | 2 ++ 4 files changed, 63 insertions(+)
arch/x86/kvm/vmx/sgx.c +35 -0
@@ -11,6 +11,9 @@bool __read_mostly enable_sgx; +/* Initial value of guest's virtual SGX_LEPUBKEYHASHn MSRs */ +static u64 sgx_pubkey_hash[4 ]__ro_after_init ; +/* * ENCLS's memory operands use a fixed segment (DS) and a fixed * address size based on the mode. Related prefixes are ignored.
@@ -311,3 +314,35 @@ int handle_encls(struct kvm_vcpu *vcpu) }return 1 ; } + +void setup_default_sgx_lepubkeyhash (void ) +{ +/* +* Use Intel's default value for Skylake hardware if Launch Control is +* not supported, i.e. Intel's hash is hardcoded into silicon, or if +* Launch Control is supported and enabled, i.e. mimic the reset value +* and let the guest write the MSRs at will. If Launch Control is +* supported but disabled, then use the current MSR values as the hash +* MSRs exist but are read-only (locked and not writable). +*/ +if (!enable_sgx ||boot_cpu_has (X86_FEATURE_SGX_LC ) || +rdmsrl_safe (MSR_IA32_SGXLEPUBKEYHASH0 , &sgx_pubkey_hash[0 ])) { + sgx_pubkey_hash[0 ] =0xa6053e051270b7acULL ; + sgx_pubkey_hash[1 ] =0x6cfbe8ba8b3b413dULL ; + sgx_pubkey_hash[2 ] =0xc4916d99f2b3735dULL ; + sgx_pubkey_hash[3 ] =0xd4f8c05909f9bb3bULL ; + }else { +/* MSR_IA32_SGXLEPUBKEYHASH0 is read above */ +rdmsrl (MSR_IA32_SGXLEPUBKEYHASH1 , sgx_pubkey_hash[1 ]); +rdmsrl (MSR_IA32_SGXLEPUBKEYHASH2 , sgx_pubkey_hash[2 ]); +rdmsrl (MSR_IA32_SGXLEPUBKEYHASH3 , sgx_pubkey_hash[3 ]); + } +} + +void vcpu_setup_sgx_lepubkeyhash (struct kvm_vcpu *vcpu) +{ +struct vcpu_vmx *vmx =to_vmx (vcpu); + +memcpy (vmx->msr_ia32_sgxlepubkeyhash, sgx_pubkey_hash, +sizeof (sgx_pubkey_hash)); +}
arch/x86/kvm/vmx/sgx.h +6 -0
@@ -8,8 +8,14 @@extern bool __read_mostly enable_sgx;int handle_encls (struct kvm_vcpu *vcpu); + +void setup_default_sgx_lepubkeyhash (void ); +void vcpu_setup_sgx_lepubkeyhash (struct kvm_vcpu *vcpu);#else #define enable_sgx 0 + +static inline void setup_default_sgx_lepubkeyhash (void ) { } +static inline void vcpu_setup_sgx_lepubkeyhash (struct kvm_vcpu *vcpu) { }#endif #endif /* __KVM_X86_SGX_H */
arch/x86/kvm/vmx/vmx.c +20 -0
@@ -1904,6 +1904,13 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)case MSR_IA32_FEAT_CTL : msr_info->data = vmx->msr_ia32_feature_control;break ; + case MSR_IA32_SGXLEPUBKEYHASH0 ... MSR_IA32_SGXLEPUBKEYHASH3: +if (!msr_info->host_initiated && + !guest_cpuid_has (vcpu,X86_FEATURE_SGX_LC )) +return 1 ; + msr_info->data =to_vmx (vcpu)->msr_ia32_sgxlepubkeyhash + [msr_info->index -MSR_IA32_SGXLEPUBKEYHASH0 ]; +break ; case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:if (!nested_vmx_allowed (vcpu))return 1 ;
@@ -2198,6 +2205,15 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (msr_info->host_initiated && data ==0 )vmx_leave_nested (vcpu);break ; + case MSR_IA32_SGXLEPUBKEYHASH0 ... MSR_IA32_SGXLEPUBKEYHASH3: +if (!msr_info->host_initiated && + (!guest_cpuid_has (vcpu,X86_FEATURE_SGX_LC ) || + ((vmx->msr_ia32_feature_control &FEAT_CTL_LOCKED ) && + !(vmx->msr_ia32_feature_control &FEAT_CTL_SGX_LC_ENABLED )))) +return 1 ; + vmx->msr_ia32_sgxlepubkeyhash + [msr_index -MSR_IA32_SGXLEPUBKEYHASH0 ] = data; +break ; case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:if (!msr_info->host_initiated)return 1 ;/* they are read-only */
@@ -7020,6 +7036,8 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu)else memset (&vmx->nested.msrs,0 ,sizeof (vmx->nested.msrs)); +vcpu_setup_sgx_lepubkeyhash (vcpu); + vmx->nested.posted_intr_nv =-1 ; vmx->nested.current_vmptr =-1ull ;
@@ -7953,6 +7971,8 @@ static __init int hardware_setup(void)if (!enable_ept || !cpu_has_vmx_intel_pt ()) pt_mode =PT_MODE_SYSTEM ; +setup_default_sgx_lepubkeyhash (); +if (nested) {nested_vmx_setup_ctls_msrs (&vmcs_config.nested, vmx_capability.ept);
arch/x86/kvm/vmx/vmx.h +2 -0
@@ -325,6 +325,8 @@ struct vcpu_vmx {*/ u64 msr_ia32_feature_control;u64 msr_ia32_feature_control_valid_bits; +/* SGX Launch Control public key hash */ +u64 msr_ia32_sgxlepubkeyhash[4 ];u64 ept_pointer;struct pt_desc pt_desc;
Signature · 1 line
2.29.2
PATCH v2 23/25 KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add a VM-Exit handler to trap-and-execute EINIT when SGX LC is enabled in the host. When SGX LC is enabled, the host kernel may rewrite the hardware values at will, e.g. to launch enclaves with different signers, thus KVM needs to intercept EINIT to ensure it is executed with the correct LE hash (even if the guest sees a hardwired hash).
Switching the LE hash MSRs on VM-Enter/VM-Exit is not a viable option as writing the MSRs is prohibitively expensive, e.g. on SKL hardware each WRMSR is ~400 cycles. And because EINIT takes tens of thousands of cycles to execute, the ~1500 cycle overhead to trap-and-execute EINIT is unlikely to be noticed by the guest, let alone impact its overall SGX performance.
--- arch/x86/kvm/vmx/sgx.c | 55 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)
arch/x86/kvm/vmx/sgx.c +55 -0
@@ -275,6 +275,59 @@ static int handle_encls_ecreate(struct kvm_vcpu *vcpu)return r; } +static int handle_encls_einit (struct kvm_vcpu *vcpu) +{ +unsigned long sig_hva, secs_hva, token_hva, rflags; +struct vcpu_vmx *vmx =to_vmx (vcpu); +gva_t sig_gva, secs_gva, token_gva; +gpa_t sig_gpa, secs_gpa, token_gpa; +int ret, trapnr; + +if (sgx_get_encls_gva (vcpu,kvm_rbx_read (vcpu),1808 ,4096 , &sig_gva) || +sgx_get_encls_gva (vcpu,kvm_rcx_read (vcpu),4096 ,4096 , &secs_gva) || +sgx_get_encls_gva (vcpu,kvm_rdx_read (vcpu),304 ,512 , &token_gva)) +return 1 ; + +/* +* Translate the SIGSTRUCT, SECS and TOKEN pointers from GVA to GPA. +* Resume the guest on failure to inject a #PF. +*/ +if (sgx_gva_to_gpa (vcpu, sig_gva, false, &sig_gpa) || +sgx_gva_to_gpa (vcpu, secs_gva, true, &secs_gpa) || +sgx_gva_to_gpa (vcpu, token_gva, false, &token_gpa)) +return 1 ; + +/* +* ...and then to HVA. The order of accesses isn't architectural, i.e. +* KVM doesn't have to fully process one address at a time. Exit to +* userspace if a GPA is invalid. Note, all structures are aligned and +* cannot split pages. +*/ +if (sgx_gpa_to_hva (vcpu, sig_gpa, &sig_hva) || +sgx_gpa_to_hva (vcpu, secs_gpa, &secs_hva) || +sgx_gpa_to_hva (vcpu, token_gpa, &token_hva)) +return 0 ; + + ret =sgx_virt_einit ((void __user *)sig_hva, (void __user *)token_hva, + (void __user *)secs_hva, + vmx->msr_ia32_sgxlepubkeyhash, &trapnr); + +if (ret == -EFAULT ) +return sgx_inject_fault (vcpu, secs_gva, trapnr); + + rflags =vmx_get_rflags (vcpu) & ~(X86_EFLAGS_CF |X86_EFLAGS_PF | +X86_EFLAGS_AF |X86_EFLAGS_SF | +X86_EFLAGS_OF ); +if (ret) + rflags |=X86_EFLAGS_ZF ; +else + rflags &= ~X86_EFLAGS_ZF ; +vmx_set_rflags (vcpu, rflags); + +kvm_rax_write (vcpu, ret); +return kvm_skip_emulated_instruction (vcpu); +} +static inline bool encls_leaf_enabled_in_guest (struct kvm_vcpu *vcpu,u32 leaf) {if (!enable_sgx || !guest_cpuid_has (vcpu,X86_FEATURE_SGX ))
@@ -307,6 +360,8 @@ int handle_encls(struct kvm_vcpu *vcpu) }else {if (leaf ==ECREATE )return handle_encls_ecreate (vcpu); +if (leaf ==EINIT ) +return handle_encls_einit (vcpu);WARN (1 ,"KVM: unexpected exit on ENCLS[%u]" , leaf); vcpu->run->exit_reason =KVM_EXIT_UNKNOWN ; vcpu->run->hw.hardware_exit_reason =EXIT_REASON_ENCLS ;
Signature · 1 line
2.29.2
PATCH v2 24/25 KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Enable SGX virtualization now that KVM has the VM-Exit handlers needed to trap-and-execute ENCLS to ensure correctness and/or enforce the CPU model exposed to the guest. Add a KVM module param, "sgx", to allow an admin to disable SGX virtualization independent of the kernel.
When supported in hardware and the kernel, advertise SGX1, SGX2 and SGX LC to userspace via CPUID and wire up the ENCLS_EXITING bitmap based on the guest's SGX capabilities, i.e. to allow ENCLS to be executed in an SGX-enabled guest. With the exception of the provision key, all SGX attribute bits may be exposed to the guest. Guest access to the provision key, which is controlled via securityfs, will be added in a future patch.
Note, KVM does not yet support exposing ENCLS_C leafs or ENCLV leafs.
--- arch/x86/kvm/cpuid.c | 57 +++++++++++++++++++++++++++- arch/x86/kvm/vmx/nested.c | 26 +++++++++++-- arch/x86/kvm/vmx/nested.h | 5 +++ arch/x86/kvm/vmx/sgx.c | 80 ++++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/sgx.h | 13 +++++++ arch/x86/kvm/vmx/vmcs12.c | 1 + arch/x86/kvm/vmx/vmcs12.h | 4 +- arch/x86/kvm/vmx/vmx.c | 35 ++++++++++++++++- 8 files changed, 212 insertions(+), 9 deletions(-)
arch/x86/kvm/cpuid.c +55 -2
@@ -18,6 +18,7 @@#include <asm/processor.h> #include <asm/user.h> #include <asm/fpu/xstate.h> +#include <asm/sgx.h> #include "cpuid.h" #include "lapic.h" #include "mmu.h"
@@ -171,6 +172,21 @@ static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) vcpu->arch.guest_supported_xcr0 = (best->eax | ((u64 )best->edx <<32 )) & supported_xcr0; +/* +* Bits 127:0 of the allowed SECS.ATTRIBUTES (CPUID.0x12.0x1) enumerate +* the supported XSAVE Feature Request Mask (XFRM), i.e. the enclave's +* requested XCR0 value. The enclave's XFRM must be a subset of XCRO +* at the time of EENTER, thus adjust the allowed XFRM by the guest's +* supported XCR0. Similar to XCR0 handling, FP and SSE are forced to +* '1' even on CPUs that don't support XSAVE. +*/ + best =kvm_find_cpuid_entry (vcpu,0x12 ,0x1 ); +if (best) { + best->ecx &= vcpu->arch.guest_supported_xcr0 &0xffffffff ; + best->edx &= vcpu->arch.guest_supported_xcr0 >>32 ; + best->ecx |=XFEATURE_MASK_FPSSE ; + } +kvm_update_pv_runtime (vcpu); vcpu->arch.maxphyaddr =cpuid_query_maxphyaddr (vcpu);
@@ -429,7 +445,7 @@ void kvm_set_cpu_caps(void) ); kvm_cpu_cap_mask(CPUID_7_0_EBX, - F(FSGSBASE) | F(BMI1) | F(HLE) | F(AVX2) | F(SMEP) | + F(FSGSBASE) | F(SGX) | F(BMI1) | F(HLE) | F(AVX2) | F(SMEP) | F(BMI2) | F(ERMS) | F(INVPCID) | F(RTM) | 0 /*MPX*/ | F(RDSEED) | F(ADX) | F(SMAP) | F(AVX512IFMA) | F(AVX512F) | F(AVX512PF) | F(AVX512ER) | F(AVX512CD) | F(CLFLUSHOPT) | F(CLWB) | F(AVX512DQ) |
@@ -440,7 +456,8 @@ void kvm_set_cpu_caps(void)F (AVX512VBMI ) |F (LA57 ) |F (PKU ) |0 /*OSPKE*/ |F (RDPID ) |F (AVX512_VPOPCNTDQ ) |F (UMIP ) |F (AVX512_VBMI2 ) |F (GFNI ) |F (VAES ) |F (VPCLMULQDQ ) |F (AVX512_VNNI ) |F (AVX512_BITALG ) | -F (CLDEMOTE ) |F (MOVDIRI ) |F (MOVDIR64B ) |0 /*WAITPKG*/ +F (CLDEMOTE ) |F (MOVDIRI ) |F (MOVDIR64B ) |0 /*WAITPKG*/ | +F (SGX_LC ) );/* Set LA57 based on hardware capability. */ if (cpuid_ecx (7 ) &F (LA57 ))
@@ -479,6 +496,10 @@ void kvm_set_cpu_caps(void)F (XSAVEOPT ) |F (XSAVEC ) |F (XGETBV1 ) |F (XSAVES ) ); +kvm_cpu_cap_init (CPUID_12_EAX , +SF (SGX1 ) |SF (SGX2 ) + ); + kvm_cpu_cap_mask(CPUID_8000_0001_ECX, F(LAHF_LM) | F(CMP_LEGACY) | 0 /*SVM*/ | 0 /* ExtApicSpace */ | F(CR8_LEGACY) | F(ABM) | F(SSE4A) | F(MISALIGNSSE) |
@@ -800,6 +821,38 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) entry->edx =0 ; }break ; +case 0x12 : +/* Intel SGX */ +if (!kvm_cpu_cap_has (X86_FEATURE_SGX )) { + entry->eax = entry->ebx = entry->ecx = entry->edx =0 ; +break ; + } + +/* +* Index 0: Sub-features, MISCSELECT (a.k.a extended features) +* and max enclave sizes. The SGX sub-features and MISCSELECT +* are restricted by kernel and KVM capabilities (like most +* feature flags), while enclave size is unrestricted. +*/ +cpuid_entry_override (entry,CPUID_12_EAX ); + entry->ebx &=SGX_MISC_EXINFO ; + + entry =do_host_cpuid (array, function,1 ); +if (!entry) + goto out; + +/* +* Index 1: SECS.ATTRIBUTES. ATTRIBUTES are restricted a la +* feature flags. Advertise all supported flags, including +* privileged attributes that require explicit opt-in from +* userspace. ATTRIBUTES.XFRM is not adjusted as userspace is +* expected to derive it from supported XCR0. +*/ + entry->eax &=SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT | +/* PROVISIONKEY | */ SGX_ATTR_EINITTOKENKEY | +SGX_ATTR_KSS ; + entry->ebx &=0 ; +break ;/* Intel PT */ case 0x14 :if (!kvm_cpu_cap_has (X86_FEATURE_INTEL_PT )) {
arch/x86/kvm/vmx/nested.c +23 -3
@@ -11,6 +11,7 @@#include "mmu.h" #include "nested.h" #include "pmu.h" +#include "sgx.h" #include "trace.h" #include "vmx.h" #include "x86.h"
@@ -2306,6 +2307,9 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)if (!nested_cpu_has2 (vmcs12,SECONDARY_EXEC_UNRESTRICTED_GUEST )) exec_control &= ~SECONDARY_EXEC_UNRESTRICTED_GUEST ; +if (exec_control &SECONDARY_EXEC_ENCLS_EXITING ) +vmx_write_encls_bitmap (&vmx->vcpu, vmcs12); +secondary_exec_controls_set (vmx, exec_control); }
@@ -5707,6 +5711,20 @@ static bool nested_vmx_exit_handled_cr(struct kvm_vcpu *vcpu,return false; } +static bool nested_vmx_exit_handled_encls (struct kvm_vcpu *vcpu, +struct vmcs12 *vmcs12) +{ +u32 encls_leaf; + +if (!nested_cpu_has2 (vmcs12,SECONDARY_EXEC_ENCLS_EXITING )) +return false; + + encls_leaf =kvm_rax_read (vcpu); +if (encls_leaf >62 ) + encls_leaf =63 ; +return vmcs12->encls_exiting_bitmap &BIT_ULL (encls_leaf); +} +static bool nested_vmx_exit_handled_vmcs_access (struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12,gpa_t bitmap) {
@@ -5803,9 +5821,6 @@ static bool nested_vmx_l0_wants_exit(struct kvm_vcpu *vcpu,case EXIT_REASON_VMFUNC :/* VM functions are emulated through L2->L0 vmexits. */ return true; -case EXIT_REASON_ENCLS : -/* SGX is never exposed to L1 */ -return true;default :break ; }
@@ -5929,6 +5944,8 @@ static bool nested_vmx_l1_wants_exit(struct kvm_vcpu *vcpu,case EXIT_REASON_TPAUSE :return nested_cpu_has2 (vmcs12,SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE ); +case EXIT_REASON_ENCLS : +return nested_vmx_exit_handled_encls (vcpu, vmcs12);default :return true; }
@@ -6504,6 +6521,9 @@ void nested_vmx_setup_ctls_msrs(struct nested_vmx_msrs *msrs, u32 ept_caps) msrs->secondary_ctls_high |=SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; +if (enable_sgx) + msrs->secondary_ctls_high |=SECONDARY_EXEC_ENCLS_EXITING ; +/* miscellaneous data */ rdmsr(MSR_IA32_VMX_MISC, msrs->misc_low,
arch/x86/kvm/vmx/nested.h +5 -0
@@ -244,6 +244,11 @@ static inline bool nested_exit_on_intr(struct kvm_vcpu *vcpu)PIN_BASED_EXT_INTR_MASK ; } +static inline bool nested_cpu_has_encls_exit (struct vmcs12 *vmcs12) +{ +return nested_cpu_has2 (vmcs12,SECONDARY_EXEC_ENCLS_EXITING ); +} +/* * if fixed0[i] == 1: val[i] must be 1 * if fixed1[i] == 0: val[i] must be 0
arch/x86/kvm/vmx/sgx.c +79 -1
93 lines not on this page · raw message
arch/x86/kvm/vmx/sgx.h +13 -0
@@ -4,6 +4,9 @@#include <linux/kvm_host.h> +#include "capabilities.h" +#include "vmx_ops.h" +#ifdef CONFIG_X86_SGX_KVM extern bool __read_mostly enable_sgx;
@@ -11,11 +14,21 @@ int handle_encls(struct kvm_vcpu *vcpu);void setup_default_sgx_lepubkeyhash (void );void vcpu_setup_sgx_lepubkeyhash (struct kvm_vcpu *vcpu); + +void vmx_write_encls_bitmap (struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12);#else #define enable_sgx 0static inline void setup_default_sgx_lepubkeyhash (void ) { }static inline void vcpu_setup_sgx_lepubkeyhash (struct kvm_vcpu *vcpu) { } + +static inline void vmx_write_encls_bitmap (struct kvm_vcpu *vcpu, +struct vmcs12 *vmcs12) +{ +/* Nothing to do if hardware doesn't support SGX */ +if (cpu_has_vmx_encls_vmexit ()) +vmcs_write64 (ENCLS_EXITING_BITMAP ,-1ull ); +}#endif #endif /* __KVM_X86_SGX_H */
arch/x86/kvm/vmx/vmcs12.c +1 -0
@@ -50,6 +50,7 @@ const unsigned short vmcs_field_to_offset_table[] = {FIELD64 (VMREAD_BITMAP , vmread_bitmap),FIELD64 (VMWRITE_BITMAP , vmwrite_bitmap),FIELD64 (XSS_EXIT_BITMAP , xss_exit_bitmap), +FIELD64 (ENCLS_EXITING_BITMAP , encls_exiting_bitmap),FIELD64 (GUEST_PHYSICAL_ADDRESS , guest_physical_address),FIELD64 (VMCS_LINK_POINTER , vmcs_link_pointer), FIELD64(GUEST_IA32_DEBUGCTL, guest_ia32_debugctl),
arch/x86/kvm/vmx/vmcs12.h +3 -1
@@ -69,7 +69,8 @@ struct __packed vmcs12 {u64 vm_function_control;u64 eptp_list_address;u64 pml_address; -u64 padding64[3 ];/* room for future expansion */ +u64 encls_exiting_bitmap; +u64 padding64[2 ];/* room for future expansion */ /* * To allow migration of L1 (complete with its L2 guests) between * machines of different natural widths (32 or 64 bit), we cannot have
@@ -256,6 +257,7 @@ static inline void vmx_check_vmcs12_offsets(void)CHECK_OFFSET (vm_function_control,296 );CHECK_OFFSET (eptp_list_address,304 );CHECK_OFFSET (pml_address,312 ); +CHECK_OFFSET (encls_exiting_bitmap,320 );CHECK_OFFSET (cr0_guest_host_mask,344 );CHECK_OFFSET (cr4_guest_host_mask,352 );CHECK_OFFSET (cr0_read_shadow,360 );
arch/x86/kvm/vmx/vmx.c +33 -2
@@ -2204,6 +2204,9 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) vmx->msr_ia32_feature_control = data;if (msr_info->host_initiated && data ==0 )vmx_leave_nested (vcpu); + +/* SGX may be enabled/disabled by guest's firmware */ +vmx_write_encls_bitmap (vcpu,NULL );break ; case MSR_IA32_SGXLEPUBKEYHASH0 ... MSR_IA32_SGXLEPUBKEYHASH3: if (!msr_info->host_initiated &&
@@ -4366,6 +4369,15 @@ static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx)if (!vcpu->kvm->arch.bus_lock_detection_enabled) exec_control &= ~SECONDARY_EXEC_BUS_LOCK_DETECTION ; +if (cpu_has_vmx_encls_vmexit () && nested) { +if (guest_cpuid_has (vcpu,X86_FEATURE_SGX )) + vmx->nested.msrs.secondary_ctls_high |= +SECONDARY_EXEC_ENCLS_EXITING ; +else + vmx->nested.msrs.secondary_ctls_high &= + ~SECONDARY_EXEC_ENCLS_EXITING ; + } + vmx->secondary_exec_control = exec_control; }
@@ -4465,8 +4477,7 @@ static void init_vmcs(struct vcpu_vmx *vmx)vmcs_write16 (GUEST_PML_INDEX ,PML_ENTITY_NUM -1 ); } -if (cpu_has_vmx_encls_vmexit ()) -vmcs_write64 (ENCLS_EXITING_BITMAP ,-1ull ); +vmx_write_encls_bitmap (&vmx->vcpu,NULL );if (vmx_pt_mode_is_host_guest ()) {memset (&vmx->pt_desc,0 ,sizeof (vmx->pt_desc));
@@ -7364,6 +7375,19 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)set_cr4_guest_host_mask (vmx); +vmx_write_encls_bitmap (vcpu,NULL ); +if (guest_cpuid_has (vcpu,X86_FEATURE_SGX )) + vmx->msr_ia32_feature_control_valid_bits |=FEAT_CTL_SGX_ENABLED ; +else + vmx->msr_ia32_feature_control_valid_bits &= ~FEAT_CTL_SGX_ENABLED ; + +if (guest_cpuid_has (vcpu,X86_FEATURE_SGX_LC )) + vmx->msr_ia32_feature_control_valid_bits |= +FEAT_CTL_SGX_LC_ENABLED ; +else + vmx->msr_ia32_feature_control_valid_bits &= + ~FEAT_CTL_SGX_LC_ENABLED ; +/* Refresh #PF interception to account for MAXPHYADDR changes. */ vmx_update_exception_bitmap (vcpu); }
@@ -7384,6 +7408,13 @@ static __init void vmx_set_cpu_caps(void)if (vmx_pt_mode_is_host_guest ())kvm_cpu_cap_check_and_set (X86_FEATURE_INTEL_PT ); +if (!enable_sgx) { +kvm_cpu_cap_clear (X86_FEATURE_SGX ); +kvm_cpu_cap_clear (X86_FEATURE_SGX_LC ); +kvm_cpu_cap_clear (X86_FEATURE_SGX1 ); +kvm_cpu_cap_clear (X86_FEATURE_SGX2 ); + } +if (vmx_umip_emulated ())kvm_cpu_cap_set (X86_FEATURE_UMIP );
Signature · 1 line
2.29.2
PATCH v2 25/25 KVM: x86: Add capability to grant VM access to privileged SGX attribute
In reply to Kai HuangFrom: Sean Christopherson <[email protected]>
Add a capability, KVM_CAP_SGX_ATTRIBUTE, that can be used by userspace to grant a VM access to a priveleged attribute, with args[0] holding a file handle to a valid SGX attribute file.
The SGX subsystem restricts access to a subset of enclave attributes to provide additional security for an uncompromised kernel, e.g. to prevent malware from using the PROVISIONKEY to ensure its nodes are running inside a geniune SGX enclave and/or to obtain a stable fingerprint.
To prevent userspace from circumventing such restrictions by running an enclave in a VM, KVM restricts guest access to privileged attributes by default.
--- Documentation/virt/kvm/api.rst | 23 +++++++++++++++++++++++ arch/x86/kvm/cpuid.c | 2 +- arch/x86/kvm/x86.c | 21 +++++++++++++++++++++ include/uapi/linux/kvm.h | 1 + 4 files changed, 46 insertions(+), 1 deletion(-)
Documentation/virt/kvm/api.rst +23 -0
@@ -6227,6 +6227,29 @@ KVM_RUN_BUS_LOCK flag is used to distinguish between them. This capability can be used to check / enable 2nd DAWR feature provided by POWER10 processor. +7.24 KVM_CAP_SGX_ATTRIBUTE +---------------------- + +:Architectures: x86 +:Target: VM +:Parameters: args[0] is a file handle of a SGX attribute file in securityfs +:Returns: 0 on success, -EINVAL if the file handle is invalid or if a requested + attribute is not supported by KVM. + +KVM_CAP_SGX_ATTRIBUTE enables a userspace VMM to grant a VM access to one or +more priveleged enclave attributes. args[0] must hold a file handle to a valid +SGX attribute file corresponding to an attribute that is supported/restricted +by KVM (currently only PROVISIONKEY). + +The SGX subsystem restricts access to a subset of enclave attributes to provide +additional security for an uncompromised kernel, e.g. use of the PROVISIONKEY +is restricted to deter malware from using the PROVISIONKEY to obtain a stable +system fingerprint. To prevent userspace from circumventing such restrictions +by running an enclave in a VM, KVM prevents access to privileged attributes by +default. + +See Documentation/x86/sgx/2.Kernel-internals.rst for more details. + 8. Other capabilities. ======================
arch/x86/kvm/cpuid.c +1 -1
@@ -849,7 +849,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* expected to derive it from supported XCR0. */ entry->eax &=SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT | -/* PROVISIONKEY | */ SGX_ATTR_EINITTOKENKEY | +SGX_ATTR_PROVISIONKEY |SGX_ATTR_EINITTOKENKEY |SGX_ATTR_KSS ; entry->ebx &=0 ;break ;
arch/x86/kvm/x86.c +21 -0
@@ -75,6 +75,7 @@#include <asm/tlbflush.h> #include <asm/intel_pt.h> #include <asm/emulate_prefix.h> +#include <asm/sgx.h> #include <clocksource/hyperv_timer.h> #define CREATE_TRACE_POINTS
@@ -3759,6 +3760,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_X86_USER_SPACE_MSR :case KVM_CAP_X86_MSR_FILTER :case KVM_CAP_ENFORCE_PV_FEATURE_CPUID : +#ifdef CONFIG_X86_SGX_KVM +case KVM_CAP_SGX_ATTRIBUTE : +#endif r =1 ;break ;#ifdef CONFIG_KVM_XEN
@@ -5345,6 +5349,23 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, kvm->arch.bus_lock_detection_enabled = true; r =0 ;break ; +#ifdef CONFIG_X86_SGX_KVM +case KVM_CAP_SGX_ATTRIBUTE : { +unsigned long allowed_attributes =0 ; + + r =sgx_set_attribute (&allowed_attributes, cap->args[0 ]); +if (r) +break ; + +/* KVM only supports the PROVISIONKEY privileged attribute. */ +if ((allowed_attributes &SGX_ATTR_PROVISIONKEY ) && + !(allowed_attributes & ~SGX_ATTR_PROVISIONKEY )) + kvm->arch.sgx_provisioning_allowed = true; +else + r = -EINVAL ; +break ; + } +#endif default : r = -EINVAL ;break ;
include/uapi/linux/kvm.h +1 -0
@@ -1078,6 +1078,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_DIRTY_LOG_RING 192#define KVM_CAP_X86_BUS_LOCK_EXIT 193#define KVM_CAP_PPC_DAWR1 194 +#define KVM_CAP_SGX_ATTRIBUTE 195#ifdef KVM_CAP_IRQ_ROUTING
Signature · 1 line
2.29.2
Borislav Petkov
maintainerQuoted 3 lines · Kai Huang
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Ok, I guess I'll queue 1-14 once Sean doesn't find anything objectionable then give Paolo an immutable commit to base the KVM stuff ontop.
Unless folks have better suggestions, ofc.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 5 lines · Borislav Petkov
Quoted 3 lines · Kai Huang
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Quoted 8 lines
Ok, I guess I'll queue 1-14 once Sean doesn't find anything objectionable then give Paolo an immutable commit to base the KVM stuff ontop.
Unless folks have better suggestions, ofc.
Thx.
Thanks Boris!
Hi Sean, Paolo,
Could you take a look? Thanks.
Paolo Bonzini
maintainerQuoted 8 lines · Borislav Petkov
Quoted 3 lines · Kai Huang
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Ok, I guess I'll queue 1-14 once Sean doesn't find anything objectionable then give Paolo an immutable commit to base the KVM stuff ontop.
Sounds great.
Paolo
Quoted 5 lines · Borislav Petkov
Quoted 3 lines · Kai Huang
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Quoted 8 lines
Ok, I guess I'll queue 1-14 once Sean doesn't find anything objectionable then give Paolo an immutable commit to base the KVM stuff ontop.
Unless folks have better suggestions, ofc.
Thx.
Hi Boris,
Sorry that we found a bug in below patch in series:
[PATCH v2 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
that I made a mistake when copying & pasting, which results in SECS page and va_page not being freed correctly in sgx_encl_release().
Sorry for the mistake. I will send out another version with that fixed.
Re: [PATCH v2 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Kai HuangQuoted 75 lines · On Tue, 9 Mar 2021 14:39:01 +1300 Kai Huang wrote:
From: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original
sgx_free_epc_page()assgx_encl_free_epc_page()to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.Signed-off-by Jarkko Sakkinen <[email protected]>Co-developed-by Kai Huang <[email protected]>Signed-off-by Kai Huang <[email protected]>---
v1->v2:- Merge original
WARN()andpr_err_once()into one singleWARN(), suggested by Sean.--- arch/x86/kernel/cpu/sgx/encl.c | 27 ++++++++++++++++++++++++--- arch/x86/kernel/cpu/sgx/main.c | 12 ++++-------- 2 files changed, 28 insertions(+), 11 deletions(-)arch/x86/kernel/cpu/sgx/encl.c +22
-1@@ -381,6 +381,27 @@ const struct vm_operations_struct sgx_vm_ops = { .access = sgx_vma_access, }; +static void sgx_encl_free_epc_page (struct sgx_epc_page *epc_page) +{ +int ret; + +WARN_ON_ONCE (epc_page->flags &SGX_EPC_PAGE_RECLAIMER_TRACKED ); + +/* +* Give a message to remind EPC page is leaked when EREMOVE fails, +* and requires machine reboot to get leaked pages back. This can +* be improved in future by adding stats of leaked pages, etc. +*/ +#define EREMOVE_ERROR_MESSAGE \ + "EREMOVE returned %d (0x%x). EPC page leaked. Reboot required to retrieve leaked pages." + ret =__eremove (sgx_get_epc_virt_addr (epc_page)); +if (WARN_ONCE (ret,EREMOVE_ERROR_MESSAGE , ret, ret)) +return ; +#undef EREMOVE_ERROR_MESSAGE + +sgx_free_epc_page (epc_page); +} +/** * sgx_encl_release - Destroy an enclave instance * @kref: address of a kref inside &sgx_encl @@ -404,7 +425,7 @@ void sgx_encl_release(struct kref *ref)if (sgx_unmark_page_reclaimable (entry->epc_page))continue ; -sgx_free_epc_page (entry->epc_page); +sgx_encl_free_epc_page (entry->epc_page); encl->secs_child_cnt--; entry->epc_page =NULL ; }
arch/x86/kernel/cpu/sgx/encl.c void sgx_encl_release(struct kref *ref) · line 439
Quoted 5 lines
arch/x86/kernel/cpu/sgx/encl.c +1
-1@@ -415,7 +436,7 @@ void sgx_encl_release(struct kref *ref)xa_destroy (&encl->page_array);if (!encl->secs_child_cnt && encl->secs.epc_page) { -sgx_free_epc_page (encl->secs.epc_page); +sgx_encl_free_epc_page (entry->epc_page);
Sorry. A mistake during copy/paste. Will fix.
arch/x86/kernel/cpu/sgx/encl.c void sgx_encl_release(struct kref *ref) · line 447
Quoted 9 lines
arch/x86/kernel/cpu/sgx/encl.c +1
-1encl->secs.epc_page =NULL ; }@@ -423,7 +444,7 @@ void sgx_encl_release(struct kref *ref) va_page = list_first_entry(&encl->va_pages, struct sgx_va_page, list);list_del (&va_page->list); -sgx_free_epc_page (va_page->epc_page); +sgx_encl_free_epc_page (entry->epc_page);
Sorry. A mistake during copy/paste. Will fix.
Quoted 33 lines
arch/x86/kernel/cpu/sgx/encl.c +0
-0kfree (va_page); }arch/x86/kernel/cpu/sgx/main.c +4
-8@@ -598,18 +598,14 @@ struct sgx_epc_page *sgx_alloc_epc_page(void *owner, bool reclaim)* sgx_free_epc_page() - Free an EPC page * @page: an EPC page * -* Call EREMOVE for an EPC page and insert it back to the list of free pages. +* Put the EPC page back to the list of free pages. It's the caller's +* responsibility to make sure that the page is in uninitialized state. In other +* words, do EREMOVE, EWB or whatever operation is necessary before calling +* this function. */ void sgx_free_epc_page (struct sgx_epc_page *page) {struct sgx_epc_section *section = &sgx_epc_sections[page->section]; -int ret; - -WARN_ON_ONCE (page->flags &SGX_EPC_PAGE_RECLAIMER_TRACKED ); - - ret =__eremove (sgx_get_epc_virt_addr (page)); -if (WARN_ONCE (ret,"EREMOVE returned %d (0x%x)" , ret, ret)) -return ;spin_lock (§ion->lock);list_add_tail (&page->list, §ion->page_list);Signature · 1 line
2.29.2
Borislav Petkov
maintainerQuoted 1 line · Kai Huang
Sorry for the mistake. I will send out another version with that fixed.
If patch 3 is the only one which needs to change, you can send only that one as a reply to the original patch 3 message...
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteJarkko Sakkinen
maintainerQuoted 10 lines · Borislav Petkov
Quoted 3 lines · Kai Huang
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Ok, I guess I'll queue 1-14 once Sean doesn't find anything objectionable then give Paolo an immutable commit to base the KVM stuff ontop.
Unless folks have better suggestions, ofc.
I'm otherwise cool with that, except patch #2.
It's based on this series:
https://lore.kernel.org/linux-sgx/[email protected]/
It's not reasonable to create driver specific wrapper for sgx_free_epc_page() because there is exactly *2* call sites of the function in the driver. The driver contains 10 call sites (11 after my NUMA patches have been applied) of sgx_free_epc_page() in total.
Instead, it is better to add explicit EREMOVE to those call sites.
The wrapper only trashes the codebase. I'm not happy with it, given all the trouble to make it clean and sound.
Quoted 7 lines
Thx.
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Quoted 23 lines · Jarkko Sakkinen
Quoted 10 lines · Borislav Petkov
Quoted 3 lines · Kai Huang
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Ok, I guess I'll queue 1-14 once Sean doesn't find anything objectionable then give Paolo an immutable commit to base the KVM stuff ontop.
Unless folks have better suggestions, ofc.
I'm otherwise cool with that, except patch #2.
It's based on this series:
https://lore.kernel.org/linux-sgx/[email protected]/
It's not reasonable to create driver specific wrapper for
sgx_free_epc_page()because there is exactly *2* call sites of the function in the driver. The driver contains 10 call sites (11 after my NUMA patches have been applied) ofsgx_free_epc_page()in total.
Quoted 4 lines
Instead, it is better to add explicit EREMOVE to those call sites.
The wrapper only trashes the codebase. I'm not happy with it, given all the trouble to make it clean and sound.
However, your change has side effort: it always put page back into free pool, even EREMOVE fails. To make your change w/o having any functional change, it has to be:
if(!sgx_reset_epc_page()) sgx_free_epc_page();
And for this, Dave raised one concern we should add a WARN() to let user know EPC page is leaked, and reboot is requied to get them back.
However with sgx_reset_epc_page(), there's no place to add such WARN(), and implementing original sgx_free_epc_page() as sgx_encl_free_epc_page() looks very reasonable to me:
https://www.spinics.net/lists/linux-sgx/msg04631.html
Hi Dave,
What is your comment here?
Quoted 11 lines
Quoted 8 lines
Thx.
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette/Jarkko
PATCH v3 3/25 x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Kai HuangFrom: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original sgx_free_epc_page() as sgx_encl_free_epc_page() to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.
---v2->v3:
- Fixed bug during copy/paste which results in SECS page and va pages are not correctly freed in sgx_encl_release() (sorry for the mistake).
- Added Jarkko's Acked-by.
v1->v2:
- Changed to hide both SGX1 and SGX2 from /proc/cpuinfo, since no concrete use case, per Boris.
- Refined commit msg to explain why to hide SGX1 and SGX2 in /proc/cpuinfo.
--- arch/x86/kernel/cpu/sgx/encl.c | 27 ++++++++++++++++++++++++--- arch/x86/kernel/cpu/sgx/main.c | 12 ++++-------- 2 files changed, 28 insertions(+), 11 deletions(-)
arch/x86/kernel/cpu/sgx/encl.c +24 -3
@@ -381,6 +381,27 @@ const struct vm_operations_struct sgx_vm_ops = { .access = sgx_vma_access, }; +static void sgx_encl_free_epc_page (struct sgx_epc_page *epc_page) +{ +int ret; + +WARN_ON_ONCE (epc_page->flags &SGX_EPC_PAGE_RECLAIMER_TRACKED ); + +/* +* Give a message to remind EPC page is leaked when EREMOVE fails, +* and requires machine reboot to get leaked pages back. This can +* be improved in future by adding stats of leaked pages, etc. +*/ +#define EREMOVE_ERROR_MESSAGE \ + "EREMOVE returned %d (0x%x). EPC page leaked. Reboot required to retrieve leaked pages." + ret =__eremove (sgx_get_epc_virt_addr (epc_page)); +if (WARN_ONCE (ret,EREMOVE_ERROR_MESSAGE , ret, ret)) +return ; +#undef EREMOVE_ERROR_MESSAGE + +sgx_free_epc_page (epc_page); +} +/** * sgx_encl_release - Destroy an enclave instance * @kref: address of a kref inside &sgx_encl
@@ -404,7 +425,7 @@ void sgx_encl_release(struct kref *ref)if (sgx_unmark_page_reclaimable (entry->epc_page))continue ; -sgx_free_epc_page (entry->epc_page); +sgx_encl_free_epc_page (entry->epc_page); encl->secs_child_cnt--; entry->epc_page =NULL ; }
@@ -415,7 +436,7 @@ void sgx_encl_release(struct kref *ref)xa_destroy (&encl->page_array);if (!encl->secs_child_cnt && encl->secs.epc_page) { -sgx_free_epc_page (encl->secs.epc_page); +sgx_encl_free_epc_page (encl->secs.epc_page); encl->secs.epc_page =NULL ; }
@@ -423,7 +444,7 @@ void sgx_encl_release(struct kref *ref) va_page = list_first_entry(&encl->va_pages, struct sgx_va_page, list);list_del (&va_page->list); -sgx_free_epc_page (va_page->epc_page); +sgx_encl_free_epc_page (va_page->epc_page);kfree (va_page); }
arch/x86/kernel/cpu/sgx/main.c +4 -8
@@ -598,18 +598,14 @@ struct sgx_epc_page *sgx_alloc_epc_page(void *owner, bool reclaim)* sgx_free_epc_page() - Free an EPC page * @page: an EPC page * -* Call EREMOVE for an EPC page and insert it back to the list of free pages. +* Put the EPC page back to the list of free pages. It's the caller's +* responsibility to make sure that the page is in uninitialized state. In other +* words, do EREMOVE, EWB or whatever operation is necessary before calling +* this function. */ void sgx_free_epc_page (struct sgx_epc_page *page) {struct sgx_epc_section *section = &sgx_epc_sections[page->section]; -int ret; - -WARN_ON_ONCE (page->flags &SGX_EPC_PAGE_RECLAIMER_TRACKED ); - - ret =__eremove (sgx_get_epc_virt_addr (page)); -if (WARN_ONCE (ret,"EREMOVE returned %d (0x%x)" , ret, ret)) -return ;spin_lock (§ion->lock);list_add_tail (&page->list, §ion->page_list);
Signature · 1 line
2.29.2
Quoted 7 lines · On Wed, 10 Mar 2021 14:29:48 +0100 Borislav Petkov wrote:
Quoted 1 line · Kai Huang
Sorry for the mistake. I will send out another version with that fixed.
If patch 3 is the only one which needs to change, you can send only that one as a reply to the original patch 3 message...
Thx.
Hi Boris,
Yes it is the only patch needs change. I have send out updated v3 patch 3.
I provided some changelog history to explain and also added Jarkko's Acked-by in the new patch. Sorry for the trouble.
Hi Sean,
If you see this, could you take another check on whether this series is OK?
Thanks in advance.
Re: [PATCH v2 01/25] x86/cpufeatures: Make SGX_LC feature bit depend on SGX bit
In reply to Kai HuangQuoted 9 lines · Kai Huang
Move
SGX_LCfeature bit to CPUID dependency table to make clearing all SGX feature bits easier. Also removeclear_sgx_caps()since it is just a wrapper ofsetup_clear_cpu_cap(X86_FEATURE_SGX)now.Suggested-by Sean Christopherson <[email protected]>Acked-by Dave Hansen <[email protected]>Acked-by Jarkko Sakkinen <[email protected]>Signed-off-by Kai Huang <[email protected]>---
Sean Christopherson
maintainerRe: [PATCH v2 02/25] x86/cpufeatures: Add SGX1 and SGX2 sub-features
In reply to Kai Huangarch/x86/include/asm/cpufeatures.h line 294
Quoted 6 lines · Kai Huang
arch/x86/include/asm/cpufeatures.h +2
-0@@ -290,6 +290,8 @@#define X86_FEATURE_FENCE_SWAPGS_KERNEL (11*32+ 5)/* "" LFENCE in kernel entry SWAPGS path */ #define X86_FEATURE_SPLIT_LOCK_DETECT (11*32+ 6)/* #AC for split lock */ #define X86_FEATURE_PER_THREAD_MBA (11*32+ 7)/* "" Per-thread Memory Bandwidth Allocation */ +#define X86_FEATURE_SGX1 (11*32+ 8)/* "" Basic SGX */ +#define X86_FEATURE_SGX2 (11*32+ 9)/* "" SGX Enclave Dynamic Memory Management (EDMM) */
There are spaces immediately after _SGX2 that can be replace by a tab.
Re: [PATCH v3 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Kai HuangQuoted 23 lines · Kai Huang
From: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original
sgx_free_epc_page()assgx_encl_free_epc_page()to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.Signed-off-by Jarkko Sakkinen <[email protected]>Co-developed-by Kai Huang <[email protected]>Acked-by Jarkko Sakkinen <[email protected]>Signed-off-by Kai Huang <[email protected]>
Quoted 6 lines
---
v2->v3:- Fixed bug during copy/paste which results in SECS page and va pages are not correctly freed in
sgx_encl_release()(sorry for the mistake).
- Added Jarkko's Acked-by.
That Acked-by should either be dropped or moved above Co-developed-by to make checkpatch happy.
Sean Christopherson
maintainerRe: [PATCH v2 06/25] x86/cpu/intel: Allow SGX virtualization without Launch Control support
In reply to Kai Huangarch/x86/kernel/cpu/feat_ctl.c void init_ia32_feat_ctl(struct cpuinfo_x86 *c) · line 125
Quoted 5 lines · Kai Huang
arch/x86/kernel/cpu/feat_ctl.c +5
-0+if (cpu_has (c,X86_FEATURE_SGX ) &&IS_ENABLED (CONFIG_X86_SGX )) { +/* +* Separate out SGX driver enabling from KVM. This allows KVM +* guests to use SGX even if the kernel SGX driver refuses to +* use it. This happens if flexible Faunch Control is not
/Faunch/Launch
Clever handling of the multitude of combinations!
Re: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Kai HuangQuoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }
arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();
I love this code.
Quoted 6 lines
arch/x86/kernel/cpu/sgx/main.c +0
-0if (ret) goto err_kthread;Signature · 1 line
2.29.2
Sean Christopherson
maintainerRe: [PATCH v2 08/25] x86/sgx: Expose SGX architectural definitions to the kernel
In reply to Kai HuangQuoted 5 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Expose SGX architectural structures, as KVM will use many of the architectural constants and structs to virtualize SGX.
Quoted 7 lines
Name the new header file as
asm/sgx.h, rather thanasm/sgx_arch.h, to have single header to provide SGX facilities to share with other kernel componments.Signed-off-by Sean Christopherson <[email protected]>Co-developed-by Kai Huang <[email protected]>Acked-by Dave Hansen <[email protected]>
Same checkpatch warning. Probably doesn't matter.
Quoted 42 lines
Signed-off-by Kai Huang <[email protected]>--- .../cpu/sgx/arch.h => include/asm/sgx.h} | 20 ++++++++++++++----- arch/x86/kernel/cpu/sgx/encl.c | 2 +- arch/x86/kernel/cpu/sgx/sgx.h | 2 +- tools/testing/selftests/sgx/defines.h | 2 +- 4 files changed, 18 insertions(+), 8 deletions(-) rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx.h} (95%)arch/x86/include/asm/sgx.h renamed from arch/x86/kernel/cpu/sgx/arch.h +9
-4@@ -2,15 +2,20 @@/** * Copyright(c) 2016-20 Intel Corporation. * -* Contains data structures defined by the SGX architecture. Data structures -* defined by the Linux software stack should not be placed here. +* Intel Software Guard Extensions (SGX) support. */ -#ifndef _ASM_X86_SGX_ARCH_H -#define _ASM_X86_SGX_ARCH_H +#ifndef _ASM_X86_SGX_H +#define _ASM_X86_SGX_H#include <linux/bits.h> #include <linux/types.h> +/* +* This file contains both data structures defined by SGX architecture and Linux +* defined software data structures and functions. The two should not be mixed +* together for better readibility. The architectural definitions come first. +*/ +/* The SGX specific CPUID function. */ #define SGX_CPUID 0x12/* EPC enumeration. */
arch/x86/include/asm/sgx.h struct sgx_sigstruct { · line 347
Quoted 7 lines
arch/x86/include/asm/sgx.h renamed from arch/x86/kernel/cpu/sgx/arch.h +3
-1@@ -337,4 +342,9 @@ struct sgx_sigstruct {#define SGX_LAUNCH_TOKEN_SIZE 304 -#endif /* _ASM_X86_SGX_ARCH_H */ +/* +* Do not put any hardware-defined SGX structure representations below this +* line!
Heh, which line? Yep, it's Friday afternoon...
Quoted 1 line
arch/x86/include/asm/sgx.h renamed from arch/x86/kernel/cpu/sgx/arch.h +1
-0+*/
Sean Christopherson
maintainerQuoted 11 lines · Paolo Bonzini
Quoted 8 lines · Borislav Petkov
Quoted 3 lines · Kai Huang
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Ok, I guess I'll queue 1-14 once Sean doesn't find anything objectionable then give Paolo an immutable commit to base the KVM stuff ontop.
Sounds great.
Patches 1-14 look good, just a few minor nits, nothing functional. I'll look at the KVM patches next week.
Thanks for picking this up Kai!
Quoted 11 lines · Sean Christopherson
Quoted 10 lines · Paolo Bonzini
Quoted 8 lines · Borislav Petkov
Quoted 3 lines · Kai Huang
This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.
Ok, I guess I'll queue 1-14 once Sean doesn't find anything objectionable then give Paolo an immutable commit to base the KVM stuff ontop.
Quoted 6 lines
Quoted 1 line
Sounds great.
Patches 1-14 look good, just a few minor nits, nothing functional. I'll look at the KVM patches next week.
Thanks for picking this up Kai!
Thank you Sean! I'll address your comments in next version.
Jarkko Sakkinen
maintainerRe: [PATCH v3 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Sean ChristophersonQuoted 27 lines · Sean Christopherson
Quoted 26 lines · Kai Huang
From: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original
sgx_free_epc_page()assgx_encl_free_epc_page()to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.Signed-off-by Jarkko Sakkinen <[email protected]>Co-developed-by Kai Huang <[email protected]>Acked-by Jarkko Sakkinen <[email protected]>Signed-off-by Kai Huang <[email protected]>---
v2->v3:
Quoted 8 lines
Quoted 3 lines
- Fixed bug during copy/paste which results in SECS page and va pages are not correctly freed in
sgx_encl_release()(sorry for the mistake).
- Added Jarkko's Acked-by.That Acked-by should either be dropped or moved above Co-developed-by to make checkpatch happy.
Reviewed-by Sean Christopherson <[email protected]>
Oops, my bad. Yup, ack should be removed.
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Sean ChristophersonQuoted 34 lines · Sean Christopherson
Quoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();I love this code.
Quoted 1 line
Reviewed-by Sean Christopherson <[email protected]>
I'm still wondering why this code let's go through when sgx_drv_init() succeeds and sgx_vepc_init() fails.
The inline comment explains only the mirrored case (which does make sense).
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
Quoted 35 lines · Jarkko Sakkinen
Quoted 34 lines · Sean Christopherson
Quoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();I love this code.
Quoted 7 lines
Quoted 1 line
Reviewed-by Sean Christopherson <[email protected]>I'm still wondering why this code let's go through when
sgx_drv_init()succeeds andsgx_vepc_init()fails.The inline comment explains only the mirrored case (which does make sense).
I.e. if sgx_drv_init() succeeds, I'd expect that sgx_vepc_init() must succeed. Why expect legitly anything else?
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
Quoted 38 lines · Jarkko Sakkinen
Quoted 37 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();I love this code.
Reviewed-by Sean Christopherson <[email protected]>
Quoted 8 lines
Quoted 5 lines
I'm still wondering why this code let's go through when
sgx_drv_init()succeeds andsgx_vepc_init()fails.The inline comment explains only the mirrored case (which does make sense).
I.e. if
sgx_drv_init()succeeds, I'd expect thatsgx_vepc_init()must succeed. Why expect legitly anything else?
Apologies coming with these ideas at this point, but here is what this led me.
I think that the all this complexity comes from a bad code structure.
So, what is essentially happening here:
- We essentially want to make EPC always work.
- Driver optionally.
So what this sums to is something like:
ret = sgx_epc_init();
if (ret) {
pr_err("EPC initialization failed.\n");
return ret;
}
ret = sgx_drv_init();
if (ret)
pr_info("Driver could not be initialized.\n");
/* continue */I.e. I think there should be a single EPC init, which does both EPC bootstrapping and vepc, and driver initialization comes after that.
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
Quoted 72 lines · Jarkko Sakkinen
Quoted 38 lines · Jarkko Sakkinen
Quoted 37 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();I love this code.
Reviewed-by Sean Christopherson <[email protected]>Quoted 8 lines
Quoted 5 lines
I'm still wondering why this code let's go through when
sgx_drv_init()succeeds andsgx_vepc_init()fails.The inline comment explains only the mirrored case (which does make sense).
I.e. if
sgx_drv_init()succeeds, I'd expect thatsgx_vepc_init()must succeed. Why expect legitly anything else?Apologies coming with these ideas at this point, but here is what this led me.
I think that the all this complexity comes from a bad code structure.
So, what is essentially happening here:
- We essentially want to make EPC always work.
- Driver optionally.So what this sums to is something like:
ret = sgx_epc_init(); if (ret) { pr_err("EPC initialization failed.\n"); return ret; } ret = sgx_drv_init(); if (ret) pr_info("Driver could not be initialized.\n"); /* continue */
Quoted 2 lines
I.e. I think there should be a single EPC init, which does both EPC bootstrapping and vepc, and driver initialization comes after that.
In other words, from SGX point of view, the thing that KVM needs is to cut out EPC and driver part into different islands. How this is now implemented in the current patch set is half-way there but not yet what it should be.
/Jarkko
Re: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
Quoted 73 lines · On Sun, 14 Mar 2021 17:27:18 +0200 Jarkko Sakkinen wrote:
Quoted 72 lines · Jarkko Sakkinen
Quoted 38 lines · Jarkko Sakkinen
Quoted 37 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();I love this code.
Reviewed-by Sean Christopherson <[email protected]>Quoted 8 lines
Quoted 5 lines
I'm still wondering why this code let's go through when
sgx_drv_init()succeeds andsgx_vepc_init()fails.The inline comment explains only the mirrored case (which does make sense).
I.e. if
sgx_drv_init()succeeds, I'd expect thatsgx_vepc_init()must succeed. Why expect legitly anything else?Apologies coming with these ideas at this point, but here is what this led me.
I think that the all this complexity comes from a bad code structure.
So, what is essentially happening here:
- We essentially want to make EPC always work.
- Driver optionally.So what this sums to is something like:
ret = sgx_epc_init(); if (ret) { pr_err("EPC initialization failed.\n"); return ret; } ret = sgx_drv_init(); if (ret) pr_info("Driver could not be initialized.\n"); /* continue */
Quoted 7 lines
Quoted 2 lines
I.e. I think there should be a single EPC init, which does both EPC bootstrapping and vepc, and driver initialization comes after that.
In other words, from SGX point of view, the thing that KVM needs is to cut out EPC and driver part into different islands. How this is now implemented in the current patch set is half-way there but not yet what it should be.
Well conceptually, SGX virtualization and SGX driver are two independently functionalities can be enabled separately, although they both requires some come functionalities, such as /dev/sgx_provision, which we have moved to sgx/main.c exactly for this purpose. THerefore, conceptually, it is bad to make assumption that, if SGX virtualization initialization succeeded, SGX driver must succeed -- we can potentially add more staff in SGX virtualization in the future..
If the name sgx_vepc_init() confuses you, I can rename it to sgx_virt_init().
Hi Sean,
Please let us know if you have comments.
Re: [PATCH v2 08/25] x86/sgx: Expose SGX architectural definitions to the kernel
In reply to Sean ChristophersonQuoted 10 lines · On Fri, 12 Mar 2021 13:58:40 -0800 Sean Christopherson wrote:
Quoted 9 lines · Kai Huang
From: Sean Christopherson <[email protected]>
Expose SGX architectural structures, as KVM will use many of the architectural constants and structs to virtualize SGX.
Name the new header file as
asm/sgx.h, rather thanasm/sgx_arch.h, to have single header to provide SGX facilities to share with other kernel componments.
Quoted 5 lines
Quoted 3 lines
Signed-off-by Sean Christopherson <[email protected]>Co-developed-by Kai Huang <[email protected]>Acked-by Dave Hansen <[email protected]>Same checkpatch warning. Probably doesn't matter.
Will change order to make checkpatch happy for this whole series.
Thanks for pointing out.
Quoted 44 lines
Quoted 44 lines
Signed-off-by Kai Huang <[email protected]>--- .../cpu/sgx/arch.h => include/asm/sgx.h} | 20 ++++++++++++++----- arch/x86/kernel/cpu/sgx/encl.c | 2 +- arch/x86/kernel/cpu/sgx/sgx.h | 2 +- tools/testing/selftests/sgx/defines.h | 2 +- 4 files changed, 18 insertions(+), 8 deletions(-) rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx.h} (95%)arch/x86/include/asm/sgx.h renamed from arch/x86/kernel/cpu/sgx/arch.h +9
-4@@ -2,15 +2,20 @@/** * Copyright(c) 2016-20 Intel Corporation. * -* Contains data structures defined by the SGX architecture. Data structures -* defined by the Linux software stack should not be placed here. +* Intel Software Guard Extensions (SGX) support. */ -#ifndef _ASM_X86_SGX_ARCH_H -#define _ASM_X86_SGX_ARCH_H +#ifndef _ASM_X86_SGX_H +#define _ASM_X86_SGX_H#include <linux/bits.h> #include <linux/types.h> +/* +* This file contains both data structures defined by SGX architecture and Linux +* defined software data structures and functions. The two should not be mixed +* together for better readibility. The architectural definitions come first. +*/ +/* The SGX specific CPUID function. */ #define SGX_CPUID 0x12/* EPC enumeration. */ @@ -337,4 +342,9 @@ struct sgx_sigstruct {
Quoted 8 lines
arch/x86/include/asm/sgx.h struct sgx_sigstruct { · line 347
Quoted 6 lines
arch/x86/include/asm/sgx.h renamed from arch/x86/kernel/cpu/sgx/arch.h +3
-1#define SGX_LAUNCH_TOKEN_SIZE 304 -#endif /* _ASM_X86_SGX_ARCH_H */ +/* +* Do not put any hardware-defined SGX structure representations below this +* line! Heh, which line? Yep, it's Friday afternoon...
Hmm.. I will change to below this comment. :)
Re: [PATCH v2 13/25] x86/sgx: Add helpers to expose ECREATE and EINIT to KVM
In reply to Kai HuangQuoted 112 lines · On Tue, 9 Mar 2021 14:39:46 +1300 Kai Huang wrote:
From: Sean Christopherson <[email protected]>
The host kernel must intercept ECREATE to impose policies on guests, and intercept EINIT to be able to write guest's virtual
SGX_LEPUBKEYHASHMSR values to hardware before running guest's EINIT so it can run correctly according to hardware behavior.Provide wrappers around
__ecreate()and__einit()to hide the ugliness of overloading the ENCLS return value to encode multiple error formats in a single int. KVM will trap-and-execute ECREATE and EINIT as part of SGX virtualization, and reflect ENCLS execution result to guest by setting up guest's GPRs, or on an exception, injecting the correct fault based on return value of__ecreate()and__einit().Use host userspace addresses (provided by KVM based on guest physical address of ENCLS parameters) to execute ENCLS/EINIT when possible. Accesses to both EPC and memory originating from ENCLS are subject to segmentation and paging mechanisms. It's also possible to generate kernel mappings for ENCLS parameters by resolving PFN but using
__uaccess_xx()is simpler.Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Kai Huang <[email protected]>---
v1->v2:- Tried to address Dave's comments:
- Refined comments around @secs insgx_virt_ecreate().
- Refined commit msg to explain why to use userspace address for ENCLS, instead of generating kernel mapping.
- Addedaccess_ok()on userspace addresses, and giveWARN()if check fails.--- arch/x86/include/asm/sgx.h | 7 +++ arch/x86/kernel/cpu/sgx/virt.c | 110 +++++++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+)arch/x86/include/asm/sgx.h +7
-0@@ -365,4 +365,11 @@ struct sgx_sigstruct {* line! */ +#ifdef CONFIG_X86_SGX_KVM +int sgx_virt_ecreate (struct sgx_pageinfo *pageinfo,void __user *secs, +int *trapnr); +int sgx_virt_einit (void __user *sigstruct,void __user *token, +void __user *secs,u64 *lepubkeyhash,int *trapnr); +#endif +#endif /* _ASM_X86_SGX_H */ arch/x86/kernel/cpu/sgx/virt.c +51
-0@@ -258,3 +258,113 @@ int __init sgx_vepc_init(void)return misc_register (&sgx_vepc_dev); } + +/** +* sgx_virt_ecreate() - Run ECREATE on behalf of guest +* @pageinfo: Pointer to PAGEINFO structure +* @secs: Userspace pointer to SECS page +* @trapnr: trap number injected to guest in case of ECREATE error +* +* Run ECREATE on behalf of guest after KVM traps ECREATE for the purpose +* of enforcing policies of guest's enclaves, and return the trap number +* which should be injected to guest in case of any ECREATE error. +* +* Return: +* - 0: ECREATE was successful. +* - -EFAULT: ECREATE returned error. +*/ +int sgx_virt_ecreate (struct sgx_pageinfo *pageinfo,void __user *secs, +int *trapnr) +{ +int ret; + +/* +* @secs is an untrusted, userspace-provided address. It comes from +* KVM and is assumed to be a valid pointer which points somewhere in +* userspace. This can fault and call SGX or other fault handlers when +* userspace mapping @secs doesn't exist. +* +* Add a WARN() to make sure @secs is already valid userspace pointer +* from caller (KVM), who should already have handled invalid pointer +* case (for instance, made by malicious guest). All other checks, +* such as alignment of @secs, are deferred to ENCLS itself. +*/ +WARN_ON_ONCE (!access_ok (PTR_ALIGN_DOWN (secs,PAGE_SIZE ),PAGE_SIZE )); +__uaccess_begin (); + ret =__ecreate (pageinfo, (void *)secs); +__uaccess_end (); + +if (encls_faulted (ret)) { + *trapnr =ENCLS_TRAPNR (ret); +return -EFAULT ; + } + +/* ECREATE doesn't return an error code, it faults or succeeds. */ +WARN_ON_ONCE (ret); +return 0 ; +} +EXPORT_SYMBOL_GPL (sgx_virt_ecreate); + +static int __sgx_virt_einit (void __user *sigstruct,void __user *token, +void __user *secs) +{ +int ret;
arch/x86/kernel/cpu/sgx/virt.c int __init sgx_vepc_init(void) · line 321
Quoted 10 lines
arch/x86/kernel/cpu/sgx/virt.c +10
-0+ +/* +* Make sure all userspace pointers from caller (KVM) are valid. +* All other checks deferred to ENCLS itself. Also see comment +* for @secs in sgx_virt_ecreate(). +*/ +WARN_ON_ONCE (!access_ok (PTR_ALIGN_DOWN (sigstruct,PAGE_SIZE ), +PAGE_SIZE ) || + !access_ok (PTR_ALIGN_DOWN (token,PAGE_SIZE ),PAGE_SIZE ) || + !access_ok (PTR_ALIGN_DOWN (secs,PAGE_SIZE ),PAGE_SIZE ));
I think I was overthinking about those access_ok()s. The align down is not needed, and size should just be the size of the structures. I'll change in next version.
One thing is SGX driver doesn't use EINITTOKEN, so it doesn't have 'struct sgx_einittoken {}' defined. The size of EINITTOKEN is fixed size (304 bytes), so I will add a macro definition SGX_EINITTOKEN_SIZE and use it.
Hi Sean, please let me know if you have any comments here (and this patch).
Thanks!
Quoted 52 lines
arch/x86/kernel/cpu/sgx/virt.c +49
-0+__uaccess_begin (); + ret =__einit ((void *)sigstruct, (void *)token, (void *)secs); +__uaccess_end (); + +return ret; +} + +/** +* sgx_virt_einit() - Run EINIT on behalf of guest +* @sigstruct: Userspace pointer to SIGSTRUCT structure +* @token: Userspace pointer to EINITTOKEN structure +* @secs: Userspace pointer to SECS page +* @lepubkeyhash: Pointer to guest's *virtual* SGX_LEPUBKEYHASH MSR +* values +* @trapnr: trap number injected to guest in case of EINIT error +* +* Run EINIT on behalf of guest after KVM traps EINIT. If SGX_LC is available +* in host, SGX driver may rewrite the hardware values at wish, therefore KVM +* needs to update hardware values to guest's virtual MSR values in order to +* ensure EINIT is executed with expected hardware values. +* +* Return: +* - 0: EINIT was successful. +* - -EFAULT: EINIT returned error. +*/ +int sgx_virt_einit (void __user *sigstruct,void __user *token, +void __user *secs,u64 *lepubkeyhash,int *trapnr) +{ +int ret; + +if (!boot_cpu_has (X86_FEATURE_SGX_LC )) { + ret =__sgx_virt_einit (sigstruct, token, secs); + }else { +preempt_disable (); + +sgx_update_lepubkeyhash (lepubkeyhash); + + ret =__sgx_virt_einit (sigstruct, token, secs); +preempt_enable (); + } + +if (encls_faulted (ret)) { + *trapnr =ENCLS_TRAPNR (ret); +return -EFAULT ; + } + +return ret; +} +EXPORT_SYMBOL_GPL (sgx_virt_einit);Signature · 1 line
2.29.2
Re: [PATCH v3 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Jarkko SakkinenQuoted 32 lines · On Sat, 13 Mar 2021 12:45:53 +0200 Jarkko Sakkinen wrote:
Quoted 31 lines · Sean Christopherson
Quoted 29 lines · Kai Huang
From: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original
sgx_free_epc_page()assgx_encl_free_epc_page()to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.Signed-off-by Jarkko Sakkinen <[email protected]>Co-developed-by Kai Huang <[email protected]>Acked-by Jarkko Sakkinen <[email protected]>Signed-off-by Kai Huang <[email protected]>---
v2->v3:- Fixed bug during copy/paste which results in SECS page and va pages are not correctly freed in
sgx_encl_release()(sorry for the mistake).
- Added Jarkko's Acked-by.
Quoted 8 lines
Quoted 4 lines
That Acked-by should either be dropped or moved above Co-developed-by to make checkpatch happy.
Reviewed-by Sean Christopherson <[email protected]>Oops, my bad. Yup, ack should be removed.
/Jarkko
Hi Jarkko,
Your reply of your concern of this patch to the cover-letter
https://lore.kernel.org/lkml/[email protected]/
reminds me to do more sanity check of whether removing EREMOVE in sgx_free_epc_page() will impact other code path or not, and I think sgx_encl_release() is not the only place should be changed:
- sgx_encl_shrink() needs to call sgx_encl_free_epc_page(), since when this is called, the VA page can be already valid -- there are other failures can trigger sgx_encl_shrink().
- sgx_encl_add_page() should call sgx_encl_free_epc_page() in "err_out_free:" label, since the EPC page can be already valid when error happened, i.e. when EEXTEND fails.
Other places should be OK per my check, but I'd prefer to just replacing all sgx_free_epc_page() call sites in driver with sgx_encl_free_epc_page(), with one exception: sgx_alloc_va_page(), which calls sgx_free_epc_page() when EPA fails, in which case EREMOVE is not required for sure.
Your idea, please?
Btw, introducing a driver wrapper of sgx_free_epc_page() does make sense to me, because virtualization has a counterpart in sgx/virt.c too.
Jarkko Sakkinen
maintainerRe: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Kai HuangQuoted 90 lines · Kai Huang
Quoted 73 lines · On Sun, 14 Mar 2021 17:27:18 +0200 Jarkko Sakkinen wrote:
Quoted 72 lines · Jarkko Sakkinen
Quoted 38 lines · Jarkko Sakkinen
Quoted 37 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();I love this code.
Reviewed-by Sean Christopherson <[email protected]>Quoted 8 lines
Quoted 5 lines
I'm still wondering why this code let's go through when
sgx_drv_init()succeeds andsgx_vepc_init()fails.The inline comment explains only the mirrored case (which does make sense).
I.e. if
sgx_drv_init()succeeds, I'd expect thatsgx_vepc_init()must succeed. Why expect legitly anything else?Apologies coming with these ideas at this point, but here is what this led me.
I think that the all this complexity comes from a bad code structure.
So, what is essentially happening here:
- We essentially want to make EPC always work.
- Driver optionally.So what this sums to is something like:
ret = sgx_epc_init(); if (ret) { pr_err("EPC initialization failed.\n"); return ret; } ret = sgx_drv_init(); if (ret) pr_info("Driver could not be initialized.\n"); /* continue */Quoted 7 lines
Quoted 2 lines
I.e. I think there should be a single EPC init, which does both EPC bootstrapping and vepc, and driver initialization comes after that.
In other words, from SGX point of view, the thing that KVM needs is to cut out EPC and driver part into different islands. How this is now implemented in the current patch set is half-way there but not yet what it should be.
Well conceptually, SGX virtualization and SGX driver are two independently functionalities can be enabled separately, although they both requires some come functionalities, such as
/dev/sgx_provision, which we have moved tosgx/main.cexactly for this purpose. THerefore, conceptually, it is bad to make assumption that, if SGX virtualization initialization succeeded, SGX driver must succeed -- we can potentially add more staff in SGX virtualization in the future..
Quoted 1 line
If the name
sgx_vepc_init()confuses you, I can rename it tosgx_virt_init().
I don't understand what would be the bad thing here. Can you open that up please? I'm neither capable of predicting the future...
Quoted 4 lines
Please let us know if you have comments.
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v3 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Kai HuangQuoted 48 lines · Kai Huang
Quoted 32 lines · On Sat, 13 Mar 2021 12:45:53 +0200 Jarkko Sakkinen wrote:
Quoted 31 lines · Sean Christopherson
Quoted 29 lines · Kai Huang
From: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original
sgx_free_epc_page()assgx_encl_free_epc_page()to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.Signed-off-by Jarkko Sakkinen <[email protected]>Co-developed-by Kai Huang <[email protected]>Acked-by Jarkko Sakkinen <[email protected]>Signed-off-by Kai Huang <[email protected]>---
v2->v3:- Fixed bug during copy/paste which results in SECS page and va pages are not correctly freed in
sgx_encl_release()(sorry for the mistake).
- Added Jarkko's Acked-by.Quoted 8 lines
Quoted 4 lines
That Acked-by should either be dropped or moved above Co-developed-by to make checkpatch happy.
Reviewed-by Sean Christopherson <[email protected]>Oops, my bad. Yup, ack should be removed.
/Jarkko
Hi Jarkko,
Your reply of your concern of this patch to the cover-letter
Quoted 7 lines
reminds me to do more sanity check of whether removing EREMOVE in
sgx_free_epc_page()will impact other code path or not, and I thinksgx_encl_release()is not the only place should be changed:-
sgx_encl_shrink()needs to callsgx_encl_free_epc_page(), since when this is called, the VA page can be already valid -- there are other failures can triggersgx_encl_shrink().
You right about this, good catch.
Shrink needs to always do EREMOVE as grow has done EPA, which changes EPC page state.
Quoted 3 lines
-
sgx_encl_add_page()should callsgx_encl_free_epc_page()in "err_out_free:" label, since the EPC page can be already valid when error happened, i.e. when EEXTEND fails.
Yes, correct, good work!
Quoted 4 lines
Other places should be OK per my check, but I'd prefer to just replacing all
sgx_free_epc_page()call sites in driver withsgx_encl_free_epc_page(), with one exception:sgx_alloc_va_page(), which callssgx_free_epc_page()when EPA fails, in which case EREMOVE is not required for sure.
I would not unless they require it.
Quoted 4 lines
Your idea, please?
Btw, introducing a driver wrapper of
sgx_free_epc_page()does make sense to me, because virtualization has a counterpart insgx/virt.ctoo.
It does make sense to use sgx_free_epc_page() everywhere where it's the right thing to call and here's why.
If there is some unrelated regression that causes EPC page not get uninitialized when it actually should, doing extra EREMOVE could mask those bugs. I.e. it can postpone a failure, which can make a bug harder to backtrace.
Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v3 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
Quoted 80 lines · Jarkko Sakkinen
Quoted 48 lines · Kai Huang
Quoted 32 lines · On Sat, 13 Mar 2021 12:45:53 +0200 Jarkko Sakkinen wrote:
Quoted 31 lines · Sean Christopherson
Quoted 29 lines · Kai Huang
From: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original
sgx_free_epc_page()assgx_encl_free_epc_page()to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.Signed-off-by Jarkko Sakkinen <[email protected]>Co-developed-by Kai Huang <[email protected]>Acked-by Jarkko Sakkinen <[email protected]>Signed-off-by Kai Huang <[email protected]>---
v2->v3:- Fixed bug during copy/paste which results in SECS page and va pages are not correctly freed in
sgx_encl_release()(sorry for the mistake).
- Added Jarkko's Acked-by.Quoted 8 lines
Quoted 4 lines
That Acked-by should either be dropped or moved above Co-developed-by to make checkpatch happy.
Reviewed-by Sean Christopherson <[email protected]>Oops, my bad. Yup, ack should be removed.
/Jarkko
Hi Jarkko,
Your reply of your concern of this patch to the cover-letter
Quoted 7 lines
reminds me to do more sanity check of whether removing EREMOVE in
sgx_free_epc_page()will impact other code path or not, and I thinksgx_encl_release()is not the only place should be changed:-
sgx_encl_shrink()needs to callsgx_encl_free_epc_page(), since when this is called, the VA page can be already valid -- there are other failures can triggersgx_encl_shrink().You right about this, good catch.
Shrink needs to always do EREMOVE as grow has done EPA, which changes EPC page state.
Quoted 3 lines
-
sgx_encl_add_page()should callsgx_encl_free_epc_page()in "err_out_free:" label, since the EPC page can be already valid when error happened, i.e. when EEXTEND fails.Yes, correct, good work!
Quoted 4 lines
Other places should be OK per my check, but I'd prefer to just replacing all
sgx_free_epc_page()call sites in driver withsgx_encl_free_epc_page(), with one exception:sgx_alloc_va_page(), which callssgx_free_epc_page()when EPA fails, in which case EREMOVE is not required for sure.I would not unless they require it.
Quoted 4 lines
Your idea, please?
Btw, introducing a driver wrapper of
sgx_free_epc_page()does make sense to me, because virtualization has a counterpart insgx/virt.ctoo.
Quoted 8 lines
It does make sense to use
sgx_free_epc_page()everywhere where it's the right thing to call and here's why.If there is some unrelated regression that causes EPC page not get uninitialized when it actually should, doing extra EREMOVE could mask those bugs. I.e. it can postpone a failure, which can make a bug harder to backtrace.
I.e. even though it is true that for correctly working code extra EREMOVE is nil functionality, it could change semantics for buggy code.
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Jarkko SakkinenQuoted 91 lines · Jarkko Sakkinen
Quoted 90 lines · Kai Huang
Quoted 73 lines · On Sun, 14 Mar 2021 17:27:18 +0200 Jarkko Sakkinen wrote:
Quoted 72 lines · Jarkko Sakkinen
Quoted 38 lines · Jarkko Sakkinen
Quoted 37 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();I love this code.
Reviewed-by Sean Christopherson <[email protected]>Quoted 8 lines
Quoted 5 lines
I'm still wondering why this code let's go through when
sgx_drv_init()succeeds andsgx_vepc_init()fails.The inline comment explains only the mirrored case (which does make sense).
I.e. if
sgx_drv_init()succeeds, I'd expect thatsgx_vepc_init()must succeed. Why expect legitly anything else?Apologies coming with these ideas at this point, but here is what this led me.
I think that the all this complexity comes from a bad code structure.
So, what is essentially happening here:
- We essentially want to make EPC always work.
- Driver optionally.So what this sums to is something like:
ret = sgx_epc_init(); if (ret) { pr_err("EPC initialization failed.\n"); return ret; } ret = sgx_drv_init(); if (ret) pr_info("Driver could not be initialized.\n"); /* continue */Quoted 7 lines
Quoted 2 lines
I.e. I think there should be a single EPC init, which does both EPC bootstrapping and vepc, and driver initialization comes after that.
In other words, from SGX point of view, the thing that KVM needs is to cut out EPC and driver part into different islands. How this is now implemented in the current patch set is half-way there but not yet what it should be.
Well conceptually, SGX virtualization and SGX driver are two independently functionalities can be enabled separately, although they both requires some come functionalities, such as
/dev/sgx_provision, which we have moved tosgx/main.cexactly for this purpose. THerefore, conceptually, it is bad to make assumption that, if SGX virtualization initialization succeeded, SGX driver must succeed -- we can potentially add more staff in SGX virtualization in the future..
Quoted 4 lines
Quoted 1 line
If the name
sgx_vepc_init()confuses you, I can rename it tosgx_virt_init().I don't understand what would be the bad thing here. Can you open that up please? I'm neither capable of predicting the future...
Right, so since vepc_init() does only just device file initialization the current function structure is fine. I totally forgot that sgx_drv_init() does not call EPC initialization when I wrote the above :-) We refactored during the inital cycle the driver so many times that I sometimes fix up thing, sorry about.
To meld this into code:
ret = sgx_vepc_init();
if (ret != -ENODEV) {
pr_err("vEPC initialization failed with %d.\n", ret);
return ret;
}
ret = sgx_drv_init();
if (ret != ENODEV)
pr_info("Driver initialization failed %d.\n", ret);This would also give more accurate information how far the initialization went.
/Jarkko
Re: [PATCH v3 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Jarkko SakkinenQuoted 84 lines · On Mon, 15 Mar 2021 15:19:32 +0200 Jarkko Sakkinen wrote:
Quoted 83 lines · Jarkko Sakkinen
Quoted 48 lines · Kai Huang
Quoted 32 lines · On Sat, 13 Mar 2021 12:45:53 +0200 Jarkko Sakkinen wrote:
Quoted 31 lines · Sean Christopherson
Quoted 29 lines · Kai Huang
From: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original
sgx_free_epc_page()assgx_encl_free_epc_page()to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.Signed-off-by Jarkko Sakkinen <[email protected]>Co-developed-by Kai Huang <[email protected]>Acked-by Jarkko Sakkinen <[email protected]>Signed-off-by Kai Huang <[email protected]>---
v2->v3:- Fixed bug during copy/paste which results in SECS page and va pages are not correctly freed in
sgx_encl_release()(sorry for the mistake).
- Added Jarkko's Acked-by.Quoted 8 lines
Quoted 4 lines
That Acked-by should either be dropped or moved above Co-developed-by to make checkpatch happy.
Reviewed-by Sean Christopherson <[email protected]>Oops, my bad. Yup, ack should be removed.
/Jarkko
Hi Jarkko,
Your reply of your concern of this patch to the cover-letter
Quoted 7 lines
reminds me to do more sanity check of whether removing EREMOVE in
sgx_free_epc_page()will impact other code path or not, and I thinksgx_encl_release()is not the only place should be changed:-
sgx_encl_shrink()needs to callsgx_encl_free_epc_page(), since when this is called, the VA page can be already valid -- there are other failures can triggersgx_encl_shrink().You right about this, good catch.
Shrink needs to always do EREMOVE as grow has done EPA, which changes EPC page state.
Quoted 3 lines
-
sgx_encl_add_page()should callsgx_encl_free_epc_page()in "err_out_free:" label, since the EPC page can be already valid when error happened, i.e. when EEXTEND fails.Yes, correct, good work!
Quoted 4 lines
Other places should be OK per my check, but I'd prefer to just replacing all
sgx_free_epc_page()call sites in driver withsgx_encl_free_epc_page(), with one exception:sgx_alloc_va_page(), which callssgx_free_epc_page()when EPA fails, in which case EREMOVE is not required for sure.I would not unless they require it.
Quoted 4 lines
Your idea, please?
Btw, introducing a driver wrapper of
sgx_free_epc_page()does make sense to me, because virtualization has a counterpart insgx/virt.ctoo.It does make sense to use
sgx_free_epc_page()everywhere where it's the right thing to call and here's why.
Quoted 8 lines
Quoted 5 lines
If there is some unrelated regression that causes EPC page not get uninitialized when it actually should, doing extra EREMOVE could mask those bugs. I.e. it can postpone a failure, which can make a bug harder to backtrace.
I.e. even though it is true that for correctly working code extra EREMOVE is nil functionality, it could change semantics for buggy code.
Thanks for feedback. Sorry I am not sure if I understand you. So if we don't want to bring functionality change, we need to replace sgx_free_epc_page() in all call sites with sgx_encl_free_epc_page(). To me for this patch only, it's better not to bring any functional change, so I intend to replace all (I now consider even leaving sgx_alloc_va_page() out is not good idea in *this* patch).
Or do you just want to replace sgx_free_epc_page() with sgx_encl_free_epc_page() in sgx_encl_shrink() and sgx_encl_add_page(), as I pointed above? In this way there will be functional change in this patch, and we need to explicitly explain why leaving others out is OK in commit message.
To me I prefer the former.
Re: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Jarkko SakkinenQuoted 92 lines · On Mon, 15 Mar 2021 15:51:17 +0200 Jarkko Sakkinen wrote:
Quoted 91 lines · Jarkko Sakkinen
Quoted 90 lines · Kai Huang
Quoted 73 lines · On Sun, 14 Mar 2021 17:27:18 +0200 Jarkko Sakkinen wrote:
Quoted 72 lines · Jarkko Sakkinen
Quoted 38 lines · Jarkko Sakkinen
Quoted 37 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();I love this code.
Reviewed-by Sean Christopherson <[email protected]>Quoted 8 lines
Quoted 5 lines
I'm still wondering why this code let's go through when
sgx_drv_init()succeeds andsgx_vepc_init()fails.The inline comment explains only the mirrored case (which does make sense).
I.e. if
sgx_drv_init()succeeds, I'd expect thatsgx_vepc_init()must succeed. Why expect legitly anything else?Apologies coming with these ideas at this point, but here is what this led me.
I think that the all this complexity comes from a bad code structure.
So, what is essentially happening here:
- We essentially want to make EPC always work.
- Driver optionally.So what this sums to is something like:
ret = sgx_epc_init(); if (ret) { pr_err("EPC initialization failed.\n"); return ret; } ret = sgx_drv_init(); if (ret) pr_info("Driver could not be initialized.\n"); /* continue */Quoted 7 lines
Quoted 2 lines
I.e. I think there should be a single EPC init, which does both EPC bootstrapping and vepc, and driver initialization comes after that.
In other words, from SGX point of view, the thing that KVM needs is to cut out EPC and driver part into different islands. How this is now implemented in the current patch set is half-way there but not yet what it should be.
Well conceptually, SGX virtualization and SGX driver are two independently functionalities can be enabled separately, although they both requires some come functionalities, such as
/dev/sgx_provision, which we have moved tosgx/main.cexactly for this purpose. THerefore, conceptually, it is bad to make assumption that, if SGX virtualization initialization succeeded, SGX driver must succeed -- we can potentially add more staff in SGX virtualization in the future..
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
If the name
sgx_vepc_init()confuses you, I can rename it tosgx_virt_init().I don't understand what would be the bad thing here. Can you open that up please? I'm neither capable of predicting the future...
Conceptually they are two different functionalities, and doesn't depend on each other. Why calling SGX driver initialization only when SGX virtualization succeeded?
We might want to add reclaiming EPC page (VMM EPC oversubscription) from KVM guest in the future, which may bring more initialization staff sgx_vepc_init(), and those new staff should not impact SGX driver.
I don't see your approach is any better, both from concept and flexibility.
Like I said, we can rename to sgx_virt_init() to be more generic, but I strongly disagree your approach.
Quoted 9 lines
Right, so since
vepc_init()does only just device file initialization the current function structure is fine. I totally forgot thatsgx_drv_init()does not call EPC initialization when I wrote the above :-) We refactored during the inital cycle the driver so many times that I sometimes fix up thing, sorry about.To meld this into code:
Quoted 9 lines
ret = sgx_vepc_init();
if (ret != -ENODEV) {
pr_err("vEPC initialization failed with %d.\n", ret);
return ret;
}
ret = sgx_drv_init();
if (ret != ENODEV)
pr_info("Driver initialization failed %d.\n", ret);Hmm.. Let's say an extreme case: misc_register() failed in sgx_vepc_init(), due to -ENOMEM. Then OOM kill gets involved, and kills bunch of apps. And then In this case, theoretically, misc_register() in sgx_drv_init() doesn't need to fail.
The point is really SGX driver and SGX virt are two independent functionalities, so don't make dependency on them, manually. Plus I don't see any benefit of your approach, but only cons.
Quoted 5 lines
This would also give more accurate information how far the initialization went.
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v3 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Kai HuangQuoted 101 lines · Kai Huang
Quoted 84 lines · On Mon, 15 Mar 2021 15:19:32 +0200 Jarkko Sakkinen wrote:
Quoted 83 lines · Jarkko Sakkinen
Quoted 48 lines · Kai Huang
Quoted 32 lines · On Sat, 13 Mar 2021 12:45:53 +0200 Jarkko Sakkinen wrote:
Quoted 31 lines · Sean Christopherson
Quoted 29 lines · Kai Huang
From: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original
sgx_free_epc_page()assgx_encl_free_epc_page()to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.Signed-off-by Jarkko Sakkinen <[email protected]>Co-developed-by Kai Huang <[email protected]>Acked-by Jarkko Sakkinen <[email protected]>Signed-off-by Kai Huang <[email protected]>---
v2->v3:- Fixed bug during copy/paste which results in SECS page and va pages are not correctly freed in
sgx_encl_release()(sorry for the mistake).
- Added Jarkko's Acked-by.Quoted 8 lines
Quoted 4 lines
That Acked-by should either be dropped or moved above Co-developed-by to make checkpatch happy.
Reviewed-by Sean Christopherson <[email protected]>Oops, my bad. Yup, ack should be removed.
/Jarkko
Hi Jarkko,
Your reply of your concern of this patch to the cover-letter
Quoted 7 lines
reminds me to do more sanity check of whether removing EREMOVE in
sgx_free_epc_page()will impact other code path or not, and I thinksgx_encl_release()is not the only place should be changed:-
sgx_encl_shrink()needs to callsgx_encl_free_epc_page(), since when this is called, the VA page can be already valid -- there are other failures can triggersgx_encl_shrink().You right about this, good catch.
Shrink needs to always do EREMOVE as grow has done EPA, which changes EPC page state.
Quoted 3 lines
-
sgx_encl_add_page()should callsgx_encl_free_epc_page()in "err_out_free:" label, since the EPC page can be already valid when error happened, i.e. when EEXTEND fails.Yes, correct, good work!
Quoted 4 lines
Other places should be OK per my check, but I'd prefer to just replacing all
sgx_free_epc_page()call sites in driver withsgx_encl_free_epc_page(), with one exception:sgx_alloc_va_page(), which callssgx_free_epc_page()when EPA fails, in which case EREMOVE is not required for sure.I would not unless they require it.
Quoted 4 lines
Your idea, please?
Btw, introducing a driver wrapper of
sgx_free_epc_page()does make sense to me, because virtualization has a counterpart insgx/virt.ctoo.It does make sense to use
sgx_free_epc_page()everywhere where it's the right thing to call and here's why.Quoted 8 lines
Quoted 5 lines
If there is some unrelated regression that causes EPC page not get uninitialized when it actually should, doing extra EREMOVE could mask those bugs. I.e. it can postpone a failure, which can make a bug harder to backtrace.
I.e. even though it is true that for correctly working code extra EREMOVE is nil functionality, it could change semantics for buggy code.
Thanks for feedback. Sorry I am not sure if I understand you. So if we don't want to bring functionality change, we need to replace
sgx_free_epc_page()in all call sites withsgx_encl_free_epc_page(). To me for this patch only, it's better not to bring any functional change, so I intend to replace all (I now consider even leavingsgx_alloc_va_page()out is not good idea in *this* patch).
Quoted 6 lines
Or do you just want to replace
sgx_free_epc_page()withsgx_encl_free_epc_page()insgx_encl_shrink()andsgx_encl_add_page(), as I pointed above? In this way there will be functional change in this patch, and we need to explicitly explain why leaving others out is OK in commit message.To me I prefer the former.
The original purpose of this patch was exactly to remove EREMOVE sgx_free_epc_page() and call it explicitly where it is required. That's why I introduced sgx_reset_epc_page(). So the latter was actually the goal of this patch at least when I did it. Now this is something completely different.
So, I don't consider myself author of this patch in any possible way, because this is not what I intended.
To move forward, for the next patch set version, you should change the author field as yourself, and remove all my tags, and I will review it. So you can work out this with former approach if you wish.
I.e. my ack/nak/etc. apply to this patch because it's not my code.
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Kai HuangQuoted 130 lines · Kai Huang
Quoted 92 lines · On Mon, 15 Mar 2021 15:51:17 +0200 Jarkko Sakkinen wrote:
Quoted 91 lines · Jarkko Sakkinen
Quoted 90 lines · Kai Huang
Quoted 73 lines · On Sun, 14 Mar 2021 17:27:18 +0200 Jarkko Sakkinen wrote:
Quoted 72 lines · Jarkko Sakkinen
Quoted 38 lines · Jarkko Sakkinen
Quoted 37 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();I love this code.
Reviewed-by Sean Christopherson <[email protected]>Quoted 8 lines
Quoted 5 lines
I'm still wondering why this code let's go through when
sgx_drv_init()succeeds andsgx_vepc_init()fails.The inline comment explains only the mirrored case (which does make sense).
I.e. if
sgx_drv_init()succeeds, I'd expect thatsgx_vepc_init()must succeed. Why expect legitly anything else?Apologies coming with these ideas at this point, but here is what this led me.
I think that the all this complexity comes from a bad code structure.
So, what is essentially happening here:
- We essentially want to make EPC always work.
- Driver optionally.So what this sums to is something like:
ret = sgx_epc_init(); if (ret) { pr_err("EPC initialization failed.\n"); return ret; } ret = sgx_drv_init(); if (ret) pr_info("Driver could not be initialized.\n"); /* continue */Quoted 7 lines
Quoted 2 lines
I.e. I think there should be a single EPC init, which does both EPC bootstrapping and vepc, and driver initialization comes after that.
In other words, from SGX point of view, the thing that KVM needs is to cut out EPC and driver part into different islands. How this is now implemented in the current patch set is half-way there but not yet what it should be.
Well conceptually, SGX virtualization and SGX driver are two independently functionalities can be enabled separately, although they both requires some come functionalities, such as
/dev/sgx_provision, which we have moved tosgx/main.cexactly for this purpose. THerefore, conceptually, it is bad to make assumption that, if SGX virtualization initialization succeeded, SGX driver must succeed -- we can potentially add more staff in SGX virtualization in the future..Quoted 4 lines
Quoted 4 lines
Quoted 1 line
If the name
sgx_vepc_init()confuses you, I can rename it tosgx_virt_init().I don't understand what would be the bad thing here. Can you open that up please? I'm neither capable of predicting the future...
Conceptually they are two different functionalities, and doesn't depend on each other. Why calling SGX driver initialization only when SGX virtualization succeeded?
We might want to add reclaiming EPC page (VMM EPC oversubscription) from KVM guest in the future, which may bring more initialization staff
sgx_vepc_init(), and those new staff should not impact SGX driver.I don't see your approach is any better, both from concept and flexibility.
Like I said, we can rename to
sgx_virt_init()to be more generic, but I strongly disagree your approach.Quoted 9 lines
Right, so since
vepc_init()does only just device file initialization the current function structure is fine. I totally forgot thatsgx_drv_init()does not call EPC initialization when I wrote the above :-) We refactored during the inital cycle the driver so many times that I sometimes fix up thing, sorry about.To meld this into code:
Quoted 9 lines
ret = sgx_vepc_init(); if (ret != -ENODEV) { pr_err("vEPC initialization failed with %d.\n", ret); return ret; } ret = sgx_drv_init(); if (ret != ENODEV) pr_info("Driver initialization failed %d.\n", ret);
Quoted 8 lines
Hmm.. Let's say an extreme case:
misc_register()failed insgx_vepc_init(), due to-ENOMEM. Then OOM kill gets involved, and kills bunch of apps. And then In this case, theoretically,misc_register()insgx_drv_init()doesn't need to fail.The point is really SGX driver and SGX virt are two independent functionalities, so don't make dependency on them, manually. Plus I don't see any benefit of your approach, but only cons.
The way I've understood it is that given that KVM can support SGX without FLC, vEPC should be available even if driver cannot be enabled.
This is also exactly what the short summary states.
"Initialize virtual EPC driver even when SGX driver is disabled"
It *does not* state:
"Initialize SGX driver even when vEPC driver is disabled"
Also, this is how I interpret the inline comment.
All this considered, the other direction is undocumented functionality.
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
Quoted 148 lines · Jarkko Sakkinen
Quoted 130 lines · Kai Huang
Quoted 92 lines · On Mon, 15 Mar 2021 15:51:17 +0200 Jarkko Sakkinen wrote:
Quoted 91 lines · Jarkko Sakkinen
Quoted 90 lines · Kai Huang
Quoted 73 lines · On Sun, 14 Mar 2021 17:27:18 +0200 Jarkko Sakkinen wrote:
Quoted 72 lines · Jarkko Sakkinen
Quoted 38 lines · Jarkko Sakkinen
Quoted 37 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();I love this code.
Reviewed-by Sean Christopherson <[email protected]>Quoted 8 lines
Quoted 5 lines
I'm still wondering why this code let's go through when
sgx_drv_init()succeeds andsgx_vepc_init()fails.The inline comment explains only the mirrored case (which does make sense).
I.e. if
sgx_drv_init()succeeds, I'd expect thatsgx_vepc_init()must succeed. Why expect legitly anything else?Apologies coming with these ideas at this point, but here is what this led me.
I think that the all this complexity comes from a bad code structure.
So, what is essentially happening here:
- We essentially want to make EPC always work.
- Driver optionally.So what this sums to is something like:
ret = sgx_epc_init(); if (ret) { pr_err("EPC initialization failed.\n"); return ret; } ret = sgx_drv_init(); if (ret) pr_info("Driver could not be initialized.\n"); /* continue */Quoted 7 lines
Quoted 2 lines
I.e. I think there should be a single EPC init, which does both EPC bootstrapping and vepc, and driver initialization comes after that.
In other words, from SGX point of view, the thing that KVM needs is to cut out EPC and driver part into different islands. How this is now implemented in the current patch set is half-way there but not yet what it should be.
Well conceptually, SGX virtualization and SGX driver are two independently functionalities can be enabled separately, although they both requires some come functionalities, such as
/dev/sgx_provision, which we have moved tosgx/main.cexactly for this purpose. THerefore, conceptually, it is bad to make assumption that, if SGX virtualization initialization succeeded, SGX driver must succeed -- we can potentially add more staff in SGX virtualization in the future..Quoted 4 lines
Quoted 4 lines
Quoted 1 line
If the name
sgx_vepc_init()confuses you, I can rename it tosgx_virt_init().I don't understand what would be the bad thing here. Can you open that up please? I'm neither capable of predicting the future...
Conceptually they are two different functionalities, and doesn't depend on each other. Why calling SGX driver initialization only when SGX virtualization succeeded?
We might want to add reclaiming EPC page (VMM EPC oversubscription) from KVM guest in the future, which may bring more initialization staff
sgx_vepc_init(), and those new staff should not impact SGX driver.I don't see your approach is any better, both from concept and flexibility.
Like I said, we can rename to
sgx_virt_init()to be more generic, but I strongly disagree your approach.Quoted 9 lines
Right, so since
vepc_init()does only just device file initialization the current function structure is fine. I totally forgot thatsgx_drv_init()does not call EPC initialization when I wrote the above :-) We refactored during the inital cycle the driver so many times that I sometimes fix up thing, sorry about.To meld this into code:
Quoted 9 lines
ret = sgx_vepc_init(); if (ret != -ENODEV) { pr_err("vEPC initialization failed with %d.\n", ret); return ret; } ret = sgx_drv_init(); if (ret != ENODEV) pr_info("Driver initialization failed %d.\n", ret);Quoted 8 lines
Hmm.. Let's say an extreme case:
misc_register()failed insgx_vepc_init(), due to-ENOMEM. Then OOM kill gets involved, and kills bunch of apps. And then In this case, theoretically,misc_register()insgx_drv_init()doesn't need to fail.The point is really SGX driver and SGX virt are two independent functionalities, so don't make dependency on them, manually. Plus I don't see any benefit of your approach, but only cons.
The way I've understood it is that given that KVM can support SGX without FLC, vEPC should be available even if driver cannot be enabled.
This is also exactly what the short summary states.
"Initialize virtual EPC driver even when SGX driver is disabled"
Quoted 7 lines
It *does not* state:
"Initialize SGX driver even when vEPC driver is disabled"
Also, this is how I interpret the inline comment.
All this considered, the other direction is undocumented functionality.
Also:
1. There is *zero* good practical reasons to support the "2nd direction". For KVM getting init'd with SGX, on the other hand, we have good practical reasons.
2. We can get something practically useful with simpler and more verbose code, i.e. better logging.
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v3 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Kai HuangQuoted 101 lines · Kai Huang
Quoted 84 lines · On Mon, 15 Mar 2021 15:19:32 +0200 Jarkko Sakkinen wrote:
Quoted 83 lines · Jarkko Sakkinen
Quoted 48 lines · Kai Huang
Quoted 32 lines · On Sat, 13 Mar 2021 12:45:53 +0200 Jarkko Sakkinen wrote:
Quoted 31 lines · Sean Christopherson
Quoted 29 lines · Kai Huang
From: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original
sgx_free_epc_page()assgx_encl_free_epc_page()to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.Signed-off-by Jarkko Sakkinen <[email protected]>Co-developed-by Kai Huang <[email protected]>Acked-by Jarkko Sakkinen <[email protected]>Signed-off-by Kai Huang <[email protected]>---
v2->v3:- Fixed bug during copy/paste which results in SECS page and va pages are not correctly freed in
sgx_encl_release()(sorry for the mistake).
- Added Jarkko's Acked-by.Quoted 8 lines
Quoted 4 lines
That Acked-by should either be dropped or moved above Co-developed-by to make checkpatch happy.
Reviewed-by Sean Christopherson <[email protected]>Oops, my bad. Yup, ack should be removed.
/Jarkko
Hi Jarkko,
Your reply of your concern of this patch to the cover-letter
Quoted 7 lines
reminds me to do more sanity check of whether removing EREMOVE in
sgx_free_epc_page()will impact other code path or not, and I thinksgx_encl_release()is not the only place should be changed:-
sgx_encl_shrink()needs to callsgx_encl_free_epc_page(), since when this is called, the VA page can be already valid -- there are other failures can triggersgx_encl_shrink().You right about this, good catch.
Shrink needs to always do EREMOVE as grow has done EPA, which changes EPC page state.
Quoted 3 lines
-
sgx_encl_add_page()should callsgx_encl_free_epc_page()in "err_out_free:" label, since the EPC page can be already valid when error happened, i.e. when EEXTEND fails.Yes, correct, good work!
Quoted 4 lines
Other places should be OK per my check, but I'd prefer to just replacing all
sgx_free_epc_page()call sites in driver withsgx_encl_free_epc_page(), with one exception:sgx_alloc_va_page(), which callssgx_free_epc_page()when EPA fails, in which case EREMOVE is not required for sure.I would not unless they require it.
Quoted 4 lines
Your idea, please?
Btw, introducing a driver wrapper of
sgx_free_epc_page()does make sense to me, because virtualization has a counterpart insgx/virt.ctoo.It does make sense to use
sgx_free_epc_page()everywhere where it's the right thing to call and here's why.Quoted 8 lines
Quoted 5 lines
If there is some unrelated regression that causes EPC page not get uninitialized when it actually should, doing extra EREMOVE could mask those bugs. I.e. it can postpone a failure, which can make a bug harder to backtrace.
I.e. even though it is true that for correctly working code extra EREMOVE is nil functionality, it could change semantics for buggy code.
Thanks for feedback. Sorry I am not sure if I understand you. So if we don't want to bring functionality change, we need to replace
sgx_free_epc_page()in all call sites withsgx_encl_free_epc_page(). To me for this patch only, it's better not to bring any functional change, so I intend to replace all (I now consider even leavingsgx_alloc_va_page()out is not good idea in *this* patch).
Quoted 6 lines
Or do you just want to replace
sgx_free_epc_page()withsgx_encl_free_epc_page()insgx_encl_shrink()andsgx_encl_add_page(), as I pointed above? In this way there will be functional change in this patch, and we need to explicitly explain why leaving others out is OK in commit message.To me I prefer the former.
But yes, I'm cool with your preference and I do get your argument, I just need to review it, and do not consider it as my patch :-)
/Jarkko
Re: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Jarkko SakkinenQuoted 145 lines · On Tue, 16 Mar 2021 01:08:44 +0200 Jarkko Sakkinen wrote:
Quoted 144 lines · Jarkko Sakkinen
Quoted 130 lines · Kai Huang
Quoted 92 lines · On Mon, 15 Mar 2021 15:51:17 +0200 Jarkko Sakkinen wrote:
Quoted 91 lines · Jarkko Sakkinen
Quoted 90 lines · Kai Huang
Quoted 73 lines · On Sun, 14 Mar 2021 17:27:18 +0200 Jarkko Sakkinen wrote:
Quoted 72 lines · Jarkko Sakkinen
Quoted 38 lines · Jarkko Sakkinen
Quoted 37 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 20 lines · Kai Huang
Modify
sgx_init()to always try to initialize the virtual EPC driver, even if the SGX driver is disabled. The SGX driver might be disabled if SGX Launch Control is in locked mode, or not supported in the hardware at all. This allows (non-Linux) guests that support non-LC configurations to use SGX.Acked-by Dave Hansen <[email protected]>Signed-off-by Kai Huang <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -712,7 +712,15 @@ static int __init sgx_init(void) goto err_page_cache; }arch/x86/kernel/cpu/sgx/main.c static int __init sgx_init(void) · line 723
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +9
-1- ret =sgx_drv_init (); +/* +* Always try to initialize the native *and* KVM drivers. +* The KVM driver is less picky than the native one and +* can function if the native one is not supported on the +* current system or fails to initialize. +* +* Error out only if both fail to initialize. +*/ + ret = !!sgx_drv_init () & !!sgx_vepc_init ();I love this code.
Reviewed-by Sean Christopherson <[email protected]>Quoted 8 lines
Quoted 5 lines
I'm still wondering why this code let's go through when
sgx_drv_init()succeeds andsgx_vepc_init()fails.The inline comment explains only the mirrored case (which does make sense).
I.e. if
sgx_drv_init()succeeds, I'd expect thatsgx_vepc_init()must succeed. Why expect legitly anything else?Apologies coming with these ideas at this point, but here is what this led me.
I think that the all this complexity comes from a bad code structure.
So, what is essentially happening here:
- We essentially want to make EPC always work.
- Driver optionally.So what this sums to is something like:
ret = sgx_epc_init(); if (ret) { pr_err("EPC initialization failed.\n"); return ret; } ret = sgx_drv_init(); if (ret) pr_info("Driver could not be initialized.\n"); /* continue */Quoted 7 lines
Quoted 2 lines
I.e. I think there should be a single EPC init, which does both EPC bootstrapping and vepc, and driver initialization comes after that.
In other words, from SGX point of view, the thing that KVM needs is to cut out EPC and driver part into different islands. How this is now implemented in the current patch set is half-way there but not yet what it should be.
Well conceptually, SGX virtualization and SGX driver are two independently functionalities can be enabled separately, although they both requires some come functionalities, such as
/dev/sgx_provision, which we have moved tosgx/main.cexactly for this purpose. THerefore, conceptually, it is bad to make assumption that, if SGX virtualization initialization succeeded, SGX driver must succeed -- we can potentially add more staff in SGX virtualization in the future..Quoted 4 lines
Quoted 4 lines
Quoted 1 line
If the name
sgx_vepc_init()confuses you, I can rename it tosgx_virt_init().I don't understand what would be the bad thing here. Can you open that up please? I'm neither capable of predicting the future...
Conceptually they are two different functionalities, and doesn't depend on each other. Why calling SGX driver initialization only when SGX virtualization succeeded?
We might want to add reclaiming EPC page (VMM EPC oversubscription) from KVM guest in the future, which may bring more initialization staff
sgx_vepc_init(), and those new staff should not impact SGX driver.I don't see your approach is any better, both from concept and flexibility.
Like I said, we can rename to
sgx_virt_init()to be more generic, but I strongly disagree your approach.Quoted 9 lines
Right, so since
vepc_init()does only just device file initialization the current function structure is fine. I totally forgot thatsgx_drv_init()does not call EPC initialization when I wrote the above :-) We refactored during the inital cycle the driver so many times that I sometimes fix up thing, sorry about.To meld this into code:
Quoted 9 lines
ret = sgx_vepc_init(); if (ret != -ENODEV) { pr_err("vEPC initialization failed with %d.\n", ret); return ret; } ret = sgx_drv_init(); if (ret != ENODEV) pr_info("Driver initialization failed %d.\n", ret);Quoted 8 lines
Hmm.. Let's say an extreme case:
misc_register()failed insgx_vepc_init(), due to-ENOMEM. Then OOM kill gets involved, and kills bunch of apps. And then In this case, theoretically,misc_register()insgx_drv_init()doesn't need to fail.The point is really SGX driver and SGX virt are two independent functionalities, so don't make dependency on them, manually. Plus I don't see any benefit of your approach, but only cons.
The way I've understood it is that given that KVM can support SGX without FLC, vEPC should be available even if driver cannot be enabled.
Quoted 7 lines
Quoted 7 lines
This is also exactly what the short summary states.
"Initialize virtual EPC driver even when SGX driver is disabled"
It *does not* state:
"Initialize SGX driver even when vEPC driver is disabled"
OK. The patch title can be improved. How about:
"Initialize SGX driver and virtual EPC driver independently"
?
Quoted 4 lines
Quoted 4 lines
Also, this is how I interpret the inline comment.
All this considered, the other direction is undocumented functionality.
OK. How about below?
/*
* Always try to initialize the native *and* KVM drivers. They are independent
* functionalities and one can be initialized even when the other is not
* supported or fails to initialize.
*/
The explicit saying of "not supported or fails to initialize" was requested by you -- you wanted to distinguish -ENODEV with other error codes.
Quoted 6 lines
Also:
1. There is *zero* good practical reasons to support the "2nd direction". For KVM getting init'd with SGX, on the other hand, we have good practical reasons.
Why there's *zero* good practical reasons? With initializing them independently, people don't need to worry about *internal* of sgx_vepc_init() and sgx_drv_init(), but just need pay attention of the logic that they are two independent functionalities. Being able to initialize them independently is much more clear and easier to understand. And like I said, in this way it is more flexible to extend -- for instance, we may add more staff to support VMM EPC oversubscription. So why there is *zero* good practical reasons?
Btw, there are customers that want to just use KVM SGX, but not SGX driver in host, for which people may want to add separate CONFIG option, say, CONFIG_X86_SGX_DRIVER, to be able to disable/enable SGX driver code, just like CONFIG_X86_SGX_KVM. Make them independent logically just make things more clear.
Quoted 2 lines
2. We can get something practically useful with simpler and more verbose code, i.e. better logging.
I can add error msg in sgx_vepc_init() upon misc_register() if you want.
Quoted 2 lines
/Jarkko
Re: [PATCH v3 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
In reply to Jarkko SakkinenQuoted 115 lines · On Tue, 16 Mar 2021 00:59:31 +0200 Jarkko Sakkinen wrote:
Quoted 101 lines · Kai Huang
Quoted 84 lines · On Mon, 15 Mar 2021 15:19:32 +0200 Jarkko Sakkinen wrote:
Quoted 83 lines · Jarkko Sakkinen
Quoted 48 lines · Kai Huang
Quoted 32 lines · On Sat, 13 Mar 2021 12:45:53 +0200 Jarkko Sakkinen wrote:
Quoted 31 lines · Sean Christopherson
Quoted 29 lines · Kai Huang
From: Jarkko Sakkinen <[email protected]>
EREMOVE takes a page and removes any association between that page and an enclave. It must be run on a page before it can be added into another enclave. Currently, EREMOVE is run as part of pages being freed into the SGX page allocator. It is not expected to fail.
KVM does not track how guest pages are used, which means that SGX virtualization use of EREMOVE might fail.
Break out the EREMOVE call from the SGX page allocator. This will allow the SGX virtualization code to use the allocator directly. (SGX/KVM will also introduce a more permissive EREMOVE helper).
Implement original
sgx_free_epc_page()assgx_encl_free_epc_page()to be more specific that it is used to free EPC page assigned to one enclave. Print an error message when EREMOVE fails to explicitly call out EPC page is leaked, and requires machine reboot to get leaked pages back.Signed-off-by Jarkko Sakkinen <[email protected]>Co-developed-by Kai Huang <[email protected]>Acked-by Jarkko Sakkinen <[email protected]>Signed-off-by Kai Huang <[email protected]>---
v2->v3:- Fixed bug during copy/paste which results in SECS page and va pages are not correctly freed in
sgx_encl_release()(sorry for the mistake).
- Added Jarkko's Acked-by.Quoted 8 lines
Quoted 4 lines
That Acked-by should either be dropped or moved above Co-developed-by to make checkpatch happy.
Reviewed-by Sean Christopherson <[email protected]>Oops, my bad. Yup, ack should be removed.
/Jarkko
Hi Jarkko,
Your reply of your concern of this patch to the cover-letter
Quoted 7 lines
reminds me to do more sanity check of whether removing EREMOVE in
sgx_free_epc_page()will impact other code path or not, and I thinksgx_encl_release()is not the only place should be changed:-
sgx_encl_shrink()needs to callsgx_encl_free_epc_page(), since when this is called, the VA page can be already valid -- there are other failures can triggersgx_encl_shrink().You right about this, good catch.
Shrink needs to always do EREMOVE as grow has done EPA, which changes EPC page state.
Quoted 3 lines
-
sgx_encl_add_page()should callsgx_encl_free_epc_page()in "err_out_free:" label, since the EPC page can be already valid when error happened, i.e. when EEXTEND fails.Yes, correct, good work!
Quoted 4 lines
Other places should be OK per my check, but I'd prefer to just replacing all
sgx_free_epc_page()call sites in driver withsgx_encl_free_epc_page(), with one exception:sgx_alloc_va_page(), which callssgx_free_epc_page()when EPA fails, in which case EREMOVE is not required for sure.I would not unless they require it.
Quoted 4 lines
Your idea, please?
Btw, introducing a driver wrapper of
sgx_free_epc_page()does make sense to me, because virtualization has a counterpart insgx/virt.ctoo.It does make sense to use
sgx_free_epc_page()everywhere where it's the right thing to call and here's why.Quoted 8 lines
Quoted 5 lines
If there is some unrelated regression that causes EPC page not get uninitialized when it actually should, doing extra EREMOVE could mask those bugs. I.e. it can postpone a failure, which can make a bug harder to backtrace.
I.e. even though it is true that for correctly working code extra EREMOVE is nil functionality, it could change semantics for buggy code.
Thanks for feedback. Sorry I am not sure if I understand you. So if we don't want to bring functionality change, we need to replace
sgx_free_epc_page()in all call sites withsgx_encl_free_epc_page(). To me for this patch only, it's better not to bring any functional change, so I intend to replace all (I now consider even leavingsgx_alloc_va_page()out is not good idea in *this* patch).Quoted 6 lines
Or do you just want to replace
sgx_free_epc_page()withsgx_encl_free_epc_page()insgx_encl_shrink()andsgx_encl_add_page(), as I pointed above? In this way there will be functional change in this patch, and we need to explicitly explain why leaving others out is OK in commit message.To me I prefer the former.
The original purpose of this patch was exactly to remove EREMOVE
sgx_free_epc_page()and call it explicitly where it is required. That's why I introducedsgx_reset_epc_page(). So the latter was actually the goal of this patch at least when I did it. Now this is something completely different.
Quoted 8 lines
So, I don't consider myself author of this patch in any possible way, because this is not what I intended.
To move forward, for the next patch set version, you should change the author field as yourself, and remove all my tags, and I will review it. So you can work out this with former approach if you wish.
I.e. my ack/nak/etc. apply to this patch because it's not my code.
OK. Thanks.
Sean Christopherson
maintainerRe: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Jarkko SakkinenQuoted 13 lines · Jarkko Sakkinen
Quoted 12 lines · Jarkko Sakkinen
The way I've understood it is that given that KVM can support SGX without FLC, vEPC should be available even if driver cannot be enabled.
This is also exactly what the short summary states.
"Initialize virtual EPC driver even when SGX driver is disabled"
It *does not* state:
"Initialize SGX driver even when vEPC driver is disabled"
Quoted 7 lines
Quoted 3 lines
Also, this is how I interpret the inline comment.
All this considered, the other direction is undocumented functionality.
Also:
1. There is *zero* good practical reasons to support the "2nd direction".
Uh, yes there is. CONFIG_KVM_INTEL=n and X86_FEATURE_VMX=n, either of which will cause vEPC initialization to fail. The former is obvious, the latter is possible via BIOS configuration.
Quoted 6 lines
For KVM getting init'd with SGX, on the other hand, we have good practical reasons.
2. We can get something practically useful with simpler and more verbose code, i.e. better logging./Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Kai HuangQuoted 183 lines · Kai Huang
Quoted 145 lines · On Tue, 16 Mar 2021 01:08:44 +0200 Jarkko Sakkinen wrote:
Quoted 144 lines · Jarkko Sakkinen
Quoted 130 lines · Kai Huang
Quoted 92 lines · On Mon, 15 Mar 2021 15:51:17 +0200 Jarkko Sakkinen wrote:
Quoted 91 lines · Jarkko Sakkinen
Quoted 90 lines · Kai Huang
Quoted 73 lines · On Sun, 14 Mar 2021 17:27:18 +0200 Jarkko Sakkinen wrote:
Quoted 72 lines · Jarkko Sakkinen
Quoted 38 lines · Jarkko Sakkinen
Quoted 37 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 30 lines · Kai Huang
Modify sgx_init() to always try to initialize the virtual EPC driver,
even if the SGX driver is disabled. The SGX driver might be disabled
if SGX Launch Control is in locked mode, or not supported in the
hardware at all. This allows (non-Linux) guests that support non-LC
configurations to use SGX.Acked-by: Dave Hansen <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
---
arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
index 44fe91a5bfb3..8c922e68274d 100644
--- a/arch/x86/kernel/cpu/sgx/main.c
+++ b/arch/x86/kernel/cpu/sgx/main.c
@@ -712,7 +712,15 @@ static int __init sgx_init(void)
goto err_page_cache;
}- ret = sgx_drv_init();
+ /*
+ * Always try to initialize the native *and* KVM drivers.
+ * The KVM driver is less picky than the native one and
+ * can function if the native one is not supported on the
+ * current system or fails to initialize.
+ *
+ * Error out only if both fail to initialize.
+ */
+ ret = !!sgx_drv_init() & !!sgx_vepc_init();I love this code.
Reviewed-by Sean Christopherson <[email protected]>Quoted 8 lines
Quoted 5 lines
I'm still wondering why this code let's go through when
sgx_drv_init()succeeds andsgx_vepc_init()fails.The inline comment explains only the mirrored case (which does make sense).
I.e. if
sgx_drv_init()succeeds, I'd expect thatsgx_vepc_init()must succeed. Why expect legitly anything else?Apologies coming with these ideas at this point, but here is what this led me.
I think that the all this complexity comes from a bad code structure.
So, what is essentially happening here:
- We essentially want to make EPC always work.
- Driver optionally.So what this sums to is something like:
ret = sgx_epc_init(); if (ret) { pr_err("EPC initialization failed.\n"); return ret; } ret = sgx_drv_init(); if (ret) pr_info("Driver could not be initialized.\n"); /* continue */Quoted 7 lines
Quoted 2 lines
I.e. I think there should be a single EPC init, which does both EPC bootstrapping and vepc, and driver initialization comes after that.
In other words, from SGX point of view, the thing that KVM needs is to cut out EPC and driver part into different islands. How this is now implemented in the current patch set is half-way there but not yet what it should be.
Well conceptually, SGX virtualization and SGX driver are two independently functionalities can be enabled separately, although they both requires some come functionalities, such as
/dev/sgx_provision, which we have moved tosgx/main.cexactly for this purpose. THerefore, conceptually, it is bad to make assumption that, if SGX virtualization initialization succeeded, SGX driver must succeed -- we can potentially add more staff in SGX virtualization in the future..Quoted 4 lines
Quoted 4 lines
Quoted 1 line
If the name
sgx_vepc_init()confuses you, I can rename it tosgx_virt_init().I don't understand what would be the bad thing here. Can you open that up please? I'm neither capable of predicting the future...
Conceptually they are two different functionalities, and doesn't depend on each other. Why calling SGX driver initialization only when SGX virtualization succeeded?
We might want to add reclaiming EPC page (VMM EPC oversubscription) from KVM guest in the future, which may bring more initialization staff
sgx_vepc_init(), and those new staff should not impact SGX driver.I don't see your approach is any better, both from concept and flexibility.
Like I said, we can rename to
sgx_virt_init()to be more generic, but I strongly disagree your approach.Quoted 9 lines
Right, so since
vepc_init()does only just device file initialization the current function structure is fine. I totally forgot thatsgx_drv_init()does not call EPC initialization when I wrote the above :-) We refactored during the inital cycle the driver so many times that I sometimes fix up thing, sorry about.To meld this into code:
Quoted 9 lines
ret = sgx_vepc_init(); if (ret != -ENODEV) { pr_err("vEPC initialization failed with %d.\n", ret); return ret; } ret = sgx_drv_init(); if (ret != ENODEV) pr_info("Driver initialization failed %d.\n", ret);Quoted 8 lines
Hmm.. Let's say an extreme case:
misc_register()failed insgx_vepc_init(), due to-ENOMEM. Then OOM kill gets involved, and kills bunch of apps. And then In this case, theoretically,misc_register()insgx_drv_init()doesn't need to fail.The point is really SGX driver and SGX virt are two independent functionalities, so don't make dependency on them, manually. Plus I don't see any benefit of your approach, but only cons.
The way I've understood it is that given that KVM can support SGX without FLC, vEPC should be available even if driver cannot be enabled.
Quoted 7 lines
Quoted 7 lines
This is also exactly what the short summary states.
"Initialize virtual EPC driver even when SGX driver is disabled"
It *does not* state:
"Initialize SGX driver even when vEPC driver is disabled"
OK. The patch title can be improved. How about:
"Initialize SGX driver and virtual EPC driver independently"
?
Quoted 4 lines
Quoted 4 lines
Also, this is how I interpret the inline comment.
All this considered, the other direction is undocumented functionality.
OK. How about below?
/*
* Always try to initialize the native *and* KVM drivers. They are independent
* functionalities and one can be initialized even when the other is not
* supported or fails to initialize.
*/The explicit saying of "not supported or fails to initialize" was requested by you -- you wanted to distinguish
-ENODEVwith other error codes.Quoted 6 lines
Also:
1. There is *zero* good practical reasons to support the "2nd direction". For KVM getting init'd with SGX, on the other hand, we have good practical reasons.
Quoted 8 lines
Why there's *zero* good practical reasons? With initializing them independently, people don't need to worry about *internal* of
sgx_vepc_init()andsgx_drv_init(), but just need pay attention of the logic that they are two independent functionalities. Being able to initialize them independently is much more clear and easier to understand. And like I said, in this way it is more flexible to extend -- for instance, we may add more staff to support VMM EPC oversubscription. So why there is *zero* good practical reasons?
Then things would be reconsidered.
Quoted 6 lines
Btw, there are customers that want to just use KVM SGX, but not SGX driver in host, for which people may want to add separate CONFIG option, say,
CONFIG_X86_SGX_DRIVER, to be able to disable/enable SGX driver code, just likeCONFIG_X86_SGX_KVM. Make them independent logically just make things more clear.
Why?
Quoted 9 lines
Quoted 3 lines
2. We can get something practically useful with simpler and more verbose code, i.e. better logging.
I can add error msg in
sgx_vepc_init()uponmisc_register()if you want.Quoted 2 lines
/Jarkko
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
In reply to Sean ChristophersonQuoted 18 lines · Sean Christopherson
Quoted 17 lines · Jarkko Sakkinen
Quoted 15 lines · Jarkko Sakkinen
The way I've understood it is that given that KVM can support SGX without FLC, vEPC should be available even if driver cannot be enabled.
This is also exactly what the short summary states.
"Initialize virtual EPC driver even when SGX driver is disabled"
It *does not* state:
"Initialize SGX driver even when vEPC driver is disabled"
Also, this is how I interpret the inline comment.
All this considered, the other direction is undocumented functionality.
Quoted 7 lines
Quoted 3 lines
Also:
1. There is *zero* good practical reasons to support the "2nd direction".
Uh, yes there is.
CONFIG_KVM_INTEL=nand X86_FEATURE_VMX=n, either of which will cause vEPC initialization to fail. The former is obvious, the latter is possible via BIOS configuration.
Hmm... So you make the checks as if ret != -ENODEV? That's the sane way to deal with that situation IMHO.
/Jarkko
Re: [PATCH v2 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
Quoted 23 lines · On Tue, 16 Mar 2021 14:46:05 +0200 Jarkko Sakkinen wrote:
Quoted 22 lines · Sean Christopherson
Quoted 20 lines · Jarkko Sakkinen
Quoted 8 lines · Jarkko Sakkinen
The way I've understood it is that given that KVM can support SGX without FLC, vEPC should be available even if driver cannot be enabled.
This is also exactly what the short summary states.
"Initialize virtual EPC driver even when SGX driver is disabled"
Quoted 7 lines
It *does not* state:
"Initialize SGX driver even when vEPC driver is disabled"
Also, this is how I interpret the inline comment.
All this considered, the other direction is undocumented functionality.
Also:
1. There is *zero* good practical reasons to support the "2nd direction".
Quoted 8 lines
Quoted 3 lines
Uh, yes there is.
CONFIG_KVM_INTEL=nand X86_FEATURE_VMX=n, either of which will cause vEPC initialization to fail. The former is obvious, the latter is possible via BIOS configuration.Hmm... So you make the checks as if ret !=
-ENODEV? That's the sane way to deal with that situation IMHO./Jarkko
OK. I actually wrote the code to show your idea:
- ret = sgx_drv_init();
- if (ret)
+ /*
+ * Only continue to initialize SGX driver when SGX virtualization
+ * initialization is successful, or is not supported (-ENODEV), since
+ * any other initialization failure means SGX driver is unlikely to be
+ * initialized successfully.
+ */
+ ret = sgx_vepc_init();
+ if (ret && ret != -ENODEV)
goto err_kthread;+ if (sgx_drv_init()) {
+ /*
+ * Cleanup when *both* SGX virtualization and SGX driver are + * not enabled, due to either not supported (-ENODEV), or + * somehow fail to initialize. + */
+ if (ret && ret != -ENODEV)
+ goto err_kthread;
+ }
+
return 0;
err_kthread:
Is this the code you want?