Skip to content

RFC KVM SGX virtualization support

Kai Huang kvm linux-sgx 10 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 posting 2/211 patches 12 repliesApplied
  2. v4 posting 2/211 patches 13 repliesSuperseded
  3. v3 posting 2/225 patches 79 repliesApplied
Earlier versions · 7
VersionPostedPatchesRepliesStatus
v2 posting 2/225 patches33 repliesSuperseded
v6 RFC Latest25 patches9 repliesIn review
v5 RFC posting 1/226 patches37 repliesSuperseded
v4 RFC posting 1/226 patches22 repliesSuperseded
v3 RFC posting 1/227 patches127 repliesSuperseded
v2 RFC posting 1/226 patches64 repliesSuperseded
v1 RFC23 patches87 repliesSuperseded

Changes in v6 from the cover letter

- Main change for x86 part patches is asm/sgx_arch.h is removed and merged to asm/sgx.h, based on Boris's suggestion, and this caused some minor changes to other patches due to header file name being different.
- For KVM part patches, removed patch (KVM: VMX: Convert vcpu_vmx.exit_reason to a union), since it has already been merged to upstream KVM. Also rebased rest of KVM part patches based on latest KVM upstream code.

Patches in v6 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/25x86/cpufeatures: Make SGX_LC feature bit depend on SGX bitarch/x86/kernel/cpu/cpuid-deps.c arch/x86/kernel/cpu/feat_ctl.c+4 -90 replies
2/25x86/cpufeatures: Add SGX1 and SGX2 sub-featuresarch/x86/include/asm/cpufeatures.h arch/x86/kernel/cpu/cpuid-deps.c arch/x86/kernel/cpu/scattered.c+6 -00 replies
3/25x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()arch/x86/kernel/cpu/sgx/encl.c arch/x86/kernel/cpu/sgx/main.c+27 -115 replies
4/25x86/sgx: Add SGX_CHILD_PRESENT hardware error codearch/x86/kernel/cpu/sgx/arch.h+2 -00 replies
5/25x86/sgx: Introduce virtual EPC for use by KVM guestsarch/x86/Kconfig arch/x86/kernel/cpu/sgx/Makefile arch/x86/kernel/cpu/sgx/sgx.h +1 more+282 -00 replies
6/25x86/cpu/intel: Allow SGX virtualization without Launch Control supportarch/x86/kernel/cpu/feat_ctl.c+44 -130 replies
7/25x86/sgx: Initialize virtual EPC driver even when SGX driver is disabledarch/x86/kernel/cpu/sgx/main.c+9 -10 replies
8/25x86/sgx: Expose SGX architectural definitions to the kernelarch/x86/include/asm/sgx.h arch/x86/kernel/cpu/sgx/encl.c arch/x86/kernel/cpu/sgx/sgx.h +1 more+18 -81 reply
9/25x86/sgx: Move ENCLS leaf definitions to sgx.harch/x86/include/asm/sgx.h arch/x86/kernel/cpu/sgx/encls.h+15 -150 replies
10/25x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT)arch/x86/include/asm/sgx.h+3 -00 replies
11/25x86/sgx: Add encls_faulted() helperarch/x86/kernel/cpu/sgx/encls.h arch/x86/kernel/cpu/sgx/ioctl.c+15 -20 replies
12/25x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRsarch/x86/kernel/cpu/sgx/ioctl.c arch/x86/kernel/cpu/sgx/main.c arch/x86/kernel/cpu/sgx/sgx.h+19 -30 replies
13/25x86/sgx: Add helpers to expose ECREATE and EINIT to KVMarch/x86/include/asm/sgx.h arch/x86/kernel/cpu/sgx/virt.c+102 -02 replies
14/25x86/sgx: Move provisioning device creation out of SGX driverarch/x86/include/asm/sgx.h arch/x86/kernel/cpu/sgx/driver.c arch/x86/kernel/cpu/sgx/ioctl.c +1 more+61 -321 reply
15/25KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)arch/x86/kvm/x86.c+2 -00 replies
16/25KVM: x86: Define new #PF SGX error code bitarch/x86/include/asm/kvm_host.h+2 -00 replies
17/25KVM: x86: Add support for reverse CPUID lookup of scattered featuresarch/x86/kvm/cpuid.c arch/x86/kvm/cpuid.h+61 -100 replies
18/25KVM: x86: Add reverse-CPUID lookup support for scattered SGX featuresarch/x86/kvm/cpuid.h+12 -10 replies
19/25KVM: VMX: Add basic handling of VM-Exit from SGX enclavearch/x86/include/asm/vmx.h arch/x86/include/uapi/asm/vmx.h arch/x86/kvm/vmx/nested.c +1 more+40 -20 replies
20/25KVM: VMX: Frame in ENCLS handler for SGX virtualizationarch/x86/kvm/Makefile arch/x86/kvm/vmx/sgx.c arch/x86/kvm/vmx/sgx.h +1 more+73 -30 replies
21/25KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictionsarch/x86/include/asm/kvm_host.h arch/x86/kvm/vmx/sgx.c+250 -00 replies
22/25KVM: VMX: Add emulation of SGX Launch Control LE hash MSRsarch/x86/kvm/vmx/sgx.c arch/x86/kvm/vmx/sgx.h arch/x86/kvm/vmx/vmx.c +1 more+63 -00 replies
23/25KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)arch/x86/kvm/vmx/sgx.c+55 -00 replies
24/25KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LCarch/x86/kvm/cpuid.c arch/x86/kvm/vmx/nested.c arch/x86/kvm/vmx/nested.h +5 more+212 -90 replies
25/25KVM: x86: Add capability to grant VM access to privileged SGX attributeDocumentation/virt/kvm/api.rst arch/x86/kvm/cpuid.c arch/x86/kvm/x86.c +1 more+46 -10 replies

Reviews trailers per patch

14 of 25 patches have a Reviewed-by or Acked-by.

PatchDave HansenJarkko Sakkinen
1/25A-b A-b
2/25A-b
3/25A-b
4/25A-b A-b
5/25A-b
6/25A-b
7/25A-b
8/25A-b
9/25A-b A-b
10/25A-b A-b
11/25A-b A-b
12/25A-b A-b
13/25
14/25A-b R-b
15/25A-b
16/25
17/25
18/25
19/25
20/25
21/25
22/25
23/25
24/25
25/25

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v6 thread

Cover letter [RFC PATCH v6 00/25] KVM SGX virtualization support

--- Disclaimer ---

These patches were originally written by Sean Christopherson while at Intel. Now that Sean has left Intel, I (Kai) have taken over getting them upstream. This series needs more review before it can be merged. It is being posted publicly and under RFC so Sean and others can review it. X86 Maintainers are safe ignoring it for now.

------------------

Hi all,

This series adds KVM SGX virtualization support. The first 14 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem.

Please help to review this series. Any feedback is highly appreciated. Please let me know if I forgot to CC anyone, or anyone wants to be removed from CC. Thanks in advance!

This series is based against latest kvm/queue branch (+4 SGX patches reside on tip/x86/sgx to avoid conflict on x86 part patches), since patch (KVM: VMX:
Convert vcpu_vmx.exit_reason to a union) in previous version (v5) has already been merged via another series. Some other recent KVM patches also requires KVM patches in this series to be rebased.

You can also get the code from upstream branch of kvm-sgx repo on github:

https://github.com/intel/kvm-sgx.git upstream

It also requires Qemu changes to create VM with SGX support. You can find Qemu repo here:

https://github.com/intel/qemu-sgx.git upstream

Please refer to README.md of above qemu-sgx repo for detail on how to create guest with SGX support. At meantime, for your quick reference you can use below command to create SGX guest:

	#qemu-system-x86_64 -smp 4 -m 2G -drive file=<your_vm_image>,if=virtio \
		-cpu host,+sgx_provisionkey \
		-sgx-epc id=epc1,memdev=mem1 \
		-object memory-backend-epc,id=mem1,size=64M,prealloc

Please note that the SGX relevant part is:

		-cpu host,+sgx_provisionkey \
		-sgx-epc id=epc1,memdev=mem1 \
		-object memory-backend-epc,id=mem1,size=64M,prealloc

And you can change other parameters of your qemu command based on your needs.

=========
Changelog:

(Changelog here is for global changes. Please see each patch's changelog for
changes made to specific patch.)

v5->v6:

- Main change for x86 part patches is asm/sgx_arch.h is removed and merged to asm/sgx.h, based on Boris's suggestion, and this caused some minor changes to other patches due to header file name being different.
- For KVM part patches, removed patch (KVM: VMX: Convert vcpu_vmx.exit_reason to a union), since it has already been merged to upstream KVM. Also rebased rest of KVM part patches based on latest KVM upstream code.

v4->v5 (No big update from v4):

- Updated commit message of patch 4, per Jarkko.
- Updated comments and commit msg for patch 3, per Dave and Sean.

v3->v4:

- Removed Jarkko's first patch, which removes WARN() against EPC page's SGX_EPC_PAGE_RECLAIMER_TRACKED flag. Dave and Sean were not convinced, and Sean "tripped more than once in the past during one of the many rebases of the virtual EPC and EPC cgroup branches".
- Switched the patch to introduce SGX1/SGX2 feature bits, and the patch to move SGX_LC to cpuid_deps[] table, per Jarkko. Now SGX_LC patch can be applied to existing upstream code directly, if needed.
- Fixed two problems in one KVM patch (trap ECREATE) after review from Rick and Sean.
- Some other minor changes, i.e. update new file's copyright to 2021.

Hi Dave, Sean, Paolo, Rick, and others,

Btw, in last review, we identified that malicious guest can cause KVM to access EPC, which will trigger "page abort", which drops on write, and returns all ones on read. If I understand correctly, we agreed that a better option is to make memremap() deny EPC. However so far this series doesn't contain this work, since looks more discussion is required, and this work can be done independently from KVM SGX series.

v2->v3:

- Split original "x86/cpufeatures: Add SGX1 and SGX2 sub-features" patch into two patches, by splitting moving SGX_LC bit also into cpuid-deps table logic into a separate patch 2:
[RFC PATCH v3 01/27] x86/cpufeatures: Add SGX1 and SGX2 sub-features [RFC PATCH v3 02/27] x86/cpufeatures: Make SGX_LC feature bit depend on SGX bit
- Changed from /dev/sgx_virt_epc to /dev/sgx_vepc, per Jarkko. And accordingly, changed prefix 'sgx_virt_epc_xx' to 'sgx_vepc_xx' in various functions and structures.
- Changed CONFIG_X86_SGX_VIRTUALIZATION to CONFIG_X86_SGX_KVM, per Dave. Couple of x86 patches and KVM patches are changed too due to the renaming.

v1->v2:

- Refined this cover letter by addressing comments from Dave and Jarkko.
- The original patch which introduced new X86_FEATURE_SGX1/SGX2 were replaced
by 3 new patches from Sean, following Boris and Sean's discussion.
[RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features
[RFC PATCH v2 18/26] KVM: x86: Add support for reverse CPUID lookup of scattered features [RFC PATCH v2 19/26] KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
- The original patch 1
x86/sgx: Split out adding EPC page to free list to separate helper
was replaced with 2 new patches from Jarkko
[RFC PATCH v2 02/26] x86/sgx: Remove a warn from sgx_free_epc_page()
[RFC PATCH v2 03/26] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
addressing Jarkko's comments.
- Moved modifying sgx_init() to always initialize sgx_virt_epc_init() out of
patch
x86/sgx: Introduce virtual EPC for use by KVM guests
to a separate patch:
[RFC PATCH v2 07/26] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled
to address Dave's comment that patch ordering can be improved due to before patch "Allow SGX virtualization without Launch Control support", all SGX, including SGX virtualization, is actually disabled when SGX LC is not
present.

=========
KVM SGX virtualization Overview

- Virtual EPC

SGX enclave memory is special and is reserved specifically for enclave use. In bare-metal SGX enclaves, the kernel allocates enclave pages, copies data into the pages with privileged instructions, then allows the enclave to start. In this scenario, only initialized pages already assigned to an enclave are mapped to userspace.

In virtualized environments, the hypervisor still needs to do the physical enclave page allocation. The guest kernel is responsible for the data copying (among other things). This means that the job of starting an enclave is now split between hypervisor and guest.

This series introduces a new misc device: /dev/sgx_vepc. This device allows the host to map *uninitialized* enclave memory into userspace, which can then be passed into a guest.

While it might be *possible* to start a host-side enclave with /dev/sgx_enclave and pass its memory into a guest, it would be wasteful and convoluted.

Implement the *raw* EPC allocation in the x86 core-SGX subsystem via /dev/sgx_vepc rather than in KVM. Doing so has two major advantages:

- Does not require changes to KVM's uAPI, e.g. EPC gets handled as just another memory backend for guests.

- EPC management is wholly contained in the SGX subsystem, e.g. SGX does not have to export any symbols, changes to reclaim flows don't need to be routed through KVM, SGX's dirty laundry doesn't have to get aired out for the world to see, and so on and so forth.

The virtual EPC pages allocated to guests are currently not reclaimable. Reclaiming EPC page used by enclave requires a special reclaim mechanism separate from normal page reclaim, and that mechanism is not supported for virutal EPC pages. Due to the complications of handling reclaim conflicts between guest and host, reclaiming virtual EPC pages is significantly more complex than basic support for SGX virtualization.

- Support SGX virtualization without SGX Flexible Launch Control

SGX hardware supports two "launch control" modes to limit which enclaves can run. In the "locked" mode, the hardware prevents enclaves from running unless they are blessed by a third party. In the unlocked mode, the kernel is in full control of which enclaves can run. The bare-metal SGX code refuses to launch enclaves unless it is in the unlocked mode.

This sgx_virt_epc driver does not have such a restriction. This allows guests which are OK with the locked mode to use SGX, even if the host kernel refuses to.

- Support exposing SGX2

Due to the same reason above, SGX2 feature detection is added to core SGX code to allow KVM to expose SGX2 to guest, even currently SGX driver doesn't support SGX2, because SGX2 can work just fine in guest w/o any interaction to host SGX driver.

- Restricit SGX guest access to provisioning key

To grant guest being able to fully use SGX, guest needs to be able to access provisioning key. The provisioning key is sensitive, and accessing to it should be restricted. In bare-metal driver, allowing enclave to access provisioning key is restricted by being able to open /dev/sgx_provision.

Add a new KVM_CAP_SGX_ATTRIBUTE to KVM uAPI to extend above mechanism to KVM guests as well. When userspace hypervisor creates a new VM, the new cap is only added to VM when userspace hypervisior is able to open /dev/sgx_provision, following the same role as in bare-metal driver. KVM then traps ECREATE from guest, and only allows ECREATE with provisioning key bit to run when guest supports KVM_CAP_SGX_ATTRIBUTE.

Jarkko Sakkinen (1):
x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()

Kai Huang (3):
x86/cpufeatures: Make SGX_LC feature bit depend on SGX bit x86/sgx: Initialize virtual EPC driver even when SGX driver is
disabled
x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs

Sean Christopherson (21):
x86/cpufeatures: Add SGX1 and SGX2 sub-features
x86/sgx: Add SGX_CHILD_PRESENT hardware error code x86/sgx: Introduce virtual EPC for use by KVM guests x86/cpu/intel: Allow SGX virtualization without Launch Control support x86/sgx: Expose SGX architectural definitions to the kernel x86/sgx: Move ENCLS leaf definitions to sgx.h
x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT) x86/sgx: Add encls_faulted() helper
x86/sgx: Add helpers to expose ECREATE and EINIT to KVM x86/sgx: Move provisioning device creation out of SGX driver
KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)
KVM: x86: Define new #PF SGX error code bit
KVM: x86: Add support for reverse CPUID lookup of scattered features
KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
KVM: VMX: Add basic handling of VM-Exit from SGX enclave
KVM: VMX: Frame in ENCLS handler for SGX virtualization
KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions
KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs
KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)
KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC
KVM: x86: Add capability to grant VM access to privileged SGX
attribute

 Documentation/virt/kvm/api.rst                |  23 +
 arch/x86/Kconfig                              |  12 +
 arch/x86/include/asm/cpufeatures.h            |   2 +
 arch/x86/include/asm/kvm_host.h               |   5 +
 .../cpu/sgx/arch.h => include/asm/sgx.h}      |  50 +-
 arch/x86/include/asm/vmx.h                    |   1 +
 arch/x86/include/uapi/asm/vmx.h               |   1 +
 arch/x86/kernel/cpu/cpuid-deps.c              |   3 +
 arch/x86/kernel/cpu/feat_ctl.c                |  69 ++-
 arch/x86/kernel/cpu/scattered.c               |   2 +
 arch/x86/kernel/cpu/sgx/Makefile              |   1 +
 arch/x86/kernel/cpu/sgx/driver.c              |  17 -
 arch/x86/kernel/cpu/sgx/encl.c                |  28 +-
 arch/x86/kernel/cpu/sgx/encls.h               |  30 +-
 arch/x86/kernel/cpu/sgx/ioctl.c               |  23 +-
 arch/x86/kernel/cpu/sgx/main.c                |  92 +++-
 arch/x86/kernel/cpu/sgx/sgx.h                 |  13 +-
 arch/x86/kernel/cpu/sgx/virt.c                | 355 +++++++++++++
 arch/x86/kvm/Makefile                         |   2 +
 arch/x86/kvm/cpuid.c                          |  89 +++-
 arch/x86/kvm/cpuid.h                          |  50 +-
 arch/x86/kvm/vmx/nested.c                     |  28 +-
 arch/x86/kvm/vmx/nested.h                     |   5 +
 arch/x86/kvm/vmx/sgx.c                        | 465 ++++++++++++++++++
 arch/x86/kvm/vmx/sgx.h                        |  34 ++
 arch/x86/kvm/vmx/vmcs12.c                     |   1 +
 arch/x86/kvm/vmx/vmcs12.h                     |   4 +-
 arch/x86/kvm/vmx/vmx.c                        | 102 +++-
 arch/x86/kvm/vmx/vmx.h                        |   2 +
 arch/x86/kvm/x86.c                            |  23 +
 include/uapi/linux/kvm.h                      |   1 +
 tools/testing/selftests/sgx/defines.h         |   2 +-
 32 files changed, 1419 insertions(+), 116 deletions(-)
 rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx.h} (89%)
 create mode 100644 arch/x86/kernel/cpu/sgx/virt.c
 create mode 100644 arch/x86/kvm/vmx/sgx.c
 create mode 100644 arch/x86/kvm/vmx/sgx.h
Signature · 1 line
2.29.2