RFC: SGX virtualization design and draft patches
cover RFC: SGX virtualization design and draft patches
Hi all,
This series is RFC Xen SGX virtualization support design and RFC draft patches.
Intel SGX (Software Guard Extensions) is a new set of instructions and memory access mechanisms targetting for application developers seeking to protect select code and data from disclosure or modification.
The SGX specification can be found in latest Intel SDM as Volume 3D:
https://software.intel.com/sites/default/files/managed/7c/f1/332831-sdm-vol-3d.pdf
SGX is relatively more complicated on specification (entire Volume D) so it is unrealistic to list all hardware details here. First part of the design is the brief SGX introduction, which I think is mandatory for introducing SGX virtualization support. Part 2 is design itself. And I put some reference at last.
In first part I only introduced the info related virtualization support, although this is definitely not the most important part of SGX. Other parts of SGX (most related to cryptography), ie, enclave measurement, SGX key architecture, Sealing & Attestion (which is critical feature actually) are ommited. Please refer to SGX specification for detailed info.
In the design there are some particualr points that I don't know which implementation is better. For those I added a question mark (?) at the right of the menu. Your comments on those parts (and other comments as well, of course) are highly appreciated.
Because SGX has lots of details, so I think the design itself can only be high level, and I also included the RFC patches which contains lots of details. Your comments on the patches are also highly appreciated.
The code can also be found at below github repo for your access:
# git clone https://github.com/01org/xen-sgx -b rfc-v1
And there is another branch named 4.6-sgx is another implementation based on Xen 4.6, it is old but it has some different implementation with this rfc-v1 patches in terms of design choice (ex, it adds a dedicated hypercall).
Please help to review and give comments. Thanks in advance.
==============================================================================
1. SGX Introduction
1.1 Overview
1.1.1 Enclave
1.1.2 EPC (Enclave Paage Cache)
1.1.3 ENCLS and ENCLU
1.2 Discovering SGX Capability
1.2.1 Enumerate SGX via CPUID
1.2.2 Intel SGX Opt-in Configuration
1.3 Enclave Life Cycle
1.3.1 Constructing & Destroying Enclave 1.3.2 Enclave Entry and Exit
1.3.2.1 Synchonous Entry and Exit 1.3.2.2 Asynchounous Enclave Exit
1.3.3 EPC Eviction and Reload
1.4 SGX Launch Control
1.5 SGX Interaction with IA32 and IA64 Architecture
2. SGX Virtualization Design
2.1 High Level Toolstack Changes
2.1.1 New 'epc' parameter
2.1.2 New XL commands (?)
2.1.3 Notify domain's virtual EPC base and size to Xen 2.1.4 Launch Control Support (?)
2.2 High Level Hypervisor Changes
2.2.1 EPC Management (?)
2.2.2 EPC Virtualization (?)
2.2.3 Populate EPC for Guest
2.2.4 New Dedicated Hypercall (?)
2.2.5 Launch Control Support
2.2.6 CPUID Emulation
2.2.7 MSR Emulation
2.2.8 EPT Violation & ENCLS Trapping Handling 2.2.9 Guest Suspend & Resume
2.2.10 Destroying Domain
2.3 Additional Point: Live Migration, Snapshot Support (?)
3. Reference
1. SGX Introduction
1.1 Overview
1.1.1 Enclave
Intel Software Guard Extensions (SGX) is a set of instructions and mechanisms for memory accesses in order to provide security accesses for sensitive applications and data. SGX allows an application to use it's pariticular address space as an *enclave*, which is a protected area provides confidentiality and integrity even in the presence of privileged malware. Accesses to the enclave memory area from any software not resident in the enclave are prevented, including those from privileged software. Below diagram illustrates the presence of Enclave in application.
|-----------------------|
| |
| |---------------| |
| | OS kernel | | |-----------------------|
| |---------------| | | |
| | | | | |---------------| |
| |---------------| | | | Entry table | |
| | Enclave |---|-----> | |---------------| |
| |---------------| | | | Enclave stack | |
| | App code | | | |---------------| |
| |---------------| | | | Enclave heap | |
| | Enclave | | | |---------------| |
| |---------------| | | | Enclave code | |
| | App code | | | |---------------| |
| |---------------| | | |
| | | |-----------------------|
|-----------------------|SGX supports SGX1 and SGX2 extensions. SGX1 provides basic enclave support, and SGX2 allows additional flexibility in runtime management of enclave resources and thread execution within an enclave.
1.1.2 EPC (Enclave Page Cache)
Just like normal application memory management, enclave memory management can be devided into two parts: address space allocation and memory commitment. Address space allocation is allocating particular range of linear address space for enclave. Memory commitment is assigning actual resource for the enclave.
Enclave Page Cache (EPC) is the physical resource used to commit to enclave. EPC is divided to 4K pages. An EPC page is 4K in size and always aligned to 4K boundary. Hardware performs additional access control checks to restrict access to the EPC page. The Enclave Page Cache Map (EPCM) is a secure structure which holds one entry for each EPC page, and is used by hardware to track the status of each EPC page (invisibe to software). Typically EPC and EPCM are reserved by BIOS as Processor Reserved Memory but the actual amount, size, and layout of EPC are model-specific, and dependent on BIOS settings. EPC is enumerated via new SGX CPUID, and is reported as reserved memory.
EPC pages can either be invalid or valid. There are 4 valid EPC types in SGX1:
regular EPC page, SGX Enclave Control Structure (SECS) page, Thread Control Structure (TCS) page, and Version Array (VA) page. SGX2 adds Trimmed EPC page. Each enclave is associated with one SECS page. Each thread in enclave is associated with one TCS page. VA page is used in EPC page eviction and reload. Trimmed EPC page is introduced in SGX2 when particular 4K page in enclave is going to be freed (trimmed) at runtime after enclave is initialized.
1.1.3 ENCLS and ENCLU
Two new instructions ENCLS and ENCLU are introduced to manage enclave and EPC. ENCLS can only run in ring 0, while ENCLU can only run in ring 3. Both ENCLS and ENCLU have multiple leaf functions, with EAX indicating the specific leaf function.
SGX1 supports below ENCLS and ENCLU leaves:
ENCLS:
- ECREATE, EADD, EEXTEND, EINIT, EREMOVE (Enclave build and destroy)
- EPA, EBLOCK, ETRACK, EWB, ELDU/ELDB (EPC eviction & reload)
ENCLU:
- EENTER, EEXIT, ERESUME (Enclave entry, exit, re-enter)
- EGETKEY, EREPORT (SGX key derivation, attestation)
Additionally, SGX2 supports below ENCLS and ENCLU leaves for runtime add/remove EPC page to enclave after enclave is initialized, along with permission change.
ENCLS:
- EAUG, EMODT, EMODPR
ENCLU:
- EACCEPT, EACCEPTCOPY, EMODPE
VMM is able to interfere with ENCLS running in guest (see 1.2.x SGX interaction with VMX) but is unable to interfere with ENCLU.
1.2 Discovering SGX Capability
1.2.1 Enumerate SGX via CPUID
If CPUID.0x7.0:EBX.SGX (bit 2) is 1, then processor supports SGX and SGX capability and resource can be enumerated via new SGX CPUID (0x12). CPUID.0x12.0x0 reports SGX capability, such as the presence of SGX1, SGX2, enclave's maximum size for both 32-bit and 64-bit application. CPUID.0x12.0x1 reports the availability of bits that can be set for SECS.ATTRIBUTES. CPUID.0x12.0x2 reports the EPC resource's base and size. Platform may support multiple EPC sections, and CPUID.0x12.0x3 and further sub-leaves can be used to detect the existence of multiple EPC sections (until CPUID reports invalid EPC).
Refer to 37.7.2 Intel SGX Resource Enumeration Leaves for full description of SGX CPUID 0x12.
1.2.2 Intel SGX Opt-in Configuration
On processors that support Intel SGX, IA32_FEATURE_CONTROL also provides the SGX_ENABLE bit (bit 18) to turn on/off SGX. Before system software can enable and use SGX, BIOS is required to set IA32_FEATURE_CONTROL.SGX_ENABLE = 1 to opt-in SGX.
Setting SGX_ENABLE follows the rules of IA32_FEATURE_CONTROL.LOCK (bit 0). Software is considered to have opted into Intel SGX if and only if IA32_FEATURE_CONTROL.SGX_ENABLE and IA32_FEATURE_CONTROL.LOCK are set to 1.
The setting of IA32_FEATURE_CONTROL.SGX_ENABLE (bit 18) is not reflected by SGX CPUID. Enclave instructions will behavior differently according to value of CPUID.0x7.0x0:EBX.SGX and whether BIOS has opted-in SGX.
Refer to 37.7.1 Intel SGX Opt-in Configuration for more information.
1.3 Enclave Life Cycle
1.3.1 Constructing & Destroying Enclave
Enclave is created via ENCLS[ECREATE] leaf by previleged software. Basically ECREATE converts an invalid EPC page into SECS page, according to a source SECS structure resides in normal memory. The source SECS contains enclave's info such as base (linear) address, size, enclave attributes, enclave's measurement, etc.
After ECREATE, for each 4K linear address space page, priviledged software uses EADD and EEXTEND to add one EPC page to it. Enclave code/data (resides in normal memory) is loaded to enclave during EADD for enclave's each 4K page. After all EPC pages are added to enclave, priviledged software calls EINIT to initialize the enclave, and then enclave is ready to run.
During enclave is constructed, enclave measurement, which is a SHA256 hash value, is also built according to enclave's size, code/data itself and its location in enclave, etc. The measurement can be used to uniquely identify the enclave. SIGSTRUCT in EINIT leaf also contains the measurement specified by untrusted software, via MRENCLAVE. EINIT will check the two measurements and will only succeed when the two matches.
Enclave is destroyed by running EREMOVE for all Enclave's EPC page, and then for enclave's SECS. EREMOVE will report SGX_CHILD_PRESENT error if it is called for SECS when there's still regular EPC pages that haven't been removed from enclave.
Please refer to SDM chapter 39.1 Constructing an Enclave for more infomation.
1.3.2 Enclave Entry and Exit
1.3.2.1 Synchonous Entry and Exit
After enclave is constructed, non-priviledged software use ENCLU[EENTER] to enter enclave to run. While process runs in enclave, non-priviledged software can use ENCLU[EEXIT] to exit from enclave and return to normal mode.
1.3.2.2 Asynchounous Enclave Exit
Asynchronous and synchronous events, such as exceptions, interrupts, traps, SMIs, and VM exits may occur while executing inside an enclave. These events are referred to as Enclave Exiting Events (EEE). Upon an EEE, the processor state is securely saved inside the enclave and then replaced by a synthetic state to prevent leakage of secrets. The process of securely saving state and establishing the synthetic state is called an Asynchronous Enclave Exit (AEX).
After AEX, non-priviledged software uses ENCLU[ERESUME] to re-enter enclave. The SGX userspace software maintains a small piece of code (resides in normal memory) which basically calls ERESUME to re-enter enclave. The address of this piece of code is called Asynchronous Exit Pointer (AEP). AEP is specified as parameter in EENTER and will be kept internally in enclave. Upon AEX, AEP will be pushed to stack and upon returning from EEE handling, such as IRET, AEP will be loaded to RIP and ERESUME will be called subsequently to re-enter enclave.
During AEX the processor will do context saving and restore automatically therefore no change to interrupt handling of OS kernel and VMM is required. It is SGX userspace software's responsibility to setup AEP correctly.
Please refer to SDM chapter 39.2 Enclave Entry and Exit for more infomation.
1.3.3 EPC Eviction and Reload
SGX also allows priviledged software to evict any EPC pages that are used by enclave. The idea is the same as normal memory swapping. Below is the detail info of how to evict EPC pages.
Below is the sequence to evict regular EPC page:
1) Select one or multiple regular EPC pages from one enclave
2) Remove EPT/PT mapping for selected EPC pages
3) Send IPIs to remote CPUs to flush TLB of selected EPC pages
4) EBLOCK on selected EPC pages
5) ETRACK on enclave's SECS page
6) allocate one available slot (8-byte) in VA page
7) EWB on selected EPC pages
With EWB taking:
- VA slot, to restore eviction version info.
- one normal 4K page in memory, to store encrypted content of EPC page.
- one struct PCMD in memory, to store meta data.
(VA slot is a 8-byte slot in VA page, which is a particualr EPC page.)
And below is the sequence to evict an SECS page or VA page:
1) locate SECS (or VA) page
2) remove EPT/PT mapping for SECS (or VA) page
3) Send IPIs to remote CPUs
6) allocate one available slot (8-byte) in VA page
4) EWB on SECS (or) page
And for evicting SECS page, all regular EPC pages that belongs to that SECS must be evicted out prior, otherwise EWB returns SGX_CHILD_PRESENT error.
And to reload an EPC page:
1) ELDU/ELDB on EPC page
2) setup EPT/PT mapping
With ELDU/ELDB taking:
- location of SECS page
- linear address of enclave's 4K page (that we are going to reload to)
- VA slot (used in EWB)
- 4K page in memory (used in EWB)
- struct PCMD in memory (used in EWB)
Please refer to SDM chapter 39.5 EPC and Management of EPC pages for more information.
*********** Instruction Behavior changes in Enclave
- Illegal instructions inside enclave
Instruction Result Comment
CPUID,GETSEC,RDPMC,SGDT,SIDT,SLDT,STR,VMCALL,
1.4 SGX Launch Control
SGX requires running "Launch Enclave" (LE) before running any other enclaves. This is because LE is the only enclave that does not requires EINITTOKEN in EINIT. Running any other enclave requires a valid EINITTOKEN, which contains MAC of the (first 192 bytes) EINITTOKEN calculated by EINITTOKEN key. EINIT will verify the MAC via internally deriving the EINITTOKEN key, and only the EINITTOKEN that has matched MAC will be accepted by EINIT. The EINITTOKEN key derivation depends on some info from LE. The typical process is LE generates EINITTOKEN for other enclave according to LE itself and the target enclave, and calcualtes the MAC by using ENCLU[EGETKEY] to get the EINITTOKEN key. Only LE is able to get the EINITTOKEN key.
Running LE requies the SHA256 hash of LE signer's RSA public key (SHA256 of sigstruct->modulus) to equal to IA32_SGXLEPUBKEYHASH[0-3] MSRs (the 4 MSRs together makes up 256-bit SHA256 hash value).
If CPUID.0x7.0x0:EBX.SGX is set, then IA32_SGXLEPUBKEYHASHn are readable. If CPUID.0x7.0x0:ECX.SGX_LAUNCH_CONTROL[bit 30] is set, then IA32_FEATURE_CONTROL MSR has SGX_LAUNCH_CONTROL_ENABLE bit (bit 17) available. 1-setting of SGX_LAUNCH_CONTROL_ENABLE bit enables runtime change of IA32_SGXLEPUBKEYHASHn after IA32_FEATURE_CONTROL is locked. Otherwise, IA32_SGXLEPUBKEYHASHn are read-only after IA32_FEATURE_CONTROL is locked. IA32_SGXLEPUBKEYHASHn will be set to SHA256 hash of Intel's default RSA public key.
Above mechanism allows 3rd party to run their own LE.
On physical machine, typically BIOS will provide option to *lock* or *unlock* IA32_SGXLEPUBKEYHASHn before transfering to OS. BIOS may also provide interface for user to change default value of IA32_SGXLEPUBKEYHASHn, but what interfaces will be provided by BIOS is BIOS implementation dependent.
1.5 SGX Interaction with IA32 and IA64 Architecture
SDM Chapter 42 describes SGX interaction with various features in IA32 and IA64 architecture. Below outlines the major ones. Refer to Chapter 42 for full description of SGX interaction with various IA32 and IA64 features.
1.5.1 VMX Changes for Supporting SGX Virtualization
A new 64-bit ENCLS-exiting bitmap control field is added to VMCS (encoding 0202EH) to control VMEXIT on ENCLS leaf functions. And a new "Enable ENCLS exiting" control bit (bit 15) is defined in secondary processor based vm execution control. 1-Setting of "Enable ENCLS exiting" enables ENCLS-exiting bitmap control. ENCLS-exiting bitmap controls which ENCLS leaves will trigger VMEXIT.
Additionally two new bits are added to indicate whether VMEXIT (any) is from enclave. Below two bits will be set if VMEXIT is from enclave:
- Bit 27 in the Exit reason filed of Basic VM-exit information.
- Bit 4 in the Interruptibility State of Guest Non-Register State of VMCS.
Refer to 42.5 Interactions with VMX, 27.2.1 Basic VM-Exit Information, and 27.3.4 Saving Non-Register.
1.5.2 Interaction with XSAVE
SGX defines a sub-field called X-Feature Request Mask (XFRM) in the attributes field of SECS. On enclave entry, SGX HW verifies XFRM in SECS.ATTRIBUTES are already enabled in XCR0.
Upon AEX, SGX saves the processor extended state and miscellaneous state to enclave's state-save area (SSA), and clear the secrets from processor extended state that is used by enclave (from leaking secrets).
Refer to 42.7 Interaction with Processor Extended State and Miscellaneous State
1.5.3 Interaction with S state
When processor goes into S3-S5 state, EPC is destroyed, thus all enclaves are destroyed as well consequently.
Refer to 42.14 Interaction with S States.
2. SGX Virtualization Design
2.1 High Level Toolstack Changes:
2.1.1 New 'epc' parameter
EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin
alao should be able to get all domain's virtual EPC size.
For this purpose, a new 'epc = <size>' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted.
2.1.2 New XL commands (?)
Administrator should be able to get physical EPC size, and all domain's virtual EPC size. For this purpose, we can introduce 2 additional commands:
# xl sgxinfo
Which will print out physical EPC size, and other SGX info (such as SGX1, SGX2, etc) if necessary.
# xl sgxlist <did>
Which will print out particular domain's virtual EPC size, or list all virtual EPC sizes for all supported domains.
Alternatively, we can also extend existing XL commands by adding new option
# xl info -sgx
Which will print out physical EPC size along with other physinfo. And
# xl list <did> -sgx
Which will print out domain's virtual EPC size.
Comments?
In my RFC patches I didn't implement the commands as I don't know which is better. In the github repo I mentioned at the beginning, there's an old branch in which I implemented 'xl sgxinfo' and 'xl sgxlist', but they are implemented via dedicated hypercall for SGX, which I am not sure whether is a good option so I didn't include it in my RFC patches.
2.1.3 Notify domain's virtual EPC base and size to Xen
Xen needs to know guest's EPC base and size in order to populate EPC pages for it. Toolstack notifies EPC base and size to Xen via XEN_DOMCTL_set_cpuid.
2.1.4 Launch Control Support (?)
Xen Launch Control Support is about to support running multiple domains with each running its own LE signed by different owners (if HW allows, explained below). As explained in 1.4 SGX Launch Control, EINIT for LE (Launch Enclave) only succeeds when SHA256(SIGSTRUCT.modulus) matches IA32_SGXLEPUBKEYHASHn, and EINIT for other enclaves will derive EINITTOKEN key according to IA32_SGXLEPUBKEYHASHn. Therefore, to support this, guest's virtual IA32_SGXLEPUBKEYHASHn must be updated to phyiscal MSRs before EINIT (which also means the physical IA32_SGXLEPUBKEYHASHn need to be *unlocked* in BIOS before booting to OS).
For physical machine, it is BIOS's writer's decision that whether BIOS would provide interface for user to specify customerized IA32_SGXLEPUBKEYHASHn (it is default to digest of Intel's signing key after reset). In reality, OS's SGX driver may require BIOS to make MSRs *unlocked* and actively write the hash value to MSRs in order to run EINIT successfully, as in this case, the driver will not depend on BIOS's capability (whether it allows user to customerize IA32_SGXLEPUBKEYHASHn value).
The problem is for Xen, do we need a new parameter, such as 'lehash=<SHA256>' to specify the default value of guset's virtual IA32_SGXLEPUBKEYHASHn? And do we need a new parameter, such as 'lewr' to specify whether guest's virtual MSRs are locked or not before handling to guest's OS?
I tends to not introduce 'lehash', as it seems SGX driver would actively update the MSRs. And new parameter would add additional changes for upper layer software (such as openstack). And 'lewr' is not needed either as Xen can always *unlock* the MSRs to guest.
Please give comments?
Currently in my RFC patches above two parameters are not implemented. Xen hypervisor will always *unlock* the MSRs. Whether there is 'lehash' parameter or not doesn't impact Xen hypervisor's emulation of IA32_SGXLEPUBKEYHASHn. See below Xen hypervisor changes for details.
2.2 High Level Xen Hypervisor Changes:
2.2.1 EPC Management (?)
Xen hypervisor needs to detect SGX, discover EPC, and manage EPC before supporting SGX to guest. EPC is detected via SGX CPUID 0x12.0x2. It's possible that there are multiple EPC sections (enumerated via sub-leaves 0x3 and so on, until invaid EPC is reported), but this is only true on multiple-socket server machines. For server machines there are additional things also needs to be done, such as NUMA EPC, scheduling, etc. We will support server machine in the future but currently we only support one EPC.
EPC is reported as reserved memory (so it is not reported as normal memory). EPC must be managed in 4K pages. CPU hardware uses EPCM to track status of each EPC pages. Xen needs to manage EPC and provide functions to, ie, alloc and free EPC pages for guest.
There are two ways to manage EPC: Manage EPC separately; or Integrate it to existing memory management framework.
It is easy to manage EPC separately, as currently EPC is pretty small (~100MB), and we can even put them in a single list. However it is not flexible, for example, you will have to write new algorithms when EPC becomes larger, ex, GB. And you have to write new code to support NUMA EPC (although this will not come in short time).
Integrating EPC to existing memory management framework seems more reasonable, as in this way we can resume memory management data structures/algorithms, and it will be more flexible to support larger EPC and potentially NUMA EPC. But modifying MM framework has a higher risk to break existing memory management code (potentially more bugs).
In my RFC patches currently we choose to manage EPC separately. A new structure epc_page is added to represent a single 4K EPC page. A whole array of struct epc_page will be allocated during EPC initialization, so that given the other, one of PFN of EPC page and 'struct epc_page' can be got by adding offset.
But maybe integrating EPC to MM framework is more reasonable. Comments?
2.2.2 EPC Virtualization (?)
This part is how to populate EPC for guests. We have 3 choices:
- Static Partitioning
- Oversubscription
- Ballooning
Static Partitioning means all EPC pages will be allocated and mapped to guest when it is created, and there's no runtime change of page table mappings for EPC pages. Oversubscription means Xen hypervisor supports EPC page swapping between domains, meaning Xen is able to evict EPC page from another domain and assign it to the domain that needs the EPC. With oversubscription, EPC can be assigned to domain on demand, when EPT violation happens. Ballooning is similar to memory ballooning. It is basically "Static Partitioning" + "Balloon driver" in guest.
Static Partitioning is the easiest way in terms of implementation, and there will be no hypervisor overhead (except EPT overhead of course), because in "Static partitioning", there is no EPT violation for EPC, and Xen doesn't need to turn on ENCLS VMEXIT for guest as ENCLS runs perfectly in non-root mode.
Ballooning is "Static Partitioning" + "Balloon driver" in guest. Like "Static Paratitioning", ballooning doesn't need to turn on ENCLS VMEXIT, and doesn't have EPT violation for EPC either. To support ballooning, we need ballooning driver in guest to issue hypercall to give up or reclaim EPC pages. In terms of hypercall, we have two choices: 1) Add new hypercall for EPC ballooning; 2) Using existing XENMEM_{increase/decrease}_reservation with new memory flag, ie, XENMEMF_epc. I'll discuss more regarding to adding dedicated hypercall or not later.
Oversubscription looks nice but it requires more complicated implemetation. Firstly, as explained in 1.3.3 EPC Eviction & Reload, we need to follow specific steps to evict EPC pages, and in order to do that, basically Xen needs to trap ENCLS from guest and keep track of EPC page status and enclave info from all guest. This is because:
- To evict regular EPC page, Xen needs to know SECS location
- Xen needs to know EPC page type: evicting regular EPC and evicting SECS, VA page have different steps.
- Xen needs to know EPC page status: whether the page is blocked or not.
Those info can only be got by trapping ENCLS from guest, and parsing its parameters (to identify SECS page, etc). Parsing ENCLS parameters means we need to know which ENCLS leaf is being trapped, and we need to translate guest's virtual address to get physical address in order to locate EPC page. And once ENCLS is trapped, we have to emulate ENCLS in Xen, which means we need to reconstruct ENCLS parameters by remapping all guest's virtual address to Xen's virtual address (gva->gpa->pa->xen_va), as ENCLS always use *effective address* which is able to be traslated by processor when running ENCLS.
--------------------------------------------------------------
| ENCLS |
--------------------------------------------------------------
| /|\
ENCLS VMEXIT| | VMENTRY
| |
\|/ |
1) parse ENCLS parameters
2) reconstruct(remap) guest's ENCLS parameters
3) run ENCLS on behalf of guest (and skip ENCLS)
4) on success, update EPC/enclave info, or inject error
And Xen needs to maintain each EPC page's status (type, blocked or not, in enclave or not, etc). Xen also needs to maintain all Enclave's info from all guests, in order to find the correct SECS for regular EPC page, and enclave's linear address as well.
So in general, "Static Partitioning" has simplest implementation, but obviously not the best way to use EPC efficiently; "Ballooning" has all pros of Static Partitioning but requies guest balloon driver; "Oversubscription" is best in terms of flexibility but requires complicated hypervisor implemetation.
We have implemented "Static Partitioning" in RFC patches, but needs your feedback on whether it is enough. If not, which one should we do at next stage -- Ballooning or Oversubscription. IMO Ballooning may be good enough, given fact that currently memory is also "Static Partitioning" + "Ballooning".
Comments?
2.2.3 Populate EPC for Guest
Toolstack notifies Xen about domain's EPC base and size by XEN_DOMCTL_set_cpuid, so currently Xen populates all EPC pages for guest in XEN_DOMCTL_set_cpuid, particularly, in handling XEN_DOMCTL_set_cpuid for CPUID.0x12.0x2. Once Xen checks the values passed from toolstack is valid, Xen will allocate all EPC pages and setup EPT mappings for guest.
2.2.4 New Dedicated Hypercall (?)
So far for all the changes mentioned above, if without a dedicated new hypercall, we have to implement those changes in:
- xl sgxifo (or xl info -sgx)
Toolstack can do this by running SGX CPUID directly, along with checking host cpu featureset.
- xl sgxlist (or xl list -sgx)
This is not quite straightforward. Looks we have to extend xen_domctl_getdomaininfo. However SGX is Intel specific feature, so I am not sure it's a good idea to extend xen_domctl_getdomaininfo.
- Populate EPC for guest
In XEN_DOMCTL_set_cpuid, Xen populates EPC pages for guest after receiving EPC base and size from toolstack.
- Potential EPC Ballooning
Need to add new XENMEMF_epc and use existing
XENMEM_{increase/decrease}_reservation.
With new hypercall for SGX (ie, XEN_sgx_op), all of above can be consolidated into the hypercall. We can also extend it to more generic hypercall for Intel platform genrally (ie, XEN_intel_op). For example, the new hypercall would look like:
#define XEN_INTEL_SGX_physinfo 0x1
struct xen_sgx_physinfo {
/* OUT */
unsigned long total_epc_pages;
unsigned long free_epc_pages;
};
typedef struct xen_sgx_physinfo xen_sgx_physinfo_t;
DEFINE_XEN_GUEST_HANDLE(xen_sgx_physinfo_t);
#define XEN_INTEL_SGX_setup_epc 0x2
struct xen_sgx_setup_epc {
/* IN */
domid_t domid;
unsigned long epc_base_gfn;
unsigned long total_epc_pages;
};
typedef struct xen_sgx_setup_epc xen_sgx_setup_epc_t;
DEFINE_XEN_GUEST_HANDLE(xen_sgx_setup_epc_t);
#define XEN_INTEL_SGX_dominfo 0x3
struct xen_sgx_dominfo {
/* IN */
domid_t domid;
/* OUT */
unsigned long epc_base_gfn;
unsigned long total_epc_pages;
};
DEFINE_XEN_GUEST_HANDLE(xen_sgx_dominfo);
struct xen_sgx_op {
/* XEN_INTEL_SGX_* */
int cmd;
union {
struct xen_sgx_physinfo physinfo;
struct xen_sgx_setup_epc setup_epc;
struct xen_sgx_dominfo dominfo;
} u;
};
typedef struct xen_sgx_op xen_sgx_op_t;
DEFINE_XEN_GUEST_HANDLE(xen_sgx_op);
/* New arch specific hypercall for Intel platform specific operations,
* __HYPERVISOR_arch_0 is used by Xen x86 machine check... */
#define __HYPERVISOR_intel_op __HYPERVISOR_arch_1
/* Currently only SGX uses this */
#define XEN_INTEL_OP_sgx (0x1 << 1)
struct xen_intel_op {
int cmd; /* XEN_INTEL_OP_*** */
union {
struct xen_sgx_op sgx_op;
} u;
}
typedef struct xen_intel_op xen_intel_op_t;
DEFINE_XEN_GUEST_HANDLE(xen_intel_op_t);In my RFC patches, the new hypercall is not implemented as I am not sure whether it is a good idea.
Comments?
2.2.5 Launch Control Support
To support running multiple domains with each running its own LE signed by different owners, physical machine's BIOS must leave IA32_SGXLEPUBKEYHASHn *unlocked* before handing to Xen. Xen will trap domain's write to IA32_SGXLEPUBKEYHASHn and keep the value in vcpu internally, and update the value to physical MSRs when vcpu is scheduled in. This can guarantee that when EINIT runs in guest, guest's virtual IA32_SGXLEPUBKEYHASHn have been written to physical MSRs.
SGX_LAUNCH_CONTROL_ENABLE bit will always be set in guest's IA32_FEATURE_CONTROL MSR (see 2.1.4 Launch Control Support).
If physical IA32_SGXLEPUBKEYHASHn are *locked* in machine's BIOS, then only MSR read is allowed from guest, and Xen will inject error for guest's MSR writes.
If CPUID.0x7.0x0:ECX.SGX_LAUHCN_CONTROL is not present, then this feature will not be exposed to guest as well, and SGX_LAUNCH_CONTROL_ENABLE bit is set to 0 (as it is invalid).
2.2.6 CPUID Emulation
Most of native SGX CPUID info can be exposed to guest, expect below two parts:
- Sub-leaf 0x2 needs to report domain's virtual EPC base and size, instead of physical EPC info.
- Sub-leaf 0x1 needs to be consistent with guest's XCR0. For the reason of this part please refer to 1.5.2 Interaction with XSAVE.
2.2.7 MSR Emulation
SGX_ENABLE it in IA32_FEATURE_CONTROL is always set if SGX is exposed to guest, SGX_LAUNCH_CONTROL_ENABLE bit is handled as in 2.2.4. Any write from guest to IA32_FEATURE_CONTROL is ignored.
IA32_SGXLEPUBKEYHASHn emulation is described in 2.2.4.
2.2.8 EPT Violation & ENCLS Trapping Handling
Only needed when Xen supports EPC Oversubscription, as explained above.
2.2.9 Guest Suspend & Resume
On hardware, EPC is destroyed when power goes to S3-S5. So Xen will destroy guest's EPC when guest's power goes into S3-S5. Currently Xen is notified by Qemu in terms of S State change via HVM_PARAM_ACPI_S_STATE, where Xen will destroy EPC if S State is S3-S5.
Specifically, Xen will run EREMOVE for guest's each EPC page, as guest may not handle EPC suspend & resume correctly, in which case physically guest's EPC pages may still be valid, so Xen needs to run EREMOVE to make sure all EPC pages are becoming invalid. Otherwise further operation in guest on EPC may fault as it assumes all EPC pages are invalid after guest is resumed.
For SECS page, EREMOVE may fault with SGX_CHILD_PRESENT, in which case Xen will keep this SECS page into a list, and call EREMOVE for them again after all EPC pages have been called with EREMOVE. This time the EREMOVE on SECS will succeed as all children (regular EPC pages) have already been removed.
2.2.10 Destroying Domain
Normally Xen just frees all EPC pages for domain when it is destroyed. But Xen will also do EREMOVE on all guest's EPC pages (described in above 2.2.7) before free them, as guest may shutdown unexpected (ex, user kills guest), and in this case, guest's EPC may still be valid.
2.3 Additional Point: Live Migration, Snapshot Support (?)
Actually from hardware's point of view, SGX is not migratable. There are two reasons:
- SGX key architecture cannot be virtualized.
For example, some keys are bound to CPU. For example, Sealing key, EREPORT key, etc. If VM is migrated to another machine, the same enclave will derive the different keys. Taking Sealing key as an example, Sealing key is typically used by enclave (enclave can get sealing key by EGETKEY) to *seal* its secrets to outside (ex, persistent storage) for further use. If Sealing key changes after VM migration, then the enclave can never get the sealed secrets back by using sealing key, as it has changed, and old sealing key cannot be got back.
- There's no ENCLS to evict EPC page to normal memory, but at the meaning time, still keep content in EPC. Currently once EPC page is evicted, the EPC page becomes invalid. So technically, we are unable to implement live migration (or check pointing, or snapshot) for enclave.
But, with some workaround, and some facts of existing SGX driver, technically we are able to support Live migration (or even check pointing, snapshot). This is because:
- Changing key (which is bound to CPU) is not a problem in reality
Take Sealing key as an example. Losing sealed data is not a problem, because sealing key is only supposed to encrypt secrets that can be provisioned again. The typical work model is, enclave gets secrets provisioned from remote (service provider), and use sealing key to store it for further use. When enclave tries to *unseal* use sealing key, if the sealing key is changed, enclave will find the data is some kind of corrupted (integrity check failure), so it will ask secrets to be provisioned again from remote. Another reason is, in data center, VM's typically share lots of data, and as sealing key is bound to CPU, it means the data encrypted by one enclave on one machine cannot be shared by another enclave on another mahcine. So from SGX app writer's point of view, developer should treat Sealing key as a changeable key, and should handle lose of sealing data anyway. Sealing key should only be used to seal secrets that can be easily provisioned again.
For other keys such as EREPORT key and provisioning key, which are used for local attestation and remote attestation, due to the second reason below, losing them is not a problem either.
- Sudden lose of EPC is not a problem.
On hardware, EPC will be lost if system goes to S3-S5, or reset, or shutdown, and SGX driver need to handle lose of EPC due to power transition. This is done by cooperation between SGX driver and userspace SGX SDK/apps. However during live migration, there may not be power transition in guest, so there may not be EPC lose during live migration. And technically we cannot *really* live migrate enclave (explained above), so looks it's not feasible. But the fact is that both Linux SGX driver and Windows SGX driver have already supported *sudden* lose of EPC (not EPC lose during power transition), which means both driver are able to recover in case EPC is lost at any runtime. With this, technically we are able to support live migration by simply ignoring EPC. After VM is migrated, the destination VM will only suffer *sudden* lose of EPC, which both Windows SGX driver and Linux SGX driver are already able to handle.
But we must point out such *sudden* lose of EPC is not hardware behavior, and other SGX driver for other OSes (such as FreeBSD) may not implement this, so for those guests, destination VM will behavior in unexpected manner. But I am not sure we need to care about other OSes.
For the same reason, we are able to support check pointing for SGX guest (only Linux and Windows);
For snapshot, we can support snapshot SGX guest by either:
- Suspend guest before snapshot (s3-s5). This works for all guests but requires user to manually susppend guest.
- Issue an hypercall to destroy guest's EPC in save_vm. This only works for Linux and Windows but doesn't require user intervention.
What's your comments?
3. Reference
- Intel SGX Homepage
https://software.intel.com/en-us/sgx
- Linux SGX SDK
https://01.org/intel-software-guard-extensions
- Linux SGX driver for upstreaming
https://github.com/01org/linux-sgx
- Intel SGX Specification (SDM Vol 3D)
https://software.intel.com/sites/default/files/managed/7c/f1/332831-sdm-vol-3d.pdf
- Paper: Intel SGX Explained
https://eprint.iacr.org/2016/086.pdf
- ISCA 2015 tutorial slides for Intel® SGX - Intel® Software
https://software.intel.com/sites/default/files/332680-002.pdf
Kai Huang (15):
xen: x86: expose SGX to HVM domain in CPU featureset
xen: vmx: detect ENCLS VMEXIT
xen: x86: add early stage SGX feature detection
xen: mm: add ioremap_cache
xen: p2m: new 'p2m_epc' type for EPC mapping
xen: x86: add SGX basic EPC management
xen: x86: add functions to populate and destroy EPC for domain
xen: x86: add SGX cpuid handling support.
xen: vmx: handle SGX related MSRs
xen: vmx: handle ENCLS VMEXIT
xen: vmx: handle VMEXIT from SGX enclave
xen: x86: reset EPC when guest got suspended.
xen: tools: add new 'epc' parameter support
xen: tools: add SGX to applying CPUID policy
xen: tools: expose EPC in ACPI table
tools/firmware/hvmloader/util.c | 23 + tools/firmware/hvmloader/util.h | 3 + tools/libacpi/build.c | 3 + tools/libacpi/dsdt.asl | 49 ++ tools/libacpi/dsdt_acpi_info.asl | 6 +- tools/libacpi/libacpi.h | 1 + tools/libxc/include/xc_dom.h | 4 + tools/libxc/include/xenctrl.h | 10 + tools/libxc/xc_cpuid_x86.c | 68 ++- tools/libxl/libxl.h | 3 +- tools/libxl/libxl_cpuid.c | 15 +- tools/libxl/libxl_create.c | 9 + tools/libxl/libxl_dom.c | 36 +- tools/libxl/libxl_internal.h | 2 + tools/libxl/libxl_nocpuid.c | 4 +- tools/libxl/libxl_types.idl | 6 + tools/libxl/libxl_x86.c | 12 + tools/libxl/libxl_x86_acpi.c | 3 + tools/ocaml/libs/xc/xenctrl_stubs.c | 11 +- tools/python/xen/lowlevel/xc/xc.c | 11 +- tools/xl/xl_parse.c | 5 + xen/arch/x86/cpuid.c | 87 ++- xen/arch/x86/domctl.c | 47 +- xen/arch/x86/hvm/hvm.c | 3 + xen/arch/x86/hvm/vmx/Makefile | 1 + xen/arch/x86/hvm/vmx/sgx.c | 871 ++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmcs.c | 21 + xen/arch/x86/hvm/vmx/vmx.c | 73 +++ xen/arch/x86/hvm/vmx/vvmx.c | 11 + xen/arch/x86/mm.c | 15 +- xen/arch/x86/mm/p2m-ept.c | 3 + xen/arch/x86/mm/p2m.c | 41 ++ xen/include/asm-x86/cpufeature.h | 4 + xen/include/asm-x86/cpuid.h | 26 +- xen/include/asm-x86/hvm/hvm.h | 3 + xen/include/asm-x86/hvm/vmx/sgx.h | 100 ++++ xen/include/asm-x86/hvm/vmx/vmcs.h | 10 + xen/include/asm-x86/hvm/vmx/vmx.h | 3 + xen/include/asm-x86/msr-index.h | 6 + xen/include/asm-x86/p2m.h | 12 +- xen/include/public/arch-x86/cpufeatureset.h | 3 +- xen/include/xen/vmap.h | 1 + xen/tools/gen-cpuid.py | 3 + 43 files changed, 1607 insertions(+), 21 deletions(-) create mode 100644 xen/arch/x86/hvm/vmx/sgx.c create mode 100644 xen/include/asm-x86/hvm/vmx/sgx.h
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 1/15 xen: x86: expose SGX to HVM domain in CPU featureset
Expose SGX in CPU featureset for HVM domain. SGX will not be supported for PV domain, as ENCLS (which SGX driver in guest essentially runs) must run in ring 0, while PV kernel runs in ring 3. Theoretically we can support SGX in PV domain via either emulating #GP caused by ENCLS running in ring 3, or by PV ENCLS but it is really not necessary at this stage. And currently SGX is only exposed to HAP HVM domain (we can add for shadow in the future).
SGX Launch Control is also exposed in CPU featureset for HVM domain. SGX Launch Control depends on SGX.
--- xen/include/public/arch-x86/cpufeatureset.h | 3 ++- xen/tools/gen-cpuid.py | 3 +++ 2 files changed, 5 insertions(+), 1 deletion(-)
xen/include/public/arch-x86/cpufeatureset.h +2 -1
@@ -193,7 +193,7 @@ XEN_CPUFEATURE(XSAVES, 4*32+ 3) /*S XSAVES/XRSTORS instructions *//* Intel-defined CPU features, CPUID level 0x00000007:0.ebx, word 5 */ XEN_CPUFEATURE(FSGSBASE, 5*32+ 0) /*A {RD,WR}{FS,GS}BASE instructions */ XEN_CPUFEATURE(TSC_ADJUST, 5*32+ 1) /*S TSC_ADJUST MSR available */ -XEN_CPUFEATURE(SGX, 5*32+ 2) /* Software Guard extensions */ +XEN_CPUFEATURE(SGX, 5*32+ 2) /*H Intel Software Guard extensions */ XEN_CPUFEATURE(BMI1, 5*32+ 3) /*A 1st bit manipulation extensions */ XEN_CPUFEATURE(HLE, 5*32+ 4) /*A Hardware Lock Elision */XEN_CPUFEATURE (AVX2 ,5 *32 +5 )/*A AVX2 instructions */
@@ -229,6 +229,7 @@ XEN_CPUFEATURE(PKU, 6*32+ 3) /*H Protection Keys for Userspace */ XEN_CPUFEATURE(OSPKE, 6*32+ 4) /*! OS Protection Keys Enable */ XEN_CPUFEATURE(AVX512_VPOPCNTDQ, 6*32+14) /*A POPCNT for vectors of DW/QW */ XEN_CPUFEATURE(RDPID, 6*32+22) /*A RDPID instruction */ +XEN_CPUFEATURE (SGX_LAUNCH_CONTROL ,6 *32 +30 )/*H Intel SGX Launch Control */ /* AMD-defined CPU features, CPUID level 0x80000007.edx, word 7 */XEN_CPUFEATURE (ITSC ,7 *32 +8 )/* Invariant TSC */
xen/tools/gen-cpuid.py +3 -0
@@ -256,6 +256,9 @@ def crunch_numbers(state): AVX512F: [AVX512DQ, AVX512IFMA, AVX512PF, AVX512ER, AVX512CD, AVX512BW, AVX512VL, AVX512VBMI, AVX512_4VNNIW, AVX512_4FMAPS, AVX512_VPOPCNTDQ], + + # SGX Launch Control depends on SGX + SGX: [SGX_LAUNCH_CONTROL], } deep_features = tuple(sorted(deps.keys()))
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 2/15 xen: vmx: detect ENCLS VMEXIT
In reply to Kai HuangIf ENCLS VMEXIT is not present then we cannot support SGX virtualization. This patch detects presence of ENCLS VMEXIT. A Xen boot boolean parameter 'sgx' is also added to manually enable/disable SGX.
--- xen/arch/x86/hvm/vmx/vmcs.c | 17 +++++++++++++++++ xen/include/asm-x86/hvm/vmx/vmcs.h | 3 +++ 2 files changed, 20 insertions(+)
xen/arch/x86/hvm/vmx/vmcs.c +17 -0
@@ -98,6 +98,9 @@ static void __init parse_ept_param(char *s) }custom_param ("ept" , parse_ept_param); +static bool_t __read_mostly opt_sgx_enabled =1 ; +boolean_param ("sgx" , opt_sgx_enabled); +/* Dynamic (run-time adjusted) execution control flags. */ u32 vmx_pin_based_exec_control__read_mostly ;u32 vmx_cpu_based_exec_control__read_mostly ;
@@ -138,6 +141,7 @@ static void __init vmx_display_features(void)P (cpu_has_vmx_virt_exceptions,"Virtualisation Exceptions" );P (cpu_has_vmx_pml,"Page Modification Logging" );P (cpu_has_vmx_tsc_scaling,"TSC Scaling" ); +P (cpu_has_vmx_encls,"SGX ENCLS Exiting" );#undef Pif ( !printed )
@@ -243,6 +247,8 @@ static int vmx_init_vmcs_config(void) opt |=SECONDARY_EXEC_UNRESTRICTED_GUEST ;if ( opt_pml_enabled ) opt |=SECONDARY_EXEC_ENABLE_PML ; +if ( opt_sgx_enabled ) + opt |=SECONDARY_EXEC_ENABLE_ENCLS ;/* * "APIC Register Virtualization" and "Virtual Interrupt Delivery"
@@ -336,6 +342,14 @@ static int vmx_init_vmcs_config(void) _vmx_secondary_exec_control &= ~SECONDARY_EXEC_PAUSE_LOOP_EXITING ; } +/* +* Turn off SGX if ENCLS VMEXIT is not present. Actually on real machine, +* if SGX CPUID is present (CPUID.0x7.0x0:EBX.SGX = 1), then ENCLS VMEXIT +* will always be present. We do the check anyway here. +*/ +if ( !(_vmx_secondary_exec_control &SECONDARY_EXEC_ENABLE_ENCLS ) ) + opt_sgx_enabled =0 ; + min =VM_EXIT_ACK_INTR_ON_EXIT ; opt =VM_EXIT_SAVE_GUEST_PAT |VM_EXIT_LOAD_HOST_PAT |VM_EXIT_CLEAR_BNDCFGS ;
@@ -1146,6 +1160,9 @@ static int construct_vmcs(struct vcpu *v)/* Disable PML anyway here as it will only be enabled in log dirty mode */ v->arch.hvm_vmx.secondary_exec_control &= ~SECONDARY_EXEC_ENABLE_PML ; +/* Disable ENCLS VMEXIT. It will only be turned on when needed. */ + v->arch.hvm_vmx.secondary_exec_control &= ~SECONDARY_EXEC_ENABLE_ENCLS ; +/* Host data selectors. */ __vmwrite (HOST_SS_SELECTOR , __HYPERVISOR_DS);__vmwrite (HOST_DS_SELECTOR , __HYPERVISOR_DS);
xen/include/asm-x86/hvm/vmx/vmcs.h +3 -0
@@ -232,6 +232,7 @@ extern u32 vmx_vmentry_control;#define SECONDARY_EXEC_ENABLE_INVPCID 0x00001000#define SECONDARY_EXEC_ENABLE_VM_FUNCTIONS 0x00002000#define SECONDARY_EXEC_ENABLE_VMCS_SHADOWING 0x00004000 +#define SECONDARY_EXEC_ENABLE_ENCLS 0x00008000#define SECONDARY_EXEC_ENABLE_PML 0x00020000#define SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS 0x00040000#define SECONDARY_EXEC_XSAVES 0x00100000
@@ -312,6 +313,8 @@ extern u64 vmx_ept_vpid_cap; (vmx_secondary_exec_control & SECONDARY_EXEC_XSAVES)#define cpu_has_vmx_tsc_scaling \ (vmx_secondary_exec_control & SECONDARY_EXEC_TSC_SCALING) +#define cpu_has_vmx_encls \ + (vmx_secondary_exec_control & SECONDARY_EXEC_ENABLE_ENCLS)#define VMCS_RID_TYPE_MASK 0x80000000
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 3/15 xen: x86: add early stage SGX feature detection
This patch adds early stage SGX feature detection via SGX CPUID 0x12. Function detect_sgx is added to detect SGX info on each CPU (called from vmx_cpu_up). SDM says SGX info returned by CPUID is per-thread, and we cannot assume all threads will return the same SGX info, so we have to detect SGX for each CPU. For simplicity, currently SGX is only supported when all CPUs reports the same SGX info.
SDM also says it's possible to have multiple EPC sections but this is only for multiple-socket server, which we don't support now (there are other things need to be done, ex, NUMA EPC, scheduling, etc, as well), so currently only one EPC is supported.
Dedicated files sgx.c and sgx.h are added (under vmx directory as SGX is Intel specific) for bulk of above SGX detection code detection code, and for further SGX code as well.
--- xen/arch/x86/hvm/vmx/Makefile | 1 + xen/arch/x86/hvm/vmx/sgx.c | 208 ++++++++++++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmcs.c | 4 + xen/include/asm-x86/cpufeature.h | 1 + xen/include/asm-x86/hvm/vmx/sgx.h | 45 +++++++++ 5 files changed, 259 insertions(+) create mode 100644 xen/arch/x86/hvm/vmx/sgx.c create mode 100644 xen/include/asm-x86/hvm/vmx/sgx.h
xen/arch/x86/hvm/vmx/Makefile +1 -0
@@ -4,3 +4,4 @@ obj-y += realmode.o obj-y += vmcs.o obj-y += vmx.o obj-y += vvmx.o +obj-y += sgx.o
xen/arch/x86/hvm/vmx/sgx.c new file +208 -0
208 lines not on this page · raw message
xen/arch/x86/hvm/vmx/vmcs.c +4 -0
@@ -40,6 +40,7 @@#include <asm/shadow.h> #include <asm/tboot.h> #include <asm/apic.h> +#include <asm/hvm/vmx/sgx.h> static bool_t __read_mostly opt_vpid_enabled =1 ;boolean_param ("vpid" , opt_vpid_enabled);
@@ -696,6 +697,9 @@ int vmx_cpu_up(void)vmx_pi_per_cpu_init (cpu); +if ( cpu_has_vmx_encls ) +detect_sgx (cpu); +return 0 ; }
xen/include/asm-x86/cpufeature.h +1 -0
@@ -85,6 +85,7 @@/* CPUID level 0x00000007:0.ebx */ #define cpu_has_fsgsbase boot_cpu_has(X86_FEATURE_FSGSBASE) +#define cpu_has_sgx boot_cpu_has(X86_FEATURE_SGX)#define cpu_has_bmi1 boot_cpu_has(X86_FEATURE_BMI1)#define cpu_has_hle boot_cpu_has(X86_FEATURE_HLE)#define cpu_has_avx2 boot_cpu_has(X86_FEATURE_AVX2)
xen/include/asm-x86/hvm/vmx/sgx.h new file +45 -0
45 lines not on this page · raw message
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 6/15 xen: x86: add SGX basic EPC management
In reply to Kai HuangEPC is limited resource reserved by BIOS. Typically EPC size is from dozens of MB to more than a hundred MB. EPC is reported as reserved memory in e820 but not normal memory. EPC must be managed in 4K pages.
From implementation's view, we can choose either to manage EPC separately, or to extend existing memory management code to support EPC. The latter has advantage of being able to use existing memory management algorithm but is more complicated to implement (thus more risky), while the former is more simple but has to write own EPC management algorithm. Currently we choose the former. Given the fact that EPC size is small, currently we simply put all EPC pages into single list, so allocation and free are very straightforward.
Like there is one 'struct page_info' for each memory page, a 'struct epc_page' is added to represent status of each EPC page, and all 'struct epc_page' will be in an array which is allocated during SGX initialization. Entire EPC is also mapped to Xen's virtual address so that each EPC page's virtual address can be calculated by base virtual address + offset.
--- xen/arch/x86/hvm/vmx/sgx.c | 154 ++++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/hvm/vmx/sgx.h | 19 +++++ 2 files changed, 173 insertions(+)
xen/arch/x86/hvm/vmx/sgx.c +154 -0
@@ -7,12 +7,89 @@#include <asm/cpufeature.h> #include <asm/msr-index.h> #include <asm/msr.h> +#include <xen/errno.h> +#include <xen/mm.h> #include <asm/hvm/vmx/sgx.h> #include <asm/hvm/vmx/vmcs.h> static struct sgx_cpuinfo __read_mostly sgx_cpudata[NR_CPUS ];static struct sgx_cpuinfo __read_mostly boot_sgx_cpudata; +/* +* epc_frametable keeps an array of struct epc_page for every EPC pages, so that +* epc_page_to_mfn, epc_mfn_to_page works straightforwardly. The array will be +* allocated dynamically according to machine's EPC size. +*/ +static struct epc_page *epc_frametable =NULL ; +/* +* EPC is mapped to Xen's virtual address at once, so that each EPC page's +* virtual address is epc_base_vaddr + offset. +*/ +static void *epc_base_vaddr =NULL ; + +/* Global free EPC pages list. */ +static struct list_head free_epc_list; +static spinlock_t epc_lock; + +#define total_epc_npages (boot_sgx_cpudata.epc_size >> PAGE_SHIFT) +#define epc_base_mfn (boot_sgx_cpudata.epc_base >> PAGE_SHIFT) + +/* Current number of free EPC pages in free_epc_list */ +static unsigned long free_epc_npages =0 ; + +unsigned long epc_page_to_mfn (struct epc_page *epg) +{ +BUG_ON (!epc_frametable); +BUG_ON (!epc_base_mfn); + +return epc_base_mfn + (epg - epc_frametable); +} + +struct epc_page *epc_mfn_to_page (unsigned long mfn) +{ +BUG_ON (!epc_frametable); +BUG_ON (!epc_base_mfn); + +return epc_frametable + (mfn - epc_base_mfn); +} + +struct epc_page *alloc_epc_page (void ) +{ +struct epc_page *epg; + +spin_lock (&epc_lock); + epg = list_first_entry_or_null(&free_epc_list, struct epc_page, list); +if ( epg ) { +list_del (&epg->list); + free_epc_npages--; + } +spin_unlock (&epc_lock); + +return epg; +} + +void free_epc_page (struct epc_page *epg) +{ +spin_lock (&epc_lock); +list_add_tail (&epg->list, &free_epc_list); + free_epc_npages++; +spin_unlock (&epc_lock); +} + +void *map_epc_page_to_xen (struct epc_page *epg) +{ +BUG_ON (!epc_base_vaddr); +BUG_ON (!epc_frametable); + +return (void *)(((unsigned long )(epc_base_vaddr)) + + ((epg - epc_frametable) <<PAGE_SHIFT )); +} + +void unmap_epc_page (void *addr) +{ +/* Nothing */ +} +static bool_t sgx_enabled_in_bios (void ) { uint64_t val, sgx_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE |
@@ -177,6 +254,80 @@ static bool_t __init check_sgx_consistency(void)return true; } +static int inline npages_to_order (unsigned long npages) +{ +int order =0 ; + +while ( (1 << order) < npages ) + order++; + +return order; +} + +static int __init init_epc_frametable (unsigned long npages) +{ +unsigned long i, order; + + order = npages *sizeof (struct epc_page ); + order >>=12 ; + order =npages_to_order (order); + + epc_frametable =alloc_xenheap_pages (order,0 ); +if ( !epc_frametable ) +return -ENOMEM ; + +for ( i =0 ; i < npages; i++ ) + { +struct epc_page *epg = epc_frametable + i; + +list_add_tail (&epg->list, &free_epc_list); + } + +return 0 ; +} + +static void destroy_epc_frametable (unsigned long npages) +{ +unsigned long order; + +if ( !epc_frametable ) +return ; + + order = npages *sizeof (struct epc_page ); + order >>=12 ; + order =npages_to_order (order); + +free_xenheap_pages (epc_frametable, order); +} + +static int __init sgx_init_epc (void ) +{ +int r; + +INIT_LIST_HEAD (&free_epc_list); +spin_lock_init (&epc_lock); + + r =init_epc_frametable (total_epc_npages); +if ( r ) + { +printk ("Failed to allocate EPC frametable. Disable SGX.\n" ); +return r; + } + + epc_base_vaddr =ioremap_cache (epc_base_mfn <<PAGE_SHIFT , + total_epc_npages <<PAGE_SHIFT ); +if ( !epc_base_vaddr ) + { +printk ("Failed to ioremap_cache EPC. Disable SGX.\n" ); +destroy_epc_frametable (total_epc_npages); +return -EFAULT ; + } + + free_epc_npages = total_epc_npages; + +return 0 ; +} +static int __init sgx_init (void ) {/* Assume CPU 0 is always online */
@@ -188,6 +339,9 @@ static int __init sgx_init(void)if ( !check_sgx_consistency () ) goto not_supported; +if (sgx_init_epc () ) + goto not_supported; +print_sgx_cpuinfo (&boot_sgx_cpudata);return 0 ;
xen/include/asm-x86/hvm/vmx/sgx.h +19 -0
@@ -12,6 +12,7 @@#include <xen/types.h> #include <xen/init.h> #include <asm/processor.h> +#include <xen/list.h> #define SGX_CPUID 0x12
@@ -42,4 +43,22 @@ struct sgx_cpuinfo {/* Detect SGX info for particular CPU via SGX CPUID */ void detect_sgx (int cpu); +/* +* EPC page infomation structure. Each EPC has one struct epc_page to keep EPC +* page info, just like struct page_info for normal memory. +* +* So far in reality machine's EPC size won't execeed 100MB, so currently just +* put all free EPC pages in global free list. +*/ +struct epc_page { +struct list_head list;/* all free EPC pages are in global free list. */ +}; + +struct epc_page *alloc_epc_page (void ); +void free_epc_page (struct epc_page *epg); +unsigned long epc_page_to_mfn (struct epc_page *epg); +struct epc_page *epc_mfn_to_page (unsigned long mfn); +void *map_epc_page_to_xen (struct epc_page *epg); +void unmap_epc_page (void *addr); +#endif /* __ASM_X86_HVM_VMX_SGX_H__ */
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 7/15 xen: x86: add functions to populate and destroy EPC for domain
In reply to Kai HuangAdd per-domain structure to store SGX per-domain info. Currently only domain's EPC base and size are stored. Also add new functions for further use:
- hvm_populate_epc # populate EPC when EPC base & size are notified.
- hvm_reset_epc # Reset domain's EPC to be invalid. Used when domain
goes to S3-S5, or being destroyed.
- hvm_destroy_epc # destroy and free domain's EPC.
--- xen/arch/x86/hvm/vmx/sgx.c | 315 +++++++++++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmx.c | 3 + xen/include/asm-x86/hvm/vmx/sgx.h | 14 ++ xen/include/asm-x86/hvm/vmx/vmcs.h | 2 + 4 files changed, 334 insertions(+)
xen/arch/x86/hvm/vmx/sgx.c +315 -0
@@ -9,6 +9,8 @@#include <asm/msr.h> #include <xen/errno.h> #include <xen/mm.h> +#include <xen/sched.h> +#include <asm/p2m.h> #include <asm/hvm/vmx/sgx.h> #include <asm/hvm/vmx/vmcs.h>
@@ -90,6 +92,319 @@ void unmap_epc_page(void *addr)/* Nothing */ } +/* ENCLS opcode */ +#define ENCLS .byte 0x0f, 0x01, 0xcf + +/* +* ENCLS leaf functions +* +* However currently we only needs EREMOVE.. +*/ +enum { +ECREATE =0x0 , +EADD =0x1 , +EINIT =0x2 , +EREMOVE =0x3 , +EDGBRD =0x4 , +EDGBWR =0x5 , +EEXTEND =0x6 , +ELDU =0x8 , +EBLOCK =0x9 , +EPA =0xA , +EWB =0xB , +ETRACK =0xC , +EAUG =0xD , +EMODPR =0xE , +EMODT =0xF , +}; + +/* +* ENCLS error code +* +* Currently we only need SGX_CHILD_PRESENT +*/ +#define SGX_CHILD_PRESENT 13 + +static inline int __encls (unsigned long rax,unsigned long rbx, +unsigned long rcx,unsigned long rdx) +{ +int ret; + + asmvolatile ("ENCLS;\n\t" + :"=a" (ret) + :"a" (rax),"b" (rbx),"c" (rcx),"d" (rdx) + :"memory" ,"cc" ); + +return ret; +} + +static inline int __eremove (void *epc) +{ +unsigned long rbx =0 , rdx =0 ; + +return __encls (EREMOVE , rbx, (unsigned long )epc, rdx); +} + +static int sgx_eremove (struct epc_page *epg) +{ +void *addr =map_epc_page_to_xen (epg); +int ret; + +BUG_ON (!addr); + + ret =__eremove (addr); + +unmap_epc_page (addr); + +return ret; +} + +/* +* Reset domain's EPC with EREMOVE. free_epc indicates whether to free EPC +* pages during reset. This will be called when domain goes into S3-S5 state +* (with free_epc being false), and when domain is destroyed (with free_epc +* being true). +* +* It is possible that EREMOVE will be called for SECS when it still has +* children present, in which case SGX_CHILD_PRESENT will be returned. In this +* case, SECS page is kept to a tmp list and after all EPC pages have been +* called with EREMOVE, we call EREMOVE for all the SECS pages again, and this +* time SGX_CHILD_PRESENT should never occur as all children should have been +* removed. +* +* If unexpected error returned by EREMOVE, it means the EPC page becomes +* abnormal, so it will not be freed even free_epc is true, as further use of +* this EPC can cause unexpected error, potentially damaging other domains. +*/ +static int __hvm_reset_epc (struct domain *d,unsigned long epc_base_pfn, +unsigned long epc_npages,bool_t free_epc) +{ +struct list_head secs_list; +struct list_head *p, *tmp; +unsigned long i; +int ret =0 ; + +INIT_LIST_HEAD (&secs_list); + +for ( i =0 ; i < epc_npages; i++ ) + { +struct epc_page *epg; +unsigned long gfn; +mfn_t mfn; +p2m_type_t t; +int r; + + gfn = i + epc_base_pfn; + mfn =get_gfn_query (d, gfn, &t); +if (unlikely (mfn_eq (mfn,INVALID_MFN )) ) + { +printk ("Domain %d: Reset EPC error: invalid MFN for gfn 0x%lx\n" , + d->domain_id, gfn); +put_gfn (d, gfn); + ret = -EFAULT ; +continue ; + } + +if (unlikely (!p2m_is_epc (t)) ) + { +printk ("Domain %d: Reset EPC error: (gfn 0x%lx, mfn 0x%lx): " +"is not p2m_epc.\n" , d->domain_id, gfn,mfn_x (mfn)); +put_gfn (d, gfn); + ret = -EFAULT ; +continue ; + } + +put_gfn (d, gfn); + + epg =epc_mfn_to_page (mfn_x (mfn)); + +/* EREMOVE the EPC page to make it invalid */ + r =sgx_eremove (epg); +if ( r ==SGX_CHILD_PRESENT ) + { +list_add_tail (&epg->list, &secs_list); +continue ; + } + +if ( r ) + { +printk ("Domain %d: Reset EPC error: (gfn 0x%lx, mfn 0x%lx): " +"EREMOVE returns %d\n" , d->domain_id, gfn,mfn_x (mfn), r); + ret = r; +if ( free_epc ) +printk ("WARNING: EPC (mfn 0x%lx) becomes abnormal. " +"Remove it from useable EPC." ,mfn_x (mfn)); +continue ; + } + +if ( free_epc ) + { +/* If EPC page is going to be freed, then also remove the mapping */ +if (clear_epc_p2m_entry (d, gfn, mfn) ) + { +printk ("Domain %d: Reset EPC error: (gfn 0x%lx, mfn 0x%lx): " +"clear p2m entry failed.\n" , d->domain_id, gfn, +mfn_x (mfn)); + ret = -EFAULT ; + } +free_epc_page (epg); + } + } + +list_for_each_safe (p, tmp, &secs_list) + { + struct epc_page *epg = list_entry(p, struct epc_page, list); +int r; + + r =sgx_eremove (epg); +if ( r ) + { +printk ("Domain %d: Reset EPC error: mfn 0x%lx: " +"EREMOVE returns %d for SECS page\n" , + d->domain_id,epc_page_to_mfn (epg), r); + ret = r; +list_del (p); + +if ( free_epc ) +printk ("WARNING: EPC (mfn 0x%lx) becomes abnormal. " +"Remove it from useable EPC." , +epc_page_to_mfn (epg)); +continue ; + } + +if ( free_epc ) +free_epc_page (epg); + } + +return ret; +} + +static void __hvm_unpopulate_epc (struct domain *d,unsigned long epc_base_pfn, +unsigned long populated_npages) +{ +unsigned long i; + +for ( i =0 ; i < populated_npages; i++ ) + { +struct epc_page *epg; +unsigned long gfn; +mfn_t mfn; +p2m_type_t t; + + gfn = i + epc_base_pfn; + mfn =get_gfn_query (d, gfn, &t); +if (unlikely (mfn_eq (mfn,INVALID_MFN )) ) + { +/* +* __hvm_unpopulate_epc only called when creating the domain on +* failure, therefore we can just ignore this error. +*/ +printk ("%s: Domain %u gfn 0x%lx returns invalid mfn\n" , __func__, + d->domain_id, gfn); +put_gfn (d, gfn); +continue ; + } + +if (unlikely (!p2m_is_epc (t)) ) + { +printk ("%s: Domain %u gfn 0x%lx returns non-EPC p2m type: %d\n" , + __func__, d->domain_id, gfn, (int )t); +put_gfn (d, gfn); +continue ; + } + +put_gfn (d, gfn); + +if (clear_epc_p2m_entry (d, gfn, mfn) ) + { +printk ("clear_epc_p2m_entry failed: gfn 0x%lx, mfn 0x%lx\n" , + gfn,mfn_x (mfn)); +continue ; + } + + epg =epc_mfn_to_page (mfn_x (mfn)); +free_epc_page (epg); + } +} + +static int __hvm_populate_epc (struct domain *d,unsigned long epc_base_pfn, +unsigned long epc_npages) +{ +unsigned long i; +int ret; + +for ( i =0 ; i < epc_npages; i++ ) + { +struct epc_page *epg =alloc_epc_page (); +unsigned long mfn; + +if ( !epg ) + { +printk ("%s: Out of EPC\n" , __func__); + ret = -ENOMEM ; + goto err; + } + + mfn =epc_page_to_mfn (epg); + ret =set_epc_p2m_entry (d, i + epc_base_pfn,_mfn (mfn)); +if ( ret ) + { +printk ("%s: set_epc_p2m_entry failed with %d: gfn 0x%lx, " +"mfn 0x%lx\n" , __func__, ret, i + epc_base_pfn, mfn); +free_epc_page (epg); + goto err; + } + } + +return 0 ; + +err: +__hvm_unpopulate_epc (d, epc_base_pfn, i); +return ret; +} + +int hvm_populate_epc (struct domain *d,unsigned long epc_base_pfn, +unsigned long epc_npages) +{ +struct sgx_domain *sgx =to_sgx (d); +int ret; + +if (hvm_epc_populated (d) ) +return -EBUSY ; + +if ( !epc_base_pfn || !epc_npages ) +return -EINVAL ; + +if ( (ret =__hvm_populate_epc (d, epc_base_pfn, epc_npages)) ) +return ret; + + sgx->epc_base_pfn = epc_base_pfn; + sgx->epc_npages = epc_npages; + +return 0 ; +} + +/* +* +* +* This function returns error immediately if there's any unexpected error +* during this process. +*/ +int hvm_reset_epc (struct domain *d,bool_t free_epc) +{ +struct sgx_domain *sgx =to_sgx (d); + +if ( !hvm_epc_populated (d) ) +return 0 ; + +return __hvm_reset_epc (d, sgx->epc_base_pfn, sgx->epc_npages, free_epc); +} + +void hvm_destroy_epc (struct domain *d) +{ +hvm_reset_epc (d, true); +} +static bool_t sgx_enabled_in_bios (void ) { uint64_t val, sgx_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE |
xen/arch/x86/hvm/vmx/vmx.c +3 -0
@@ -416,6 +416,9 @@ static int vmx_domain_initialise(struct domain *d)static void vmx_domain_destroy (struct domain *d) { +if (hvm_epc_populated (d) ) +hvm_destroy_epc (d); +if ( !has_vlapic (d) )return ;
xen/include/asm-x86/hvm/vmx/sgx.h +14 -0
@@ -13,6 +13,7 @@#include <xen/init.h> #include <asm/processor.h> #include <xen/list.h> +#include <public/hvm/params.h> /* HVM_PARAM_SGX */ #define SGX_CPUID 0x12
@@ -61,4 +62,17 @@ struct epc_page *epc_mfn_to_page(unsigned long mfn);void *map_epc_page_to_xen (struct epc_page *epg);void unmap_epc_page (void *addr); +struct sgx_domain { +unsigned long epc_base_pfn; +unsigned long epc_npages; +}; + +#define to_sgx (d) (&((d)->arch.hvm_domain.vmx.sgx)) +#define hvm_epc_populated (d) (!!((d)->arch.hvm_domain.vmx.sgx.epc_base_pfn)) + +int hvm_populate_epc (struct domain *d,unsigned long epc_base_pfn, +unsigned long epc_npages); +int hvm_reset_epc (struct domain *d,bool_t free_epc); +void hvm_destroy_epc (struct domain *d); +#endif /* __ASM_X86_HVM_VMX_SGX_H__ */
xen/include/asm-x86/hvm/vmx/vmcs.h +2 -0
@@ -20,6 +20,7 @@#include <asm/hvm/io.h> #include <irq_vectors.h> +#include <asm/hvm/vmx/sgx.h> extern void vmcs_dump_vcpu (struct vcpu *v);extern void setup_vmcs_dump (void );
@@ -62,6 +63,7 @@ struct vmx_domain {unsigned long apic_access_mfn;/* VMX_DOMAIN_* */ unsigned int status; +struct sgx_domain sgx; };struct pi_desc {
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 9/15 xen: vmx: handle SGX related MSRs
In reply to Kai HuangThis patch handles IA32_FEATURE_CONTROL and IA32_SGXLEPUBKEYHASHn MSRs.
For IA32_FEATURE_CONTROL, if SGX is exposed to domain, then SGX_ENABLE bit is always set. If SGX launch control is also exposed to domain, and physical IA32_SGXLEPUBKEYHASHn are writable, then SGX_LAUNCH_CONTROL_ENABLE bit is also always set. Write to IA32_FEATURE_CONTROL is ignored.
For IA32_SGXLEPUBKEYHASHn, a new 'struct sgx_vcpu' is added for per-vcpu SGX staff, and currently it has vcpu's virtual ia32_sgxlepubkeyhash[0-3]. Two boolean 'readable' and 'writable' are also added to indicate whether virtual IA32_SGXLEPUBKEYHASHn are readable and writable.
During vcpu is initialized, virtual ia32_sgxlepubkeyhash are also initialized. If physical IA32_SGXLEPUBKEYHASHn are writable, then ia32_sgxlepubkeyhash are set to Intel's default value, as for physical machine, those MSRs will have Intel's default value. If physical MSRs are not writable (it is *locked* by BIOS before handling to Xen), then we try to read those MSRs and use physical values as defult value for virtual MSRs. One thing is rdmsr_safe is used, as although SDM says if SGX is present, IA32_SGXLEPUBKEYHASHn are available for read, but in reality, skylake client (at least some, depending on BIOS) doesn't have those MSRs available, so we use rdmsr_safe and set readable to false if it returns error code.
For IA32_SGXLEPUBKEYHASHn MSR read from guest, if physical MSRs are not readable, guest is not allowed to read either, otherwise vcpu's virtual MSR value is returned.
For IA32_SGXLEPUBKEYHASHn MSR write from guest, we allow guest to write if both physical MSRs are writable and SGX launch control is exposed to domain, otherwise error is injected.
To make EINIT run successfully in guest, vcpu's virtual IA32_SGXLEPUBKEYHASHn will be update to physical MSRs when vcpu is scheduled in.
--- xen/arch/x86/hvm/vmx/sgx.c | 194 +++++++++++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmx.c | 24 +++++ xen/include/asm-x86/cpufeature.h | 3 + xen/include/asm-x86/hvm/vmx/sgx.h | 22 +++++ xen/include/asm-x86/hvm/vmx/vmcs.h | 2 + xen/include/asm-x86/msr-index.h | 6 ++ 6 files changed, 251 insertions(+)
xen/arch/x86/hvm/vmx/sgx.c +194 -0
@@ -405,6 +405,200 @@ void hvm_destroy_epc(struct domain *d)hvm_reset_epc (d, true); } +/* Whether IA32_SGXLEPUBKEYHASHn are physically *unlocked* by BIOS */ +bool_t sgx_ia32_sgxlepubkeyhash_writable (void ) +{ +uint64_t sgx_lc_enabled =IA32_FEATURE_CONTROL_SGX_ENABLE | +IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE | +IA32_FEATURE_CONTROL_LOCK ; +uint64_t val; + +rdmsrl (MSR_IA32_FEATURE_CONTROL , val); + +return (val & sgx_lc_enabled) == sgx_lc_enabled; +} + +bool_t domain_has_sgx (struct domain *d) +{ +/* hvm_epc_populated(d) implies CPUID has SGX */ +return hvm_epc_populated (d); +} + +bool_t domain_has_sgx_launch_control (struct domain *d) +{ +struct cpuid_policy *p = d->arch.cpuid; + +if ( !domain_has_sgx (d) ) +return false; + +/* Unnecessary but check anyway */ +if ( !cpu_has_sgx_launch_control ) +return false; + +return !!p->feat.sgx_launch_control; +} + +/* Digest of Intel signing key. MSR's default value after reset. */ +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH0 0xa6053e051270b7ac +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH1 0x6cfbe8ba8b3b413d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH2 0xc4916d99f2b3735d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH3 0xd4f8c05909f9bb3b + +void sgx_vcpu_init (struct vcpu *v) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); + +memset (sgxv,0 ,sizeof (*sgxv)); + +if (sgx_ia32_sgxlepubkeyhash_writable () ) + { +/* +* If physical MSRs are writable, set vcpu's default value to Intel's +* default value. For real machine, after reset, MSRs contain Intel's +* default value. +*/ + sgxv->ia32_sgxlepubkeyhash[0 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH0 ; + sgxv->ia32_sgxlepubkeyhash[1 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH1 ; + sgxv->ia32_sgxlepubkeyhash[2 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH2 ; + sgxv->ia32_sgxlepubkeyhash[3 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH3 ; + + sgxv->readable =1 ; + sgxv->writable =domain_has_sgx_launch_control (v->domain); + } +else + { +uint64_t v; +/* +* Although SDM says if SGX is present, then IA32_SGXLEPUBKEYHASHn are +* available for read, but in reality for SKYLAKE client machines, +* those MSRs are not available if SGX is present, so we cannot rely on +* cpu_has_sgx to determine whether to we are able to read MSRs, +* instead, we always use rdmsr_safe. +*/ + sgxv->readable =rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH0 , v) ?0 :1 ; + +if ( !sgxv->readable ) +return ; + +rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH0 , sgxv->ia32_sgxlepubkeyhash[0 ]); +rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH1 , sgxv->ia32_sgxlepubkeyhash[1 ]); +rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH2 , sgxv->ia32_sgxlepubkeyhash[2 ]); +rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH3 , sgxv->ia32_sgxlepubkeyhash[3 ]); + } +} + +void sgx_ctxt_switch_to (struct vcpu *v) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); + +if ( sgxv->writable &&sgx_ia32_sgxlepubkeyhash_writable () ) + { +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 , sgxv->ia32_sgxlepubkeyhash[0 ]); +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH1 , sgxv->ia32_sgxlepubkeyhash[1 ]); +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH2 , sgxv->ia32_sgxlepubkeyhash[2 ]); +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH3 , sgxv->ia32_sgxlepubkeyhash[3 ]); + } +} + +int sgx_msr_read_intercept (struct vcpu *v,unsigned int msr,u64 *msr_content) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); +u64 data; +int r =1 ; + +if ( !domain_has_sgx (v->domain) ) +return 0 ; + +switch ( msr ) + { +case MSR_IA32_FEATURE_CONTROL : + data = (IA32_FEATURE_CONTROL_LOCK | +IA32_FEATURE_CONTROL_SGX_ENABLE ); +/* +* If physical IA32_SGXLEPUBKEYHASHn are writable, then we always +* allow guest to be able to change IA32_SGXLEPUBKEYHASHn at runtime. +*/ +if (sgx_ia32_sgxlepubkeyhash_writable () && +domain_has_sgx_launch_control (v->domain) ) + data |=IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE ; + + *msr_content = data; + +break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +/* +* SDM 35.1 Model-Specific Registers, table 35-2. +* +* IA32_SGXLEPUBKEYHASH[0..3]: +* +* Read permitted if CPUID.0x12.0:EAX[0] = 1. +* +* In reality, MSRs may not be readable even SGX is present, in which +* case guest is not allowed to read either. +*/ +if ( !sgxv->readable ) + { + r =0 ; +break ; + } + + data = sgxv->ia32_sgxlepubkeyhash[msr -MSR_IA32_SGXLEPUBKEYHASH0 ]; + + *msr_content = data; + +break ; +default : + r =0 ; +break ; + } + +return r; +} + +int sgx_msr_write_intercept (struct vcpu *v,unsigned int msr,u64 msr_content) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); +int r =1 ; + +if ( !domain_has_sgx (v->domain) ) +return 0 ; + +switch ( msr ) + { +case MSR_IA32_FEATURE_CONTROL : +/* sliently drop */ +break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +/* +* SDM 35.1 Model-Specific Registers, table 35-2. +* +* IA32_SGXLEPUBKEYHASH[0..3]: +* +* - If CPUID.0x7.0:ECX[30] = 1, FEATURE_CONTROL[17] is available. +* - Write permitted if CPUID.0x12.0:EAX[0] = 1 && +* FEATURE_CONTROL[17] = 1 && FEATURE_CONTROL[0] = 1. +* +* sgxv->writable == 1 means sgx_ia32_sgxlepubkeyhash_writable() and +* domain_has_sgx_launch_control(d) both are true. +*/ +if ( !sgxv->writable ) + { + r =0 ; +break ; + } + + sgxv->ia32_sgxlepubkeyhash[msr -MSR_IA32_SGXLEPUBKEYHASH0 ] = + msr_content; + +break ; +default : + r =0 ; +break ; + } + +return r; +} +static bool_t sgx_enabled_in_bios (void ) { uint64_t val, sgx_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE |
xen/arch/x86/hvm/vmx/vmx.c +24 -0
@@ -470,6 +470,8 @@ static int vmx_vcpu_initialise(struct vcpu *v)if ( v->vcpu_id ==0 ) v->arch.user_regs.rax =1 ; +sgx_vcpu_init (v); +return 0 ; }
@@ -1048,6 +1050,9 @@ static void vmx_ctxt_switch_to(struct vcpu *v)if ( v->domain->arch.hvm_domain.pi_ops.switch_to ) v->domain->arch.hvm_domain.pi_ops.switch_to (v); + +if (domain_has_sgx (v->domain) ) +sgx_ctxt_switch_to (v); }
@@ -2876,10 +2881,20 @@ static int vmx_msr_read_intercept(unsigned int msr, uint64_t *msr_content)__vmread (GUEST_IA32_DEBUGCTL , msr_content);break ;case MSR_IA32_FEATURE_CONTROL : +/* If neither SGX nor nested is supported, this MSR should not be +* touched */ +if ( !sgx_msr_read_intercept (current, msr, msr_content) && + !nvmx_msr_read_intercept (msr, msr_content) ) + goto gp_fault; +break ; case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_VMFUNC:if ( !nvmx_msr_read_intercept (msr, msr_content) ) goto gp_fault;break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +if ( !sgx_msr_read_intercept (current, msr, msr_content) ) + goto gp_fault; +break ;case MSR_IA32_MISC_ENABLE :rdmsrl (MSR_IA32_MISC_ENABLE , *msr_content);/* Debug Trace Store is not supported. */
@@ -3119,10 +3134,19 @@ static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content)break ; }case MSR_IA32_FEATURE_CONTROL : +/* See vmx_msr_read_intercept */ +if ( !sgx_msr_write_intercept (current, msr, msr_content) && + !nvmx_msr_write_intercept (msr, msr_content) ) + goto gp_fault; +break ; case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_TRUE_ENTRY_CTLS:if ( !nvmx_msr_write_intercept (msr, msr_content) ) goto gp_fault;break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +if ( !sgx_msr_write_intercept (current, msr, msr_content) ) + goto gp_fault; +break ; case MSR_P6_PERFCTR(0)...MSR_P6_PERFCTR(7): case MSR_P6_EVNTSEL(0)...MSR_P6_EVNTSEL(7): case MSR_CORE_PERF_FIXED_CTR0...MSR_CORE_PERF_FIXED_CTR2:
xen/include/asm-x86/cpufeature.h +3 -0
@@ -98,6 +98,9 @@#define cpu_has_smap boot_cpu_has(X86_FEATURE_SMAP)#define cpu_has_sha boot_cpu_has(X86_FEATURE_SHA) +/* CPUID level 0x00000007:0.ecx */ +#define cpu_has_sgx_launch_control boot_cpu_has(X86_FEATURE_SGX_LAUNCH_CONTROL) +/* CPUID level 0x80000007.edx */ #define cpu_has_itsc boot_cpu_has(X86_FEATURE_ITSC)
xen/include/asm-x86/hvm/vmx/sgx.h +22 -0
@@ -75,4 +75,26 @@ int hvm_populate_epc(struct domain *d, unsigned long epc_base_pfn,int hvm_reset_epc (struct domain *d,bool_t free_epc);void hvm_destroy_epc (struct domain *d); +/* Per-vcpu SGX structure */ +struct sgx_vcpu { +uint64_t ia32_sgxlepubkeyhash[4 ]; +/* +* Although SDM says if SGX is present, then IA32_SGXLEPUBKEYHASHn are +* available for read, but in reality for SKYLAKE client machines, those +* those MSRs are not available if SGX is present. +*/ +bool_t readable; +bool_t writable; +}; +#define to_sgx_vcpu (v) (&(v->arch.hvm_vmx.sgx)) + +bool_t sgx_ia32_sgxlepubkeyhash_writable (void ); +bool_t domain_has_sgx (struct domain *d); +bool_t domain_has_sgx_launch_control (struct domain *d); + +void sgx_vcpu_init (struct vcpu *v); +void sgx_ctxt_switch_to (struct vcpu *v); +int sgx_msr_read_intercept (struct vcpu *v,unsigned int msr,u64 *msr_content); +int sgx_msr_write_intercept (struct vcpu *v,unsigned int msr,u64 msr_content); +#endif /* __ASM_X86_HVM_VMX_SGX_H__ */
xen/include/asm-x86/hvm/vmx/vmcs.h +2 -0
@@ -160,6 +160,8 @@ struct arch_vmx_struct {* pCPU and wakeup the related vCPU. */ struct pi_blocking_vcpu pi_blocking; + +struct sgx_vcpu sgx; };int vmx_create_vmcs (struct vcpu *v);
xen/include/asm-x86/msr-index.h +6 -0
@@ -296,6 +296,12 @@#define IA32_FEATURE_CONTROL_SENTER_PARAM_CTL 0x7f00#define IA32_FEATURE_CONTROL_ENABLE_SENTER 0x8000#define IA32_FEATURE_CONTROL_SGX_ENABLE 0x40000 +#define IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE 0x20000 + +#define MSR_IA32_SGXLEPUBKEYHASH0 0x0000008c +#define MSR_IA32_SGXLEPUBKEYHASH1 0x0000008d +#define MSR_IA32_SGXLEPUBKEYHASH2 0x0000008e +#define MSR_IA32_SGXLEPUBKEYHASH3 0x0000008f#define MSR_IA32_TSC_ADJUST 0x0000003b
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 10/15 xen: vmx: handle ENCLS VMEXIT
In reply to Kai HuangCurrently EPC are statically allocated and mapped to guest, we don't have to trap ENCLS as it runs perfectly in VMX non-root mode. But exposing SGX to guest means we also expose ENABLE_ENCLS bit to L1 hypervisor, therefore we cannot stop L1 from enabling ENCLS VMEXIT. For ENCLS VMEXIT from L2 guest, we simply inject it to L1, otherwise the ENCLS VMEXIT is unexpected in L0 and we simply crash the domain.
--- xen/arch/x86/hvm/vmx/vmx.c | 10 ++++++++++ xen/arch/x86/hvm/vmx/vvmx.c | 11 +++++++++++ xen/include/asm-x86/hvm/vmx/vmcs.h | 1 + xen/include/asm-x86/hvm/vmx/vmx.h | 1 + 4 files changed, 23 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +10 -0
@@ -4126,6 +4126,16 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)vmx_handle_apic_write ();break ; +case EXIT_REASON_ENCLS : +/* +* Currently L0 doesn't turn on ENCLS VMEXIT, but L0 cannot stop L1 +* from enabling ENCLS VMEXIT. ENCLS VMEXIT from L2 guest has already +* been handled so by reaching here it is a BUG. We simply crash the +* domain. +*/ +domain_crash (v->domain); +break ; +case EXIT_REASON_PML_FULL :vmx_vcpu_flush_pml_buffer (v);break ;
xen/arch/x86/hvm/vmx/vvmx.c +11 -0
@@ -2059,6 +2059,12 @@ int nvmx_msr_read_intercept(unsigned int msr, u64 *msr_content)SECONDARY_EXEC_ENABLE_VPID |SECONDARY_EXEC_UNRESTRICTED_GUEST |SECONDARY_EXEC_ENABLE_EPT ; +/* +* If SGX is exposed to guest, then ENABLE_ENCLS bit must also be +* exposed to guest. +*/ +if (domain_has_sgx (d) ) + data |=SECONDARY_EXEC_ENABLE_ENCLS ; data =gen_vmx_msr (data,0 , host_data);break ;case MSR_IA32_VMX_EXIT_CTLS :
@@ -2291,6 +2297,11 @@ int nvmx_n2_vmexit_handler(struct cpu_user_regs *regs,case EXIT_REASON_VMXON :case EXIT_REASON_INVEPT :case EXIT_REASON_XSETBV : +/* +* L0 doesn't turn on ENCLS VMEXIT now, so ENCLS VMEXIT must come from +* L2 guest, and is because of ENCLS VMEXIT is turned on by L1. +*/ +case EXIT_REASON_ENCLS :/* inject to L1 */ nvcpu->nv_vmexit_pending =1 ;break ;
xen/include/asm-x86/hvm/vmx/vmcs.h +1 -0
@@ -396,6 +396,7 @@ enum vmcs_field {VIRT_EXCEPTION_INFO =0x0000202a ,XSS_EXIT_BITMAP =0x0000202c ,TSC_MULTIPLIER =0x00002032 , +ENCLS_EXITING_BITMAP =0x0000202E ,GUEST_PHYSICAL_ADDRESS =0x00002400 ,VMCS_LINK_POINTER =0x00002800 , GUEST_IA32_DEBUGCTL = 0x00002802,
xen/include/asm-x86/hvm/vmx/vmx.h +1 -0
@@ -210,6 +210,7 @@ static inline void pi_clear_sn(struct pi_desc *pi_desc)#define EXIT_REASON_APIC_WRITE 56#define EXIT_REASON_INVPCID 58#define EXIT_REASON_VMFUNC 59 +#define EXIT_REASON_ENCLS 60#define EXIT_REASON_PML_FULL 62#define EXIT_REASON_XSAVES 63#define EXIT_REASON_XRSTORS 64
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 11/15 xen: vmx: handle VMEXIT from SGX enclave
In reply to Kai HuangVMX adds new bit to both exit_reason and GUEST_INTERRUPT_STATE to indicate whether VMEXIT happens in Enclave. Several instructions are also invalid or behave differently in enclave according to SDM. This patch handles those cases.
--- xen/arch/x86/hvm/vmx/vmx.c | 29 +++++++++++++++++++++++++++++ xen/include/asm-x86/hvm/vmx/vmcs.h | 2 ++ xen/include/asm-x86/hvm/vmx/vmx.h | 2 ++ 3 files changed, 33 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +29 -0
@@ -57,6 +57,7 @@#include <asm/event.h> #include <asm/monitor.h> #include <public/arch-x86/cpuid.h> +#include <asm/hvm/vmx/sgx.h> static bool_t __initdata opt_force_ept;boolean_param ("force-ept" , opt_force_ept);
@@ -3544,6 +3545,7 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)unsigned long exit_qualification, exit_reason, idtv_info, intr_info =0 ;unsigned int vector =0 , mode;struct vcpu *v = current; +bool_t exit_from_sgx_enclave;__vmread (GUEST_RIP , ®s->rip);__vmread (GUEST_RSP , ®s->rsp);
@@ -3569,6 +3571,11 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)perfc_incra (vmexits, exit_reason); +/* We need to handle several VMEXITs if VMEXIT is from enclave. Also clear +* bit 27 as it is further useless. */ + exit_from_sgx_enclave = !!(exit_reason &VMX_EXIT_REASONS_FROM_ENCLAVE ); + exit_reason &= ~VMX_EXIT_REASONS_FROM_ENCLAVE ; +/* Handle the interrupt we missed before allowing any more in. */ switch ( (uint16_t )exit_reason ) {
@@ -4070,6 +4077,18 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)break ;case EXIT_REASON_INVD : +/* +* SDM 39.6.5 INVD Handling when Enclave Are Enabled +* +* INVD cause #GP if EPC is enabled. +* FIXME: WBINVD?? +*/ +if ( exit_from_sgx_enclave ) + { +hvm_inject_hw_exception (TRAP_gp_fault,0 ); +break ; + } +/* Otherwise passthrough */ case EXIT_REASON_WBINVD : {update_guest_eip ();/* Safe: INVD, WBINVD */
@@ -4081,6 +4100,16 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs) {paddr_t gpa; +/* +* Currently EPT violation from enclave is not possible as all EPC pages +* are statically allocated to guest when guest is created. We simply +* crash guest in this case. +*/ +if ( exit_from_sgx_enclave ) + { +domain_crash (v->domain); +break ; + }__vmread (GUEST_PHYSICAL_ADDRESS , &gpa);__vmread (EXIT_QUALIFICATION , &exit_qualification);ept_handle_violation (exit_qualification, gpa);
xen/include/asm-x86/hvm/vmx/vmcs.h +2 -0
@@ -327,6 +327,8 @@ extern u64 vmx_ept_vpid_cap;#define VMX_INTR_SHADOW_MOV_SS 0x00000002#define VMX_INTR_SHADOW_SMI 0x00000004#define VMX_INTR_SHADOW_NMI 0x00000008 +#define VMX_INTR_ENCLAVE_INTR 0x00000010/* VMEXIT was incident to +enclave mode */ #define VMX_BASIC_REVISION_MASK 0x7fffffff#define VMX_BASIC_VMCS_SIZE_MASK (0x1fffULL << 32)
xen/include/asm-x86/hvm/vmx/vmx.h +2 -0
@@ -153,6 +153,8 @@ static inline void pi_clear_sn(struct pi_desc *pi_desc)* Exit Reasons */ #define VMX_EXIT_REASONS_FAILED_VMENTRY 0x80000000 +/* Bit 27 is also set if VMEXIT is from SGX enclave mode */ +#define VMX_EXIT_REASONS_FROM_ENCLAVE 0x08000000#define EXIT_REASON_EXCEPTION_NMI 0#define EXIT_REASON_EXTERNAL_INTERRUPT 1
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 12/15 xen: x86: reset EPC when guest got suspended.
In reply to Kai HuangEPC is destroyed when power state goes to S3-S5. Emulate this behavior.
A new function s3_suspend is added to hvm_function_table for this purpose.
--- xen/arch/x86/hvm/hvm.c | 3 +++ xen/arch/x86/hvm/vmx/vmx.c | 7 +++++++ xen/include/asm-x86/hvm/hvm.h | 3 +++ 3 files changed, 13 insertions(+)
xen/arch/x86/hvm/hvm.c +3 -0
@@ -3858,6 +3858,9 @@ static void hvm_s3_suspend(struct domain *d)hvm_vcpu_reset_state (d->vcpu[0 ],0xf000 ,0xfff0 ); +if ( hvm_funcs.s3_suspend ) + hvm_funcs.s3_suspend (d); +domain_unlock (d); }
xen/arch/x86/hvm/vmx/vmx.c +7 -0
@@ -2290,6 +2290,12 @@ static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info)return true; } +static void vmx_s3_suspend (struct domain *d) +{ +if (domain_has_sgx (d) ) +hvm_reset_epc (d, false); +} +static struct hvm_function_table __initdata vmx_function_table = { .name ="VMX" , .cpu_up_prepare = vmx_cpu_up_prepare,
@@ -2360,6 +2366,7 @@ static struct hvm_function_table __initdata vmx_function_table = { .max_ratio = VMX_TSC_MULTIPLIER_MAX, .setup = vmx_setup_tsc_scaling, }, + .s3_suspend = vmx_s3_suspend, };/* Handle VT-d posted-interrupt when VCPU is blocked. */
xen/include/asm-x86/hvm/hvm.h +3 -0
@@ -226,6 +226,9 @@ struct hvm_function_table {/* Architecture function to setup TSC scaling ratio */ void (*setup)(struct vcpu *v); } tsc_scaling; + +/* Domain S3 suspend */ +void (*s3_suspend)(struct domain *d); };extern struct hvm_function_table hvm_funcs;
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 4/15 xen: mm: add ioremap_cache
In reply to Kai HuangCurrently Xen only has non-cacheable version of ioremap. Although EPC is reported as reserved memory in e820 but it can be mapped as cacheable. This patch adds ioremap_cache (cacheable version of ioremap).
--- xen/arch/x86/mm.c | 15 +++++++++++++-- xen/include/xen/vmap.h | 1 + 2 files changed, 14 insertions(+), 2 deletions(-)
xen/arch/x86/mm.c +13 -2
@@ -6284,9 +6284,10 @@ void *__init arch_vmap_virt_end(void)return (void *)fix_to_virt (__end_of_fixed_addresses); } -void __iomem *ioremap (paddr_t pa,size_t len) +static void __iomem *__ioremap (paddr_t pa,size_t len,bool_t cache) {mfn_t mfn =_mfn (PFN_DOWN (pa)); +unsigned int flags = cache ?PAGE_HYPERVISOR :PAGE_HYPERVISOR_NOCACHE ;void *va;WARN_ON (page_is_ram_type (mfn_x (mfn),RAM_TYPE_CONVENTIONAL ));
@@ -6299,12 +6300,22 @@ void __iomem *ioremap(paddr_t pa, size_t len)unsigned int offs = pa & (PAGE_SIZE -1 );unsigned int nr =PFN_UP (offs + len); - va =__vmap (&mfn, nr,1 ,1 ,PAGE_HYPERVISOR_NOCACHE ,VMAP_DEFAULT ) + offs; + va =__vmap (&mfn, nr,1 ,1 , flags,VMAP_DEFAULT ) + offs; }return (void __force __iomem *)va; } +void __iomem *ioremap (paddr_t pa,size_t len) +{ +return __ioremap (pa, len, false); +} + +void __iomem *ioremap_cache (paddr_t pa,size_t len) +{ +return __ioremap (pa, len, true); +} +int create_perdomain_mapping (struct domain *d,unsigned long va,unsigned int nr,l1_pgentry_t **pl1tab,struct page_info **ppg)
xen/include/xen/vmap.h +1 -0
@@ -24,6 +24,7 @@ void *vzalloc(size_t size);void vfree (void *va);void __iomem *ioremap (paddr_t ,size_t ); +void __iomem *ioremap_cache (paddr_t ,size_t );static inline void iounmap (void __iomem *va) {
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 8/15 xen: x86: add SGX cpuid handling support.
This patch adds SGX to cpuid handling support. In init_guest_cpuid, for raw_policy and host_policy, physical EPC info is reported, but for pv_max_policy and hvm_max_policy EPC is hidden, as for particular domain, it's EPC base and size are from tookstack, and it is meaningless to contain physical EPC info in them. Before domain's EPC base and size are properly configured, guest's SGX cpuid should report invalid EPC, which is also consistent with HW behavior.
Currently all EPC pages are fully populated for domain when it is created. Xen gets domain's EPC base and size from toolstack via XEN_DOMCTL_set_cpuid, so domain's EPC pages are also populated in XEN_DOMCTL_set_cpuid, after receiving valid EPC base and size. Failure to populate EPC (such as there's no enough free EPC pages) results in domain creation failure by making XEN_DOMCTL_set_cpuid return error.
--- xen/arch/x86/cpuid.c | 87 ++++++++++++++++++++++++++++++++++++++++++++- xen/arch/x86/domctl.c | 47 +++++++++++++++++++++++- xen/include/asm-x86/cpuid.h | 26 +++++++++++++- 3 files changed, 157 insertions(+), 3 deletions(-)
xen/arch/x86/cpuid.c +86 -1
@@ -9,6 +9,7 @@#include <asm/paging.h> #include <asm/processor.h> #include <asm/xstate.h> +#include <asm/hvm/vmx/sgx.h> const uint32_t known_features[] =INIT_KNOWN_FEATURES ;const uint32_t special_features[] =INIT_SPECIAL_FEATURES ;
@@ -158,6 +159,44 @@ static void recalculate_xstate(struct cpuid_policy *p) } } +static void recalculate_sgx (struct cpuid_policy *p,bool_t hide_epc) +{ +if ( !p->feat.sgx ) + { +memset (&p->sgx,0 ,sizeof (p->sgx)); +return ; + } + +if ( !p->sgx.sgx1 ) + { +memset (&p->sgx,0 ,sizeof (p->sgx)); +return ; + } + +/* +* SDM 42.7.2.1 SECS.ATTRIBUTE.XFRM: +* +* Legal value for SECS.ATTRIBUTE.XFRM conform to these requirements: +* - XFRM[1:0] must be set to 0x3; +* - If processor does not support XSAVE, or if the system software has not +* enabled XSAVE, then XFRM[63:2] must be 0. +* - If the processor does support XSAVE, XFRM must contain a value that +* would be legal if loaded into XCR0. +*/ + p->sgx.xfrm_low =0x3 ; + p->sgx.xfrm_high =0 ; +if ( p->basic.xsave ) + { + p->sgx.xfrm_low |= p->xstate.xcr0_low; + p->sgx.xfrm_high |= p->xstate.xcr0_high; + } + +if ( hide_epc ) + { +memset (&p->sgx.raw[0x2 ],0 ,sizeof (struct cpuid_leaf )); + } +} +/* * Misc adjustments to the policy. Mostly clobbering reserved fields and * duplicating shared fields. Intentionally hidden fields are annotated.
@@ -239,7 +278,7 @@ static void __init calculate_raw_policy(void) {switch ( i ) { -case 0x4 :case 0x7 :case 0xd : +case 0x4 :case 0x7 :case 0xd :case 0x12 :/* Multi-invocation leaves. Deferred. */ continue ; }
@@ -299,6 +338,19 @@ static void __init calculate_raw_policy(void) } } +if ( p->basic.max_leaf >=SGX_CPUID ) + { +/* +* For raw policy we just report native CPUID. For EPC on native it's +* possible that we will have multiple EPC sections (meaning subleaf 3, +* 4, ... may also be valid), but as the policy is for guest so we only +* need one EPC section (subleaf 2). +*/ +cpuid_count_leaf (SGX_CPUID ,0 , &p->sgx.raw[0 ]); +cpuid_count_leaf (SGX_CPUID ,0 , &p->sgx.raw[0 ]); +cpuid_count_leaf (SGX_CPUID ,0 , &p->sgx.raw[0 ]); + } +/* Extended leaves. */ cpuid_leaf (0x80000000 , &p->extd.raw[0 ]); for ( i = 1; i < min(ARRAY_SIZE(p->extd.raw),
@@ -324,6 +376,8 @@ static void __init calculate_host_policy(void)cpuid_featureset_to_policy (boot_cpu_data.x86_capability, p);recalculate_xstate (p);recalculate_misc (p); +/* For host policy we report physical EPC */ +recalculate_sgx (p,0 );if ( p->extd.svm ) {
@@ -357,6 +411,11 @@ static void __init calculate_pv_max_policy(void)sanitise_featureset (pv_featureset);cpuid_featureset_to_policy (pv_featureset, p);recalculate_xstate (p); +/* +* For PV policy we don't report physical EPC. Actually for PV policy +* currently SGX will be disabled. +*/ +recalculate_sgx (p,1 ); p->extd.raw[0xa ] =EMPTY_LEAF ;/* No SVM for PV guests. */ }
@@ -413,6 +472,13 @@ static void __init calculate_hvm_max_policy(void)sanitise_featureset (hvm_featureset);cpuid_featureset_to_policy (hvm_featureset, p);recalculate_xstate (p); +/* +* For HVM policy we don't report physical EPC. Actually cpuid policy +* should report VM's virtual EPC base and size. However VM's virtual +* EPC info will come from toolstack, and only after Xen is notified +* VM's cpuid policy should report invalid EPC. +*/ +recalculate_sgx (p,1 ); }void __init init_guest_cpuid (void )
@@ -528,6 +594,12 @@ void recalculate_cpuid_policy(struct domain *d)if ( p->basic.max_leaf <XSTATE_CPUID )__clear_bit (X86_FEATURE_XSAVE , fs); +if ( p->basic.max_leaf <SGX_CPUID ) + { +__clear_bit (X86_FEATURE_SGX , fs); +__clear_bit (X86_FEATURE_SGX_LAUNCH_CONTROL , fs); + } +sanitise_featureset (fs);/* Fold host's FDP_EXCP_ONLY and NO_FPU_SEL into guest's view. */
@@ -550,6 +622,12 @@ void recalculate_cpuid_policy(struct domain *d)recalculate_xstate (p);recalculate_misc (p); +/* +* recalculate_cpuid_policy is also called for domain's cpuid policy, +* which is from toolstack via XEN_DOMCTL_set_cpuid, therefore we cannot +* hide domain's virtual EPC from toolstack. +*/ +recalculate_sgx (p,0 );for ( i =0 ; i <ARRAY_SIZE (p->cache.raw); ++i ) {
@@ -645,6 +723,13 @@ void guest_cpuid(const struct vcpu *v, uint32_t leaf, *res = p->xstate.raw[subleaf];break ; +case SGX_CPUID : +if ( !p->feat.sgx ) +return ; + + *res = p->sgx.raw[subleaf]; +break ; +default : *res = p->basic.raw[leaf];break ;
xen/arch/x86/domctl.c +46 -1
77 lines not on this page · raw message
xen/include/asm-x86/cpuid.h +25 -1
@@ -61,10 +61,11 @@ extern struct cpuidmasks cpuidmask_defaults;/* Whether or not cpuid faulting is available for the current domain. */ DECLARE_PER_CPU (bool, cpuid_faulting_enabled); -#define CPUID_GUEST_NR_BASIC (0xdu + 1) +#define CPUID_GUEST_NR_BASIC (0x12u + 1)#define CPUID_GUEST_NR_FEAT (0u + 1)#define CPUID_GUEST_NR_CACHE (5u + 1)#define CPUID_GUEST_NR_XSTATE (62u + 1) +#define CPUID_GUEST_NR_SGX (0x2u + 1)#define CPUID_GUEST_NR_EXTD_INTEL (0x8u + 1)#define CPUID_GUEST_NR_EXTD_AMD (0x1cu + 1)#define CPUID_GUEST_NR_EXTD MAX(CPUID_GUEST_NR_EXTD_INTEL, \
@@ -169,6 +170,29 @@ struct cpuid_policy } comp[CPUID_GUEST_NR_XSTATE ]; } xstate; +union { +struct cpuid_leaf raw[CPUID_GUEST_NR_SGX ]; + +struct { +/* Subleaf 0. */ + uint32_t sgx1:1, sgx2:1, :30; + uint32_t miscselect, /* c */ :32; + uint32_t maxenclavesize_n64:8, maxenclavesize_64:8, :16; + +/* Subleaf 1. */ + uint32_t init:1, debug:1, mode64:1, /*reserve*/:1, provisionkey:1, + einittokenkey:1, :26; + uint32_t /* reserve */:32; +uint32_t xfrm_low, xfrm_high; + +/* Subleaf 2. */ + uint32_t base_valid:1, :11, base_pfn_low:20; + uint32_t base_pfn_high:20, :12; + uint32_t size_valid:1, :11, npages_low:20; + uint32_t npages_high:20, :12; + }; + } sgx; +/* Extended leaves: 0x800000xx */ union {struct cpuid_leaf raw[CPUID_GUEST_NR_EXTD ];
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 5/15 xen: p2m: new 'p2m_epc' type for EPC mapping
In reply to Kai HuangA new 'p2m_epc' type is added for EPC mapping type. Two wrapper functions set_epc_p2m_entry and clear_epc_p2m_entry are also added for further use.
--- xen/arch/x86/mm/p2m-ept.c | 3 +++ xen/arch/x86/mm/p2m.c | 41 +++++++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/p2m.h | 12 ++++++++++-- 3 files changed, 54 insertions(+), 2 deletions(-)
xen/arch/x86/mm/p2m-ept.c +3 -0
@@ -182,6 +182,9 @@ static void ept_p2m_type_to_flags(struct p2m_domain *p2m, ept_entry_t *entry, entry->a = !!cpu_has_vmx_ept_ad; entry->d =0 ;break ; +case p2m_epc: + entry->r = entry->w = entry->x =1 ; +break ; }
xen/arch/x86/mm/p2m.c +41 -0
53 lines not on this page · raw message
xen/include/asm-x86/p2m.h +10 -2
@@ -72,6 +72,7 @@ typedef enum { p2m_ram_broken =13 ,/* Broken page, access cause domain crash */ p2m_map_foreign =14 ,/* ram pages from foreign domain */ p2m_ioreq_server =15 , + p2m_epc =16 ,/* EPC */ } p2m_type_t;/* Modifiers to the query */
@@ -142,10 +143,13 @@ typedef unsigned int p2m_query_t; | p2m_to_mask(p2m_ram_logdirty) )#define P2M_SHARED_TYPES (p2m_to_mask(p2m_ram_shared)) +#define P2M_EPC_TYPES (p2m_to_mask(p2m_epc)) +/* Valid types not necessarily associated with a (valid) MFN. */ #define P2M_INVALID_MFN_TYPES (P2M_POD_TYPES \ | p2m_to_mask(p2m_mmio_direct) \ - | P2M_PAGING_TYPES) + | P2M_PAGING_TYPES \ + | P2M_EPC_TYPES)/* Broken type: the frame backing this pfn has failed in hardware * and must not be touched. */
@@ -153,6 +157,7 @@ typedef unsigned int p2m_query_t;/* Useful predicates */ #define p2m_is_ram (_t) (p2m_to_mask(_t) & P2M_RAM_TYPES) +#define p2m_is_epc (_t) (p2m_to_mask(_t) & P2M_EPC_TYPES)#define p2m_is_hole (_t) (p2m_to_mask(_t) & P2M_HOLE_TYPES)#define p2m_is_mmio (_t) (p2m_to_mask(_t) & P2M_MMIO_TYPES)#define p2m_is_readonly (_t) (p2m_to_mask(_t) & P2M_RO_TYPES)
@@ -163,7 +168,7 @@ typedef unsigned int p2m_query_t;/* Grant types are *not* considered valid, because they can be unmapped at any time and, unless you happen to be the shadow or p2m implementations, there's no way of synchronising against that. */ -#define p2m_is_valid (_t) (p2m_to_mask(_t) & (P2M_RAM_TYPES | P2M_MMIO_TYPES)) +#define p2m_is_valid (_t) (p2m_to_mask(_t) & (P2M_RAM_TYPES | P2M_MMIO_TYPES | P2M_EPC_TYPES))#define p2m_has_emt (_t) (p2m_to_mask(_t) & (P2M_RAM_TYPES | p2m_to_mask(p2m_mmio_direct)))#define p2m_is_pageable (_t) (p2m_to_mask(_t) & P2M_PAGEABLE_TYPES)#define p2m_is_paging (_t) (p2m_to_mask(_t) & P2M_PAGING_TYPES)
@@ -634,6 +639,9 @@ int clear_identity_p2m_entry(struct domain *d, unsigned long gfn);int p2m_add_foreign (struct domain *tdom,unsigned long fgfn,unsigned long gpfn,domid_t foreign_domid); +int set_epc_p2m_entry (struct domain *d,unsigned long gfn,mfn_t mfn); +int clear_epc_p2m_entry (struct domain *d,unsigned long gfn,mfn_t mfn); +/* * Populate-on-demand */
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 13/15 xen: tools: add new 'epc' parameter support
In reply to Kai HuangIn order to be able to configure domain's EPC size when it is created, a new 'epc' parameter is added to XL configuration file. Like memory it indicates EPC size in MB. A new 'libxl_sgx_buildinfo', which contains EPC base and size, is also added to libxl_domain_buind_info. EPC base and size are also added to 'xc_dom_image' in order to add EPC to e820 table. EPC base is calculated internally.
--- tools/libxc/include/xc_dom.h | 4 ++++ tools/libxl/libxl_create.c | 9 +++++++++ tools/libxl/libxl_dom.c | 30 ++++++++++++++++++++++++++++++ tools/libxl/libxl_internal.h | 2 ++ tools/libxl/libxl_types.idl | 6 ++++++ tools/libxl/libxl_x86.c | 12 ++++++++++++ tools/xl/xl_parse.c | 5 +++++ 7 files changed, 68 insertions(+)
tools/libxc/include/xc_dom.h +4 -0
@@ -203,6 +203,10 @@ struct xc_dom_image {xen_paddr_t lowmem_end;xen_paddr_t highmem_end;xen_pfn_t vga_hole_size; +#if defined(__i386__) || defined(__x86_64__) +xen_paddr_t epc_base; +xen_paddr_t epc_size; +#endif /* If unset disables the setup of the IOREQ pages. */ bool device_model;
tools/libxl/libxl_create.c +9 -0
@@ -59,6 +59,13 @@ void libxl__rdm_setdefault(libxl__gc *gc, libxl_domain_build_info *b_info)LIBXL_RDM_MEM_BOUNDARY_MEMKB_DEFAULT ; } +void libxl__sgx_setdefault (libxl__gc *gc,libxl_domain_build_info *b_info) +{ +if (b_info->u.hvm.sgx.epckb ==LIBXL_MEMKB_DEFAULT ) + b_info->u.hvm.sgx.epckb =0 ; + b_info->u.hvm.sgx.epcbase =0 ; +} +int libxl__domain_build_info_setdefault (libxl__gc *gc,libxl_domain_build_info *b_info) {
@@ -372,6 +379,8 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,libxl_defbool_setdefault (&b_info->u.hvm.gfx_passthru, false);libxl__rdm_setdefault (gc, b_info); + +libxl__sgx_setdefault (gc, b_info);break ;case LIBXL_DOMAIN_TYPE_PV :libxl_defbool_setdefault (&b_info->u.pv.e820_host, false);
tools/libxl/libxl_dom.c +30 -0
@@ -1124,6 +1124,36 @@ int libxl__build_hvm(libxl__gc *gc, uint32_t domid, highmem_end = (1ull <<32 ) + (lowmem_end - mmio_start); lowmem_end = mmio_start; } +#if defined(__i386__) || defined(__x86_64__) +if (info->u.hvm.sgx.epckb) { +/* +* FIXME: +* +* Currently EPC base is put at highmem_end + 8G, which should be +* safe in most cases. +* +* I am not quite sure which is the best way to calcualte EPC base. +* IMO we can either: +* 1) put EPC between lowmem_end to mmio_start, but this brings +* additional logic to handle, ex, lowmem_end may become too small +* if EPC is large (shall we limit domain's EPC size?), and hvmloader +* will try to enlarge MMIO space until lowmem_end, or even relocate +* lowmem -- all those make things complicated, so probably put EPC +* in hole between lowmem_end to mmio_start is not good. +* 2) put EPC after highmem_end, but hvmloader may also relocate MMIO +* resource to the place after highmem_end. Maybe the ideal way is to +* put EPC right after highmem_end, and change hvmloader to detect +* EPC, and put high MMIO resource after EPC. I've done this but I +* found a strange bug that EPT mapping of EPC will be (at least part +* of the mappings) will be removed by whom I still cannot find. +* Currently EPC base is put at highmem_end + 8G, and hvmloader code +* is not changed to handle EPC, but this should be safe for most cases. +*/ + info->u.hvm.sgx.epcbase = highmem_end + (2ULL <<32 ); + } + dom->epc_size = (info->u.hvm.sgx.epckb <<10 ); + dom->epc_base = info->u.hvm.sgx.epcbase; +#endif dom->lowmem_end = lowmem_end; dom->highmem_end = highmem_end; dom->mmio_start = mmio_start;
tools/libxl/libxl_internal.h +2 -0
@@ -1252,6 +1252,8 @@ _hidden int libxl__device_vkb_setdefault(libxl__gc *gc, libxl_device_vkb *vkb); _hidden int libxl__device_pci_setdefault(libxl__gc *gc, libxl_device_pci *pci); _hidden void libxl__rdm_setdefault(libxl__gc *gc,libxl_domain_build_info *b_info); +_hidden void libxl__sgx_setdefault(libxl__gc *gc, +libxl_domain_build_info *b_info); _hidden int libxl__device_p9_setdefault(libxl__gc *gc,libxl_device_p9 *p9);
tools/libxl/libxl_types.idl +6 -0
@@ -450,6 +450,11 @@ libxl_altp2m_mode = Enumeration("altp2m_mode", [ (3, "limited"), ], init_val = "LIBXL_ALTP2M_MODE_DISABLED") +libxl_sgx_buildinfo = Struct("sgx_buildinfo", [ + ("epcbase", uint64), # EPC base address + ("epckb", MemKB), # EPC size in KB + ], dir=DIR_IN) + libxl_domain_build_info = Struct("domain_build_info",[ ("max_vcpus", integer), ("avail_vcpus", libxl_bitmap),
@@ -564,6 +569,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("serial_list", libxl_string_list), ("rdm", libxl_rdm_reserve), ("rdm_mem_boundary_memkb", MemKB), + ("sgx", libxl_sgx_buildinfo), ])), ("pv", Struct(None, [("kernel", string), ("slack_memkb", MemKB),
tools/libxl/libxl_x86.c +12 -0
@@ -530,6 +530,9 @@ int libxl__arch_domain_construct_memmap(libxl__gc *gc,if (dom->acpi_modules[i].length) e820_entries++; +if ( dom->epc_base && dom->epc_size ) + e820_entries++; +if (e820_entries >=E820MAX ) {LOGD (ERROR , domid,"Ooops! Too many entries in the memory map!" ); rc =ERROR_INVAL ;
@@ -570,6 +573,15 @@ int libxl__arch_domain_construct_memmap(libxl__gc *gc, e820[nr].addr = ((uint64_t )1 <<32 ); e820[nr].size = highmem_size; e820[nr].type =E820_RAM ; + nr++; + } + +/* EPC */ +if (dom->epc_base && dom->epc_size) { + e820[nr].addr = dom->epc_base; + e820[nr].size = dom->epc_size; + e820[nr].type =E820_RESERVED ; + nr++; }if (xc_domain_set_memory_map (CTX ->xch, domid, e820, e820_entries) !=0 ) {
tools/xl/xl_parse.c +5 -0
@@ -1182,6 +1182,11 @@ void parse_config_data(const char *config_source,if (!xlu_cfg_get_long (config,"rdm_mem_boundary" , &l,0 )) b_info->u.hvm.rdm_mem_boundary_memkb = l *1024 ; + +if (!xlu_cfg_get_long (config,"epc" , &l,0 )) { +/* Get EPC size. EPC base is calculated by toolstack later. */ + b_info->u.hvm.sgx.epckb = l *1024 ; + }break ;case LIBXL_DOMAIN_TYPE_PV : {
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 14/15 xen: tools: add SGX to applying CPUID policy
In reply to Kai HuangIn libxc, a new structure 'xc_cpuid_policy_build_info_t' is added to carry domain's EPC base and size info from libxl. libxl_cpuid_apply_policy is also changed to take 'libxl_domain_build_info_t' as parameter, where domain's EPC base and size can be got and passed to xc_cpuid_apply_policy. xc_cpuid_apply_policy is extended to support SGX CPUID. If hypervisor doesn't report SGX feature in host type cpufeatureset, then using 'epc' parameter results in domain creation failure as SGX cannot be supported.
--- tools/libxc/include/xenctrl.h | 10 ++++++ tools/libxc/xc_cpuid_x86.c | 68 ++++++++++++++++++++++++++++++++++--- tools/libxl/libxl.h | 3 +- tools/libxl/libxl_cpuid.c | 15 ++++++-- tools/libxl/libxl_dom.c | 6 +++- tools/libxl/libxl_nocpuid.c | 4 ++- tools/ocaml/libs/xc/xenctrl_stubs.c | 11 +++++- tools/python/xen/lowlevel/xc/xc.c | 11 +++++- 8 files changed, 117 insertions(+), 11 deletions(-)
tools/libxc/include/xenctrl.h +10 -0
@@ -1783,6 +1783,15 @@ int xc_domain_debug_control(xc_interface *xch, uint32_t vcpu);#if defined(__i386__) || defined(__x86_64__) +typedef struct xc_cpuid_policy_build_info_sgx { +uint64_t epc_base; +uint64_t epc_size; +}xc_cpuid_policy_build_info_sgx_t ; + +typedef struct xc_cpuid_policy_build_info { +xc_cpuid_policy_build_info_sgx_t sgx; +}xc_cpuid_policy_build_info_t ; + int xc_cpuid_check(xc_interface *xch, const unsigned int *input, const char **config,
@@ -1794,6 +1803,7 @@ int xc_cpuid_set(xc_interface *xch, char **config_transformed);int xc_cpuid_apply_policy (xc_interface *xch,domid_t domid, +xc_cpuid_policy_build_info_t *b_info,uint32_t *featureset,unsigned int nr_features); void xc_cpuid_to_str(const unsigned int *regs,
tools/libxc/xc_cpuid_x86.c +64 -4
141 lines not on this page · raw message
tools/libxl/libxl.h +2 -1
@@ -1949,7 +1949,8 @@ libxl_device_pci *libxl_device_pci_assignable_list(libxl_ctx *ctx, int *num);int libxl_cpuid_parse_config (libxl_cpuid_policy_list *cpuid,const char * str);int libxl_cpuid_parse_config_xend (libxl_cpuid_policy_list *cpuid,const char * str); -void libxl_cpuid_apply_policy (libxl_ctx *ctx,uint32_t domid); +int libxl_cpuid_apply_policy (libxl_ctx *ctx,uint32_t domid, +libxl_domain_build_info *info);void libxl_cpuid_set (libxl_ctx *ctx,uint32_t domid,libxl_cpuid_policy_list cpuid);
tools/libxl/libxl_cpuid.c +13 -2
@@ -332,9 +332,20 @@ int libxl_cpuid_parse_config_xend(libxl_cpuid_policy_list *cpuid,return 0 ; } -void libxl_cpuid_apply_policy (libxl_ctx *ctx,uint32_t domid) +int libxl_cpuid_apply_policy (libxl_ctx *ctx,uint32_t domid, +libxl_domain_build_info *info) { -xc_cpuid_apply_policy (ctx->xch, domid,NULL ,0 ); +xc_cpuid_policy_build_info_t cpuid_binfo; + +memset (&cpuid_binfo,0 ,sizeof (xc_cpuid_policy_build_info_t)); + +/* Currently only Intel SGX needs info when applying CPUID policy */ +if (info->type ==LIBXL_DOMAIN_TYPE_HVM ) { + cpuid_binfo.sgx.epc_base = info->u.hvm.sgx.epcbase; + cpuid_binfo.sgx.epc_size = (info->u.hvm.sgx.epckb <<10 ); + } + +return xc_cpuid_apply_policy (ctx->xch, domid, &cpuid_binfo,NULL ,0 ); } void libxl_cpuid_set(libxl_ctx *ctx, uint32_t domid,
tools/libxl/libxl_dom.c +5 -1
@@ -535,7 +535,11 @@ int libxl__build_post(libxl__gc *gc, uint32_t domid,return ERROR_FAIL ; } -libxl_cpuid_apply_policy (ctx, domid); + rc =libxl_cpuid_apply_policy (ctx, domid, info); +if (rc) { +LOG (ERROR ,"Failed to apply CPUID policy (%d)" , rc); +return ERROR_FAIL ; + }if (info->cpuid !=NULL )libxl_cpuid_set (ctx, domid, info->cpuid);
tools/libxl/libxl_nocpuid.c +3 -1
@@ -34,8 +34,10 @@ int libxl_cpuid_parse_config_xend(libxl_cpuid_policy_list *cpuid,return 0 ; } -void libxl_cpuid_apply_policy (libxl_ctx *ctx,uint32_t domid) +int libxl_cpuid_apply_policy (libxl_ctx *ctx,uint32_t domid, +libxl_domain_build_info *info) { +return 0 ; } void libxl_cpuid_set(libxl_ctx *ctx, uint32_t domid,
tools/ocaml/libs/xc/xenctrl_stubs.c +10 -1
@@ -796,7 +796,16 @@ CAMLprim value stub_xc_domain_cpuid_apply_policy(value xch, value domid)#if defined(__i386__) || defined(__x86_64__)int r; - r =xc_cpuid_apply_policy (_H (xch),_D (domid),NULL ,0 ); +/* +* FIXME: +* +* Don't support passing SGX info to xc_cpuid_apply_policy here. To be +* honest I don't know the purpose of this CAML function, so I don't +* know whether we need to allow *caller* of this function to pass SGX +* info. As EPC base is calculated internally by toolstack so I think +* it is also impossible to pass EPC base from *user*. +*/ + r =xc_cpuid_apply_policy (_H (xch),_D (domid),NULL ,NULL ,0 );if (r <0 )failwith_xc (_H (xch));#else
tools/python/xen/lowlevel/xc/xc.c +10 -1
@@ -742,7 +742,16 @@ static PyObject *pyxc_dom_set_policy_cpuid(XcObject *self,if ( !PyArg_ParseTuple (args,"i" , &domid) )return NULL ; -if (xc_cpuid_apply_policy (self->xc_handle, domid,NULL ,0 ) ) +/* +* FIXME: +* +* Don't support passing SGX info to xc_cpuid_apply_policy here. To be +* honest I don't know the purpose of this python function, so I don't +* know whether we need to allow *caller* of this function to pass SGX +* info. As EPC base is calculated internally by toolstack so I think +* it is also impossible to pass EPC base from *user*. +*/ +if (xc_cpuid_apply_policy (self->xc_handle, domid,NULL ,NULL ,0 ) )return pyxc_error_to_exception (self->xc_handle);Py_INCREF (zero);
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 15/15 xen: tools: expose EPC in ACPI table
In reply to Kai HuangOn physical machine EPC is exposed in ACPI table via "INT0E0C". Although EPC can be discovered by CPUID but Windows driver requires EPC to be exposed in ACPI table as well. This patch exposes EPC in ACPI table.
--- tools/firmware/hvmloader/util.c | 23 +++++++++++++++++++ tools/firmware/hvmloader/util.h | 3 +++ tools/libacpi/build.c | 3 +++ tools/libacpi/dsdt.asl | 49 ++++++++++++++++++++++++++++++++++++++++ tools/libacpi/dsdt_acpi_info.asl | 6 +++-- tools/libacpi/libacpi.h | 1 + tools/libxl/libxl_x86_acpi.c | 3 +++ 7 files changed, 86 insertions(+), 2 deletions(-)
tools/firmware/hvmloader/util.c +23 -0
@@ -330,6 +330,15 @@ cpuid(uint32_t idx, uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx) : "0" (idx) ); } +void cpuid_count (uint32_t idx,uint32_t count,uint32_t *eax, +uint32_t *ebx,uint32_t *ecx,uint32_t *edx) +{ + asmvolatile ( +"cpuid" + :"=a" (*eax),"=b" (*ebx),"=c" (*ecx),"=d" (*edx) + :"0" (idx),"c" (count) ); +} +static const char hex_digits[] ="0123456789abcdef" ;/* Write a two-character hex representation of 'byte' to digits[].
@@ -888,6 +897,18 @@ static uint8_t acpi_lapic_id(unsigned cpu)return LAPIC_ID (cpu); } +static void get_epc_info (struct acpi_config *config) +{ +uint32_t eax, ebx, ecx, edx; + +cpuid_count (0x12 ,0x2 , &eax, &ebx, &ecx, &edx); + + config->epc_base = (((uint64_t )(ebx &0xfffff )) <<32 ) | + (uint64_t )(eax &0xfffff000 ); + config->epc_size = (((uint64_t )(edx &0xfffff )) <<32 ) | + (uint64_t )(ecx &0xfffff000 ); +} +void hvmloader_acpi_build_tables (struct acpi_config *config,unsigned int physical) {
@@ -920,6 +941,8 @@ void hvmloader_acpi_build_tables(struct acpi_config *config, config->pci_hi_len = pci_hi_mem_end - pci_hi_mem_start; } +get_epc_info (config); + s =xenstore_read ("platform/generation-id" ,"0:0" );if ( s ) {
tools/firmware/hvmloader/util.h +3 -0
@@ -112,6 +112,9 @@ int hpet_exists(unsigned long hpet_base);void cpuid (uint32_t idx,uint32_t *eax,uint32_t *ebx,uint32_t *ecx,uint32_t *edx); +void cpuid_count (uint32_t idx,uint32_t count,uint32_t *eax, +uint32_t *ebx,uint32_t *ecx,uint32_t *edx); +/* Read the TSC register. */ static inline uint64_t rdtsc (void ) {
tools/libacpi/build.c +3 -0
@@ -54,6 +54,7 @@ struct acpi_info {uint32_t madt_lapic0_addr;/* 16 - Address of first MADT LAPIC struct */ uint32_t vm_gid_addr;/* 20 - Address of VM generation id buffer */ uint64_t pci_hi_min, pci_hi_len;/* 24, 32 - PCI I/O hole boundaries */ +uint64_t epc_min, epc_len;/* 40, 48 - EPC region */ }; static void set_checksum(
@@ -535,6 +536,8 @@ int acpi_build_tables(struct acpi_ctxt *ctxt, struct acpi_config *config) acpi_info->pci_hi_min = config->pci_hi_start; acpi_info->pci_hi_len = config->pci_hi_len; } + acpi_info->epc_min = config->epc_base; + acpi_info->epc_len = config->epc_size;/* * Fill in high-memory data structures, starting at @buf.
tools/libacpi/dsdt.asl +49 -0
55 lines not on this page · raw message
tools/libacpi/dsdt_acpi_info.asl +4 -2
@@ -5,7 +5,7 @@ * BIOS region must match struct acpi_info in build.c and * be located at ACPI_INFO_PHYSICAL_ADDRESS = 0xFC000000 */ - OperationRegion(BIOS, SystemMemory, 0xFC000000, 40) + OperationRegion(BIOS, SystemMemory, 0xFC000000, 56) Field(BIOS, ByteAcc, NoLock, Preserve) { UAR1, 1, UAR2, 1,
@@ -21,6 +21,8 @@ LMIN, 32, HMIN, 32, LLEN, 32, - HLEN, 32 + HLEN, 32, + EMIN, 64, + ELEN, 64, } }
tools/libacpi/libacpi.h +1 -0
@@ -63,6 +63,7 @@ struct acpi_config {/* PCI I/O hole */ uint32_t pci_start, pci_len;uint64_t pci_hi_start, pci_hi_len; +uint64_t epc_base, epc_size;uint32_t table_flags;uint8_t acpi_revision;
tools/libxl/libxl_x86_acpi.c +3 -0
@@ -156,6 +156,9 @@ static int init_acpi_config(libxl__gc *gc, config->lapic_id = acpi_lapic_id; config->acpi_revision =5 ; + config->epc_base = b_info->u.hvm.sgx.epcbase; + config->epc_size = (b_info->u.hvm.sgx.epckb <<10 ); + rc =0 ; out:return rc;
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
Quoted 3 lines · Kai Huang
Hi all,
This series is RFC Xen SGX virtualization support design and RFC draft patches.
Thankyou very much for this design doc.
Quoted 11 lines
2. SGX Virtualization Design
2.1 High Level Toolstack Changes:
2.1.1 New 'epc' parameter
EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin
alao should be able to get all domain's virtual EPC size.
Quoted 4 lines
For this purpose, a new 'epc = <size>' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted.
How will this interact with multi-package servers? Even though its fine to implement the single-package support first, the design should be extensible to the multi-package case.
First of all, what are the implications of multi-package SGX?
(Somewhere) you mention changes to scheduling. I presume this is because a guest with EPC mappings in EPT must be scheduled on the same package, or ENCLU[EENTER] will fail. I presume also that each package will have separate, unrelated private keys?
I presume there is no sensible way (even on native) for a single logical process to use multiple different enclaves? By extension, does it make sense to try and offer parts of multiple enclaves to a single VM?
Quoted 4 lines
2.1.3 Notify domain's virtual EPC base and size to Xen
Xen needs to know guest's EPC base and size in order to populate EPC pages for it. Toolstack notifies EPC base and size to Xen via
XEN_DOMCTL_set_cpuid.
I am currently in the process of reworking the Xen/Toolstack interface when it comes to CPUID handling. The latest design is available here: https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg00378.html but the end result will be the toolstack expressing its CPUID policy in terms of the architectural layout.
Therefore, I would expect that, however the setting is represented in the configuration file, xl/libxl would configure it with the hypervisor by setting CPUID.0x12[2] with the appropriate base and size.
Quoted 30 lines
2.1.4 Launch Control Support (?)
Xen Launch Control Support is about to support running multiple domains with each running its own LE signed by different owners (if HW allows, explained below). As explained in 1.4 SGX Launch Control, EINIT for LE (Launch Enclave) only succeeds when SHA256(SIGSTRUCT.modulus) matches
IA32_SGXLEPUBKEYHASHn, and EINIT for other enclaves will derive EINITTOKEN key according toIA32_SGXLEPUBKEYHASHn. Therefore, to support this, guest's virtualIA32_SGXLEPUBKEYHASHnmust be updated to phyiscal MSRs before EINIT (which also means the physicalIA32_SGXLEPUBKEYHASHnneed to be *unlocked* in BIOS before booting to OS).For physical machine, it is BIOS's writer's decision that whether BIOS would provide interface for user to specify customerized
IA32_SGXLEPUBKEYHASHn(it is default to digest of Intel's signing key after reset). In reality, OS's SGX driver may require BIOS to make MSRs *unlocked* and actively write the hash value to MSRs in order to run EINIT successfully, as in this case, the driver will not depend on BIOS's capability (whether it allows user to customerizeIA32_SGXLEPUBKEYHASHnvalue).The problem is for Xen, do we need a new parameter, such as 'lehash=<SHA256>' to specify the default value of guset's virtual
IA32_SGXLEPUBKEYHASHn? And do we need a new parameter, such as 'lewr' to specify whether guest's virtual MSRs are locked or not before handling to guest's OS?I tends to not introduce 'lehash', as it seems SGX driver would actively update the MSRs. And new parameter would add additional changes for upper layer software (such as openstack). And 'lewr' is not needed either as Xen can always *unlock* the MSRs to guest.
Quoted 6 lines
Please give comments?
Currently in my RFC patches above two parameters are not implemented. Xen hypervisor will always *unlock* the MSRs. Whether there is 'lehash' parameter or not doesn't impact Xen hypervisor's emulation of
IA32_SGXLEPUBKEYHASHn. See below Xen hypervisor changes for details.
Reading around, am I correct with the following?
1) Some processors have no launch control. There is no restriction on which enclaves can boot.
2) Some Skylake client processors claim to have launch control, but the MSRs are unavailable (is this an erratum?). These are limited to booting enclaves matching the Intel public key.
3) Launch control may be locked by the BIOS. There may be a custom hash, or it might be the Intel default. Xen can't adjust it at all, but can support running any number of VMs with matching enclaves.
4) Launch control may be unlocked by the BIOS. In this case, Xen can context switch a hash per domain, and run all enclaves.
The eventual plans for CPUID and MSR levelling should allow all of these to be expressed in sensible ways, and I don't forsee any issues with supporting all of these scenarios.
Quoted 38 lines
2.2 High Level Xen Hypervisor Changes:
2.2.1 EPC Management (?)
Xen hypervisor needs to detect SGX, discover EPC, and manage EPC before supporting SGX to guest. EPC is detected via SGX CPUID 0x12.0x2. It's possible that there are multiple EPC sections (enumerated via sub-leaves
0x3and so on, until invaid EPC is reported), but this is only true on multiple-socket server machines. For server machines there are additional things also needs to be done, such as NUMA EPC, scheduling, etc. We will support server machine in the future but currently we only support one EPC.EPC is reported as reserved memory (so it is not reported as normal memory). EPC must be managed in 4K pages. CPU hardware uses EPCM to track status of each EPC pages. Xen needs to manage EPC and provide functions to, ie, alloc and free EPC pages for guest.
There are two ways to manage EPC: Manage EPC separately; or Integrate it to existing memory management framework.
It is easy to manage EPC separately, as currently EPC is pretty small (~100MB), and we can even put them in a single list. However it is not flexible, for example, you will have to write new algorithms when EPC becomes larger, ex, GB. And you have to write new code to support NUMA EPC (although this will not come in short time).
Integrating EPC to existing memory management framework seems more reasonable, as in this way we can resume memory management data structures/algorithms, and it will be more flexible to support larger EPC and potentially NUMA EPC. But modifying MM framework has a higher risk to break existing memory management code (potentially more bugs).
In my RFC patches currently we choose to manage EPC separately. A new structure
epc_pageis added to represent a single 4K EPC page. A whole array of structepc_pagewill be allocated during EPC initialization, so that given the other, one of PFN of EPC page and 'structepc_page' can be got by adding offset.
Quoted 3 lines
But maybe integrating EPC to MM framework is more reasonable. Comments?
2.2.2 EPC Virtualization (?)
It looks like managing the EPC is very similar to managing the NVDIMM ranges. We have a (set of) physical address ranges which need 4k ownership granularity to different domains.
I think integrating this into struct page_struct is the better way to go.
Quoted 70 lines
This part is how to populate EPC for guests. We have 3 choices:
- Static Partitioning
- Oversubscription
- BallooningStatic Partitioning means all EPC pages will be allocated and mapped to guest when it is created, and there's no runtime change of page table mappings for EPC pages. Oversubscription means Xen hypervisor supports EPC page swapping between domains, meaning Xen is able to evict EPC page from another domain and assign it to the domain that needs the EPC. With oversubscription, EPC can be assigned to domain on demand, when EPT violation happens. Ballooning is similar to memory ballooning. It is basically "Static Partitioning" + "Balloon driver" in guest.
Static Partitioning is the easiest way in terms of implementation, and there will be no hypervisor overhead (except EPT overhead of course), because in "Static partitioning", there is no EPT violation for EPC, and Xen doesn't need to turn on ENCLS VMEXIT for guest as ENCLS runs perfectly in non-root mode.
Ballooning is "Static Partitioning" + "Balloon driver" in guest. Like "Static Paratitioning", ballooning doesn't need to turn on ENCLS VMEXIT, and doesn't have EPT violation for EPC either. To support ballooning, we need ballooning driver in guest to issue hypercall to give up or reclaim EPC pages. In terms of hypercall, we have two choices: 1) Add new hypercall for EPC ballooning; 2) Using existing XENMEM_{increase/decrease}_reservation with new memory flag, ie,
XENMEMF_epc. I'll discuss more regarding to adding dedicated hypercall or not later.Oversubscription looks nice but it requires more complicated implemetation. Firstly, as explained in 1.3.3 EPC Eviction & Reload, we need to follow specific steps to evict EPC pages, and in order to do that, basically Xen needs to trap ENCLS from guest and keep track of EPC page status and enclave info from all guest. This is because:
- To evict regular EPC page, Xen needs to know SECS location
- Xen needs to know EPC page type: evicting regular EPC and evicting SECS, VA page have different steps.
- Xen needs to know EPC page status: whether the page is blocked or not.Those info can only be got by trapping ENCLS from guest, and parsing its parameters (to identify SECS page, etc). Parsing ENCLS parameters means we need to know which ENCLS leaf is being trapped, and we need to translate guest's virtual address to get physical address in order to locate EPC page. And once ENCLS is trapped, we have to emulate ENCLS in Xen, which means we need to reconstruct ENCLS parameters by remapping all guest's virtual address to Xen's virtual address (
gva->gpa->pa->xen_va), as ENCLS always use *effective address* which is able to be traslated by processor when running ENCLS.--------------------------------------------------------------
| ENCLS |
--------------------------------------------------------------
| /|\
ENCLS VMEXIT| | VMENTRY
| |
\|/ |1) parse ENCLS parameters
2) reconstruct(remap) guest's ENCLS parameters
3) run ENCLS on behalf of guest (and skip ENCLS)
4) on success, update EPC/enclave info, or inject errorAnd Xen needs to maintain each EPC page's status (type, blocked or not, in enclave or not, etc). Xen also needs to maintain all Enclave's info from all guests, in order to find the correct SECS for regular EPC page, and enclave's linear address as well.
So in general, "Static Partitioning" has simplest implementation, but obviously not the best way to use EPC efficiently; "Ballooning" has all pros of Static Partitioning but requies guest balloon driver; "Oversubscription" is best in terms of flexibility but requires complicated hypervisor implemetation.
Quoted 6 lines
We have implemented "Static Partitioning" in RFC patches, but needs your feedback on whether it is enough. If not, which one should we do at next stage -- Ballooning or Oversubscription. IMO Ballooning may be good enough, given fact that currently memory is also "Static Partitioning" + "Ballooning".
Comments?
Definitely go for static partitioning to begin with. This is far simpler to implement.
I can't see a pressing usecase for oversubscription or ballooning. Any datacenter work will be using exclusively static, and I expect static will fine for all (or at least, most) client usecases.
Quoted 10 lines
2.2.3 Populate EPC for Guest
Toolstack notifies Xen about domain's EPC base and size by
XEN_DOMCTL_set_cpuid, so currently Xen populates all EPC pages for guest inXEN_DOMCTL_set_cpuid, particularly, in handlingXEN_DOMCTL_set_cpuidfor CPUID.0x12.0x2. Once Xen checks the values passed from toolstack is valid, Xen will allocate all EPC pages and setup EPT mappings for guest.2.2.4 New Dedicated Hypercall (?)
All this information should (eventually) be available via the appropriate SYSCTL_get_{cpuid,msr}_policy hypercalls. I don't see any need for dedicated hypercalls.
Quoted 18 lines
2.2.9 Guest Suspend & Resume
On hardware, EPC is destroyed when power goes to S3-S5. So Xen will destroy guest's EPC when guest's power goes into S3-S5. Currently Xen is notified by Qemu in terms of S State change via
HVM_PARAM_ACPI_S_STATE, where Xen will destroy EPC if S State is S3-S5.Specifically, Xen will run EREMOVE for guest's each EPC page, as guest may not handle EPC suspend & resume correctly, in which case physically guest's EPC pages may still be valid, so Xen needs to run EREMOVE to make sure all EPC pages are becoming invalid. Otherwise further operation in guest on EPC may fault as it assumes all EPC pages are invalid after guest is resumed.
For SECS page, EREMOVE may fault with
SGX_CHILD_PRESENT, in which case Xen will keep this SECS page into a list, and call EREMOVE for them again after all EPC pages have been called with EREMOVE. This time the EREMOVE on SECS will succeed as all children (regular EPC pages) have already been removed.
Quoted 8 lines
2.2.10 Destroying Domain
Normally Xen just frees all EPC pages for domain when it is destroyed. But Xen will also do EREMOVE on all guest's EPC pages (described in above 2.2.7) before free them, as guest may shutdown unexpected (ex, user kills guest), and in this case, guest's EPC may still be valid.
2.3 Additional Point: Live Migration, Snapshot Support (?)
How big is the EPC? If we are talking MB rather than GB, movement of the EPC could be after the pause, which would add some latency to live migration but should work. I expect that people would prefer to have the flexibility of migration even at the cost of extra latency.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 04/15] xen: mm: add ioremap_cache
In reply to Kai HuangHi,
Quoted 8 lines · Kai Huang
Currently Xen only has non-cacheable version of ioremap. Although EPC is reported as reserved memory in e820 but it can be mapped as cacheable. This patch adds
ioremap_cache(cacheable version of ioremap).Signed-off-by Kai Huang <[email protected]>--- xen/arch/x86/mm.c | 15 +++++++++++++-- xen/include/xen/vmap.h | 1 +
First of all, this is common code and the "REST" maintainers should have been CCed for this include.
But xen/include/xen/vmap.h is common code and going to break ARM. We already have an inline implementation of ioremap_nocache. You should move the definition in x86 specific headers.
Please make sure to at least build test ARM when touching common code.
Cheers,
Quoted 55 lines
2 files changed, 14 insertions(+), 2 deletions(-)xen/arch/x86/mm.c +13
-2@@ -6284,9 +6284,10 @@ void *__init arch_vmap_virt_end(void)return (void *)fix_to_virt (__end_of_fixed_addresses); } -void __iomem *ioremap (paddr_t pa,size_t len) +static void __iomem *__ioremap (paddr_t pa,size_t len,bool_t cache) {mfn_t mfn =_mfn (PFN_DOWN (pa)); +unsigned int flags = cache ?PAGE_HYPERVISOR :PAGE_HYPERVISOR_NOCACHE ;void *va;WARN_ON (page_is_ram_type (mfn_x (mfn),RAM_TYPE_CONVENTIONAL ));@@ -6299,12 +6300,22 @@ void __iomem *ioremap(paddr_t pa, size_t len)unsigned int offs = pa & (PAGE_SIZE -1 );unsigned int nr =PFN_UP (offs + len); - va =__vmap (&mfn, nr,1 ,1 ,PAGE_HYPERVISOR_NOCACHE ,VMAP_DEFAULT ) + offs; + va =__vmap (&mfn, nr,1 ,1 , flags,VMAP_DEFAULT ) + offs; }return (void __force __iomem *)va; } +void __iomem *ioremap (paddr_t pa,size_t len) +{ +return __ioremap (pa, len, false); +} + +void __iomem *ioremap_cache (paddr_t pa,size_t len) +{ +return __ioremap (pa, len, true); +} +int create_perdomain_mapping (struct domain *d,unsigned long va,unsigned int nr,l1_pgentry_t **pl1tab,struct page_info **ppg)xen/include/xen/vmap.h +1
-0@@ -24,6 +24,7 @@ void *vzalloc(size_t size);void vfree (void *va);void __iomem *ioremap (paddr_t ,size_t ); +void __iomem *ioremap_cache (paddr_t ,size_t );static inline void iounmap (void __iomem *va) {
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
Re: [PATCH 04/15] xen: mm: add ioremap_cache
Hi Julien,
Thanks for pointing out. I'll move to x86 specific.
I've cc-ed all maintainers reported by ./scripts/get_maintainer.pl, looks this script doesn't report all maintainers. Sorry. I'll add ARM maintainers next time.
Thanks,
-Kai
Quoted 62 lines · Julien Grall
Hi,
Quoted 8 lines · Kai Huang
Currently Xen only has non-cacheable version of ioremap. Although EPC is reported as reserved memory in e820 but it can be mapped as cacheable. This patch adds
ioremap_cache(cacheable version of ioremap).Signed-off-by Kai Huang <[email protected]>--- xen/arch/x86/mm.c | 15 +++++++++++++-- xen/include/xen/vmap.h | 1 +First of all, this is common code and the "REST" maintainers should have been CCed for this include.
But
xen/include/xen/vmap.his common code and going to break ARM. We already have an inline implementation ofioremap_nocache. You should move the definition in x86 specific headers.Please make sure to at least build test ARM when touching common code.
Cheers,
Quoted 42 lines
2 files changed, 14 insertions(+), 2 deletions(-)xen/arch/x86/mm.c +13
-2@@ -6284,9 +6284,10 @@ void *__init arch_vmap_virt_end(void)return (void *)fix_to_virt (__end_of_fixed_addresses); } -void __iomem *ioremap (paddr_t pa,size_t len) +static void __iomem *__ioremap (paddr_t pa,size_t len,bool_t cache) {mfn_t mfn =_mfn (PFN_DOWN (pa)); +unsigned int flags = cache ?PAGE_HYPERVISOR :PAGE_HYPERVISOR_NOCACHE ;void *va;WARN_ON (page_is_ram_type (mfn_x (mfn),RAM_TYPE_CONVENTIONAL ));@@ -6299,12 +6300,22 @@ void __iomem *ioremap(paddr_t pa, size_t len)unsigned int offs = pa & (PAGE_SIZE -1 );unsigned int nr =PFN_UP (offs + len); - va =__vmap (&mfn, nr,1 ,1 ,PAGE_HYPERVISOR_NOCACHE , VMAP_DEFAULT) + offs; + va =__vmap (&mfn, nr,1 ,1 , flags,VMAP_DEFAULT ) + offs; }return (void __force __iomem *)va; } +void __iomem *ioremap (paddr_t pa,size_t len) +{ +return __ioremap (pa, len, false); +} + +void __iomem *ioremap_cache (paddr_t pa,size_t len) +{ +return __ioremap (pa, len, true); +} +int create_perdomain_mapping (struct domain *d,unsigned long va,unsigned int nr,l1_pgentry_t **pl1tab,struct page_info **ppg)xen/include/xen/vmap.h +0
-1--- a/xen/include/xen/vmap.h
Quoted 9 lines
xen/include/xen/vmap.h void *vzalloc(size_t size); · line 29
Quoted 8 lines
+++
b/xen/include/xen/vmap.hxen/include/xen/vmap.h +1
-0@@ -24,6 +24,7 @@ void *vzalloc(size_t size);void vfree (void *va);void __iomem *ioremap (paddr_t ,size_t ); +void __iomem *ioremap_cache (paddr_t ,size_t );static inline void iounmap (void __iomem *va) {
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 04/15] xen: mm: add ioremap_cache
In reply to Julien GrallQuoted 11 lines
Quoted 1 line
Quoted 1 line
Julien Grall <[email protected]> 07/11/17 10:15 PM >>>
Quoted 8 lines · Kai Huang
Currently Xen only has non-cacheable version of ioremap. Although EPC is reported as reserved memory in e820 but it can be mapped as cacheable. This patch adds
ioremap_cache(cacheable version of ioremap).Signed-off-by Kai Huang <[email protected]>--- xen/arch/x86/mm.c | 15 +++++++++++++-- xen/include/xen/vmap.h | 1 +
Quoted 6 lines
First of all, this is common code and the "REST" maintainers should have been CCed for this include.
But
xen/include/xen/vmap.his common code and going to break ARM. We already have an inline implementation ofioremap_nocache. You should move the definition in x86 specific headers.
Indeed, plus the ARM implementation actually shows how this would better be done: Have a function allowing more than just true/false to be passed in, to eventually also allow having ioremap_wc() and alike as wrappers. As long as it's x86-specific I'd then also suggest calling the new wrapper function ioremap_wb() (as "cache" may also mean WT).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 04/15] xen: mm: add ioremap_cache
In reply to Huang, KaiQuoted 1 line · Kai
Hi Julien,
Hello Kai,
Please avoid top-posting.
Quoted 4 lines
Thanks for pointing out. I'll move to x86 specific.
I've cc-ed all maintainers reported by
./scripts/get_maintainer.pl, looks this script doesn't report all maintainers. Sorry. I'll add ARM maintainers next time.
I would always double check the result of scripts/get_maintainer.pl. I am aware of a bug in scripts/get_maintainers.pl where only maintainer of the specific component (here x86) are listed, even when you touch common code.
In this case, I didn't ask to CC ARM maintainers, but CC "THE REST" group (see MAINTAINERS).
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
Re: [PATCH 08/15] xen: x86: add SGX cpuid handling support.
In reply to Kai HuangQuoted 34 lines · Kai Huang
This patch adds SGX to cpuid handling support. In
init_guest_cpuid, forraw_policyandhost_policy, physical EPC info is reported, but forpv_max_policyandhvm_max_policyEPC is hidden, as for particular domain, it's EPC base and size are from tookstack, and it is meaningless to contain physical EPC info in them. Before domain's EPC base and size are properly configured, guest's SGX cpuid should report invalid EPC, which is also consistent with HW behavior.Currently all EPC pages are fully populated for domain when it is created. Xen gets domain's EPC base and size from toolstack via
XEN_DOMCTL_set_cpuid, so domain's EPC pages are also populated inXEN_DOMCTL_set_cpuid, after receiving valid EPC base and size. Failure to populate EPC (such as there's no enough free EPC pages) results in domain creation failure by makingXEN_DOMCTL_set_cpuidreturn error.Signed-off-by Kai Huang <[email protected]>--- xen/arch/x86/cpuid.c | 87 ++++++++++++++++++++++++++++++++++++++++++++- xen/arch/x86/domctl.c | 47 +++++++++++++++++++++++- xen/include/asm-x86/cpuid.h | 26 +++++++++++++- 3 files changed, 157 insertions(+), 3 deletions(-)xen/arch/x86/cpuid.c +1
-0@@ -9,6 +9,7 @@#include <asm/paging.h> #include <asm/processor.h> #include <asm/xstate.h> +#include <asm/hvm/vmx/sgx.h> const uint32_t known_features[] =INIT_KNOWN_FEATURES ;const uint32_t special_features[] =INIT_SPECIAL_FEATURES ;
xen/arch/x86/cpuid.c static void recalculate_xstate(struct cpuid_policy *p) · line 162
Quoted 4 lines
xen/arch/x86/cpuid.c +1
-0@@ -158,6 +159,44 @@ static void recalculate_xstate(struct cpuid_policy *p) } } +static void recalculate_sgx (struct cpuid_policy *p,bool_t hide_epc)
Across the entire series, please use bool rather than bool_t.
Why do we need this hide_epc parameter? If we aren't providing any epc resource to the guest, the entire sgx union should be zero and the SGX feature bit should be hidden.
xen/arch/x86/cpuid.c static void recalculate_xstate(struct cpuid_policy *p) · line 174
Quoted 12 lines
xen/arch/x86/cpuid.c +12
-0+{ +if ( !p->feat.sgx ) + { +memset (&p->sgx,0 ,sizeof (p->sgx)); +return ; + } + +if ( !p->sgx.sgx1 ) + { +memset (&p->sgx,0 ,sizeof (p->sgx)); +return ; + }
These two clauses can be combined.
Quoted 8 lines
xen/arch/x86/cpuid.c +8
-0+ +/* +* SDM 42.7.2.1 SECS.ATTRIBUTE.XFRM: +* +* Legal value for SECS.ATTRIBUTE.XFRM conform to these requirements: +* - XFRM[1:0] must be set to 0x3; +* - If processor does not support XSAVE, or if the system software has not +* enabled XSAVE, then XFRM[63:2] must be 0.
xen/arch/x86/cpuid.c static void recalculate_xstate(struct cpuid_policy *p) · line 192
Quoted 10 lines
xen/arch/x86/cpuid.c +10
-0+* - If the processor does support XSAVE, XFRM must contain a value that +* would be legal if loaded into XCR0. +*/ + p->sgx.xfrm_low =0x3 ; + p->sgx.xfrm_high =0 ; +if ( p->basic.xsave ) + { + p->sgx.xfrm_low |= p->xstate.xcr0_low; + p->sgx.xfrm_high |= p->xstate.xcr0_high; + }
There is a bug here, but it will disappear with my CPUID work. At the moment, the job of this function is to sanitise values handed by the toolstack, which includes zeroing all the reserved bits. This is because there is currently no way to signal a failure.
When I fix the toolstack interface, the toolstack will propose a new CPUID policy, and Xen will have a function to check it against the architectural requirements. At that point, we will be applying checks, but not modifying the contents.
Quoted 24 lines
xen/arch/x86/cpuid.c +9
-1+ +if ( hide_epc ) + { +memset (&p->sgx.raw[0x2 ],0 ,sizeof (struct cpuid_leaf )); + } +} +/* * Misc adjustments to the policy. Mostly clobbering reserved fields and * duplicating shared fields. Intentionally hidden fields are annotated. @@ -239,7 +278,7 @@ static void __init calculate_raw_policy(void) {switch ( i ) { -case 0x4 :case 0x7 :case 0xd : +case 0x4 :case 0x7 :case 0xd :case 0x12 :/* Multi-invocation leaves. Deferred. */ continue ; }@@ -299,6 +338,19 @@ static void __init calculate_raw_policy(void) } } +if ( p->basic.max_leaf >=SGX_CPUID )
xen/arch/x86/cpuid.c static void __init calculate_raw_policy(void) · line 351
Quoted 10 lines
xen/arch/x86/cpuid.c +10
-0+ { +/* +* For raw policy we just report native CPUID. For EPC on native it's +* possible that we will have multiple EPC sections (meaning subleaf 3, +* 4, ... may also be valid), but as the policy is for guest so we only +* need one EPC section (subleaf 2). +*/ +cpuid_count_leaf (SGX_CPUID ,0 , &p->sgx.raw[0 ]); +cpuid_count_leaf (SGX_CPUID ,0 , &p->sgx.raw[0 ]); +cpuid_count_leaf (SGX_CPUID ,0 , &p->sgx.raw[0 ]);
Copy & paste error? I presume you meant to use leaves 1 and 2 here, rather than leaf 0 each time?
Quoted 41 lines
xen/arch/x86/cpuid.c +16
-0+ } +/* Extended leaves. */ cpuid_leaf (0x80000000 , &p->extd.raw[0 ]); for ( i = 1; i < min(ARRAY_SIZE(p->extd.raw),@@ -324,6 +376,8 @@ static void __init calculate_host_policy(void)cpuid_featureset_to_policy (boot_cpu_data.x86_capability, p);recalculate_xstate (p);recalculate_misc (p); +/* For host policy we report physical EPC */ +recalculate_sgx (p,0 );if ( p->extd.svm ) {@@ -357,6 +411,11 @@ static void __init calculate_pv_max_policy(void)sanitise_featureset (pv_featureset);cpuid_featureset_to_policy (pv_featureset, p);recalculate_xstate (p); +/* +* For PV policy we don't report physical EPC. Actually for PV policy +* currently SGX will be disabled. +*/ +recalculate_sgx (p,1 ); p->extd.raw[0xa ] =EMPTY_LEAF ;/* No SVM for PV guests. */ }@@ -413,6 +472,13 @@ static void __init calculate_hvm_max_policy(void)sanitise_featureset (hvm_featureset);cpuid_featureset_to_policy (hvm_featureset, p);recalculate_xstate (p); +/* +* For HVM policy we don't report physical EPC. Actually cpuid policy +* should report VM's virtual EPC base and size. However VM's virtual +* EPC info will come from toolstack, and only after Xen is notified +* VM's cpuid policy should report invalid EPC. +*/ +recalculate_sgx (p,1 ); }void __init init_guest_cpuid (void )
xen/arch/x86/cpuid.c void recalculate_cpuid_policy(struct domain *d) · line 600
Quoted 7 lines
xen/arch/x86/cpuid.c +4
-0@@ -528,6 +594,12 @@ void recalculate_cpuid_policy(struct domain *d)if ( p->basic.max_leaf <XSTATE_CPUID )__clear_bit (X86_FEATURE_XSAVE , fs); +if ( p->basic.max_leaf <SGX_CPUID ) + { +__clear_bit (X86_FEATURE_SGX , fs); +__clear_bit (X86_FEATURE_SGX_LAUNCH_CONTROL , fs);
Because you filled in the feature dependency graph for SGX_LAUNCH depending on SGX, this second clear bit isn't necessary. Clearing SGX will cause sanitise_featureset() to automatically clear SGX_LAUNCH (and any future feature bits).
Quoted 19 lines
xen/arch/x86/cpuid.c +8
-0+ } +sanitise_featureset (fs);/* Fold host's FDP_EXCP_ONLY and NO_FPU_SEL into guest's view. */ @@ -550,6 +622,12 @@ void recalculate_cpuid_policy(struct domain *d)recalculate_xstate (p);recalculate_misc (p); +/* +* recalculate_cpuid_policy is also called for domain's cpuid policy, +* which is from toolstack via XEN_DOMCTL_set_cpuid, therefore we cannot +* hide domain's virtual EPC from toolstack. +*/ +recalculate_sgx (p,0 );for ( i =0 ; i <ARRAY_SIZE (p->cache.raw); ++i ) {
xen/arch/x86/cpuid.c void guest_cpuid(const struct vcpu *v, uint32_t leaf, · line 728
Quoted 6 lines
xen/arch/x86/cpuid.c +3
-0@@ -645,6 +723,13 @@ void guest_cpuid(const struct vcpu *v, uint32_t leaf, *res = p->xstate.raw[subleaf];break ; +case SGX_CPUID : +if ( !p->feat.sgx ) +return ;
|| subleaf >= ARRAY_SIZE(p->sgx.raw)
Otherwise, a guest CPUID query can walk read off the end of raw[].
Quoted 28 lines
xen/arch/x86/cpuid.c +4
-0+ + *res = p->sgx.raw[subleaf]; +break ; +default : *res = p->basic.raw[leaf];break ;xen/arch/x86/domctl.c +2
-0@@ -53,6 +53,7 @@ static int update_domain_cpuid_info(struct domain *d,struct cpuid_policy *p = d->arch.cpuid;const struct cpuid_leaf leaf = { ctl->eax, ctl->ebx, ctl->ecx, ctl->edx };int old_vendor = p->x86_vendor; +int ret =0 ;/* * Skip update for leaves we don't care about. This avoids the overhead @@ -74,6 +75,7 @@ static int update_domain_cpuid_info(struct domain *d,if ( ctl->input[0 ] ==XSTATE_CPUID && ctl->input[1 ] !=1 )/* Everything else automatically calculated. */ return 0 ; +break ;case 0x40000000 :case 0x40000100 :
xen/arch/x86/domctl.c static int update_domain_cpuid_info(struct domain *d, · line 111
Quoted 6 lines
xen/arch/x86/domctl.c +3
-0@@ -104,6 +106,10 @@ static int update_domain_cpuid_info(struct domain *d, p->xstate.raw[ctl->input[1 ]] = leaf;break ; +case SGX_CPUID : + p->sgx.raw[ctl->input[1 ]] = leaf; +break ;
You also need to modify the higher switch statement so the toolstack can't cause Xen to write beyond the end of .raw[].
Quoted 18 lines
xen/arch/x86/domctl.c +11
-0+default : p->basic.raw[ctl->input[0 ]] = leaf;break ;@@ -255,6 +261,45 @@ static int update_domain_cpuid_info(struct domain *d, }break ; +case 0x12 : + { +uint64_t base_pfn, npages; + +if ( boot_cpu_data.x86_vendor !=X86_VENDOR_INTEL ) +break ; + +if ( ctl->input[1 ] !=2 ) +break ; +
xen/arch/x86/domctl.c static int update_domain_cpuid_info(struct domain *d, · line 283
Quoted 10 lines
xen/arch/x86/domctl.c +10
-0+/* SGX has not enabled */ +if ( !p->feat.sgx || !p->sgx.sgx1 ) +break ; + +/* +* If SGX is enabled in CPUID, then we are expecting valid EPC resource +* in sub-leaf 0x2. Return -EFAULT to notify toolstack that there's +* something wrong. +*/ +if ( !p->sgx.base_valid || !p->sgx.size_valid )
Is there any plausible usecase where only one of these is valid? If not, why are they split?
Quoted 49 lines
xen/arch/x86/domctl.c +20
-1+ { + ret = -EINVAL ; +break ; + } + + base_pfn = (((uint64_t )(p->sgx.base_pfn_high)) <<20 ) | + (uint64_t )p->sgx.base_pfn_low; + npages = (((uint64_t )(p->sgx.npages_high)) <<20 ) | + (uint64_t )p->sgx.npages_low; + +if ( !hvm_epc_populated (d) ) + ret =hvm_populate_epc (d, base_pfn, npages); +else +if ( base_pfn !=to_sgx (d)->epc_base_pfn || + npages !=to_sgx (d)->epc_npages ) + ret = -EINVAL ; + +break ; + }case 0x80000001 :if (is_pv_domain (d) && ((levelling_caps & LCAP_e1cd) == LCAP_e1cd) ) {@@ -299,7 +344,7 @@ static int update_domain_cpuid_info(struct domain *d,break ; } -return 0 ; +return ret; } void arch_get_domain_info(const struct domain *d,xen/include/asm-x86/cpuid.h +2
-1@@ -61,10 +61,11 @@ extern struct cpuidmasks cpuidmask_defaults;/* Whether or not cpuid faulting is available for the current domain. */ DECLARE_PER_CPU (bool, cpuid_faulting_enabled); -#define CPUID_GUEST_NR_BASIC (0xdu + 1) +#define CPUID_GUEST_NR_BASIC (0x12u + 1)#define CPUID_GUEST_NR_FEAT (0u + 1)#define CPUID_GUEST_NR_CACHE (5u + 1)#define CPUID_GUEST_NR_XSTATE (62u + 1) +#define CPUID_GUEST_NR_SGX (0x2u + 1)#define CPUID_GUEST_NR_EXTD_INTEL (0x8u + 1)#define CPUID_GUEST_NR_EXTD_AMD (0x1cu + 1)#define CPUID_GUEST_NR_EXTD MAX(CPUID_GUEST_NR_EXTD_INTEL, \
xen/include/asm-x86/cpuid.h struct cpuid_policy · line 178
Quoted 9 lines
xen/include/asm-x86/cpuid.h +6
-0@@ -169,6 +170,29 @@ struct cpuid_policy } comp[CPUID_GUEST_NR_XSTATE ]; } xstate; +union { +struct cpuid_leaf raw[CPUID_GUEST_NR_SGX ]; + +struct { +/* Subleaf 0. */ + uint32_t sgx1:1, sgx2:1, :30;
Please use bool bitfields for these.
Something like:
bool sgx1:1 sgx2:2;uint32_t :30;
should be fine.
xen/include/asm-x86/cpuid.h struct cpuid_policy · line 180
Quoted 2 lines
xen/include/asm-x86/cpuid.h +2
-0+ uint32_t miscselect, /* c */ :32; + uint32_t maxenclavesize_n64:8, maxenclavesize_64:8, :16;
uint8_t for these please, rather than an 8 bit bitfield.
Can we use clearer names, such as maxsize_legacy and maxsize_long? They will be accessed via p->sgx. anyway, so the "enclave" bit of context is already present.
xen/include/asm-x86/cpuid.h struct cpuid_policy · line 184
Quoted 4 lines
xen/include/asm-x86/cpuid.h +4
-0+ +/* Subleaf 1. */ + uint32_t init:1, debug:1, mode64:1, /*reserve*/:1, provisionkey:1, + einittokenkey:1, :26;
bools as well please here.
xen/include/asm-x86/cpuid.h struct cpuid_policy · line 186
Quoted 2 lines
xen/include/asm-x86/cpuid.h +2
-0+ uint32_t /* reserve */:32; +uint32_t xfrm_low, xfrm_high;
uint64_t xfrm ?
The XSAVE words are apart because they are not adjacent in the architectural layout, but these are.
xen/include/asm-x86/cpuid.h struct cpuid_policy · line 193
Quoted 7 lines
xen/include/asm-x86/cpuid.h +7
-0+ +/* Subleaf 2. */ + uint32_t base_valid:1, :11, base_pfn_low:20; + uint32_t base_pfn_high:20, :12; + uint32_t size_valid:1, :11, npages_low:20; + uint32_t npages_high:20, :12; + };
Are the {base,size}_valid fields correct? The manual says the are 4-bit fields rather than single bit fields.
I would also drop the _pfn from the base names. The fields still need shifting to get a sensible value.
~Andrew
xen/include/asm-x86/cpuid.h struct cpuid_policy · line 198
Quoted 5 lines
xen/include/asm-x86/cpuid.h +2
-0+ } sgx; +/* Extended leaves: 0x800000xx */ union {struct cpuid_leaf raw[CPUID_GUEST_NR_EXTD ];
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 05/15] xen: p2m: new 'p2m_epc' type for EPC mapping
In reply to Kai HuangQuoted 2 lines · Kai Huang
A new '
p2m_epc' type is added for EPC mapping type. Two wrapper functionsset_epc_p2m_entryandclear_epc_p2m_entryare also added for further use.
Other groups in Intel have been looking to reduce the number of p2m types we have, so we can use more hardware defined bits in the EPT pagetable entries.
If we need a new type then we will certainly add one, but it is not clear why this type is needed.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 15/15] xen: tools: expose EPC in ACPI table
In reply to Kai HuangQuoted 3 lines · Kai Huang
On physical machine EPC is exposed in ACPI table via "INT0E0C". Although EPC can be discovered by CPUID but Windows driver requires EPC to be exposed in ACPI table as well. This patch exposes EPC in ACPI table.
:(
tools/libacpi/dsdt.asl DefinitionBlock ("DSDT.aml", "DSDT", 2, "Xen", "HVM", 0) · line 445
Quoted 10 lines
tools/libacpi/dsdt.asl +2
-0@@ -441,6 +441,55 @@ DefinitionBlock ("DSDT.aml", "DSDT", 2, "Xen", "HVM", 0) } } } + + Device (EPC)
Would it not be better to put this into an SSDT, and only expose it to the guest if SGX is advertised?
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 01/15] xen: x86: expose SGX to HVM domain in CPU featureset
In reply to Kai HuangQuoted 11 lines · Kai Huang
Expose SGX in CPU featureset for HVM domain. SGX will not be supported for PV domain, as ENCLS (which SGX driver in guest essentially runs) must run in ring 0, while PV kernel runs in ring 3. Theoretically we can support SGX in PV domain via either emulating #GP caused by ENCLS running in ring 3, or by PV ENCLS but it is really not necessary at this stage. And currently SGX is only exposed to HAP HVM domain (we can add for shadow in the future).
SGX Launch Control is also exposed in CPU featureset for HVM domain. SGX Launch Control depends on SGX.
Signed-off-by Kai Huang <[email protected]>
I think its perfectly reasonable to restrict to HVM guests to start with, although I don't see how shadow vs HAP has any impact at this stage? All that matters is that the EPC pages appear in the guests p2m.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 02/15] xen: vmx: detect ENCLS VMEXIT
In reply to Kai HuangQuoted 5 lines · Kai Huang
If ENCLS VMEXIT is not present then we cannot support SGX virtualization. This patch detects presence of ENCLS VMEXIT. A Xen boot boolean parameter 'sgx' is also added to manually enable/disable SGX.
Signed-off-by Kai Huang <[email protected]>
At a minimum, you also need to modify calculate_hvm_max_policy() to hide SGX if we don't have ENCLS intercept support.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 05/15] xen: p2m: new 'p2m_epc' type for EPC mapping
In reply to Andrew CooperQuoted 10 lines · Andrew Cooper
Quoted 2 lines · Kai Huang
A new '
p2m_epc' type is added for EPC mapping type. Two wrapper functionsset_epc_p2m_entryandclear_epc_p2m_entryare also added for further use.Other groups in Intel have been looking to reduce the number of p2m types we have, so we can use more hardware defined bits in the EPT pagetable entries.
If we need a new type then we will certainly add one, but it is not clear why this type is needed.
Does the hypervisor need to know which pages of a domain’s p2m 1) have valid config set up, but 2) aren’t accessible to itself or any other domain?
-George
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 02/15] xen: vmx: detect ENCLS VMEXIT
In reply to Andrew CooperQuoted 10 lines
Quoted 1 line
Quoted 1 line
Andrew Cooper <[email protected]> 07/12/17 1:12 PM >>>
Quoted 5 lines · Kai Huang
If ENCLS VMEXIT is not present then we cannot support SGX virtualization. This patch detects presence of ENCLS VMEXIT. A Xen boot boolean parameter 'sgx' is also added to manually enable/disable SGX.
Signed-off-by Kai Huang <[email protected]>At a minimum, you also need to modify
calculate_hvm_max_policy()to hide SGX if we don't have ENCLS intercept support.
Additionally I think the command line option should default to off initially and it needs an entry in the command line option doc.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 02/15] xen: vmx: detect ENCLS VMEXIT
Quoted 10 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Andrew Cooper <[email protected]> 07/12/17 1:12 PM >>>
Quoted 5 lines · Kai Huang
If ENCLS VMEXIT is not present then we cannot support SGX virtualization. This patch detects presence of ENCLS VMEXIT. A Xen boot boolean parameter 'sgx' is also added to manually enable/disable SGX.
Signed-off-by Kai Huang <[email protected]>At a minimum, you also need to modify
calculate_hvm_max_policy()to hide SGX if we don't have ENCLS intercept support.
Actually IMO this is not needed, as I added an __initcall(sgx_init) (see patch 0003), where I will call setup_clear_cpu_cap(X86_FEATURE_SGX) if for any reason boot_sgx_cpuidata (which contains common SGX cpuid info shared by all cores) doesn't have valid SGX info. if ENCLS VMEXIT is not present, then detect_sgx won't be called for any core so that X86_FEATURE_SGX will be cleared in boot_cpu_data. As init_guest_cpuid is called after all __initcalls are called, so if ENCLS VMEXIT is not present, or sgx is disabled via boot parameter, then even for host_policy, it won't have SGX.
Of course if we changed the implementation of __initcall(sgx_init), we probably need to explicitly clear SGX here. Anyway clearing SGX here doesn't have any harm, so I am completely fine to do it if you think it is necessary.
Quoted 3 lines
Additionally I think the command line option should default to off initially and it needs an entry in the command line option doc.
Sure. I'll change default to 0 and change the doc as well.
Thanks,
-Kai
Quoted 9 lines
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 04/15] xen: mm: add ioremap_cache
In reply to Jan BeulichQuoted 18 lines · Jan Beulich
Quoted 11 lines
Quoted 1 line
Quoted 1 line
Julien Grall <[email protected]> 07/11/17 10:15 PM >>>
Quoted 8 lines · Kai Huang
Currently Xen only has non-cacheable version of ioremap. Although EPC is reported as reserved memory in e820 but it can be mapped as cacheable. This patch adds
ioremap_cache(cacheable version of ioremap).Signed-off-by Kai Huang <[email protected]>--- xen/arch/x86/mm.c | 15 +++++++++++++-- xen/include/xen/vmap.h | 1 +Quoted 6 lines
First of all, this is common code and the "REST" maintainers should have been CCed for this include.
But
xen/include/xen/vmap.his common code and going to break ARM. We already have an inline implementation ofioremap_nocache. You should move the definition in x86 specific headers.
Quoted 5 lines
Indeed, plus the ARM implementation actually shows how this would better be done: Have a function allowing more than just true/false to be passed in, to eventually also allow having
ioremap_wc()and alike as wrappers. As long as it's x86-specific I'd then also suggest calling the new wrapper functionioremap_wb()(as "cache" may also mean WT).
Hi Jan,
Thanks for comments. I'll do as you suggested.
Thanks,
-Kai
Quoted 9 lines
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 04/15] xen: mm: add ioremap_cache
In reply to Julien GrallQuoted 8 lines · Julien Grall
Quoted 1 line · Kai
Hi Julien,
Hello Kai,
Please avoid top-posting.
Sorry. Will avoid in the future :)
Quoted 9 lines
Quoted 5 lines
Thanks for pointing out. I'll move to x86 specific.
I've cc-ed all maintainers reported by
./scripts/get_maintainer.pl, looks this script doesn't report all maintainers. Sorry. I'll add ARM maintainers next time.I would always double check the result of
scripts/get_maintainer.pl. I am aware of a bug inscripts/get_maintainers.plwhere only maintainer of the specific component (here x86) are listed, even when you touch common code.In this case, I didn't ask to CC ARM maintainers, but CC "THE REST" group (see
MAINTAINERS).
Understood. I'll follow in the future.
Thanks,
-Kai
Quoted 3 lines
Cheers,
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 08/15] xen: x86: add SGX cpuid handling support.
In reply to Andrew CooperQuoted 33 lines · Andrew Cooper
Quoted 32 lines · Kai Huang
This patch adds SGX to cpuid handling support. In
init_guest_cpuid, forraw_policyandhost_policy, physical EPC info is reported, but forpv_max_policyandhvm_max_policyEPC is hidden, as for particular domain, it's EPC base and size are from tookstack, and it is meaningless to contain physical EPC info in them. Before domain's EPC base and size are properly configured, guest's SGX cpuid should report invalid EPC, which is also consistent with HW behavior.Currently all EPC pages are fully populated for domain when it is created. Xen gets domain's EPC base and size from toolstack via
XEN_DOMCTL_set_cpuid, so domain's EPC pages are also populated inXEN_DOMCTL_set_cpuid, after receiving valid EPC base and size. Failure to populate EPC (such as there's no enough free EPC pages) results in domain creation failure by makingXEN_DOMCTL_set_cpuidreturn error.Signed-off-by Kai Huang <[email protected]>--- xen/arch/x86/cpuid.c | 87 ++++++++++++++++++++++++++++++++++++++++++++- xen/arch/x86/domctl.c | 47 +++++++++++++++++++++++- xen/include/asm-x86/cpuid.h | 26 +++++++++++++- 3 files changed, 157 insertions(+), 3 deletions(-)xen/arch/x86/cpuid.c +1
-0@@ -9,6 +9,7 @@#include <asm/paging.h> #include <asm/processor.h> #include <asm/xstate.h> +#include <asm/hvm/vmx/sgx.h> const uint32_t known_features[] =INIT_KNOWN_FEATURES ;
Quoted 8 lines
xen/arch/x86/cpuid.c static void recalculate_xstate(struct · line 162
Quoted 6 lines
xen/arch/x86/cpuid.c +1
-0const uint32_t special_features[] =INIT_SPECIAL_FEATURES ;@@ -158,6 +159,44 @@ static void recalculate_xstate(struct cpuid_policy *p) } } +static void recalculate_sgx (struct cpuid_policy *p,bool_t hide_epc)Across the entire series, please use bool rather than
bool_t.
Hi Andrew,
Thank you very much for comments.
Will do.
Quoted 2 lines
Why do we need this
hide_epcparameter? If we aren't providing any epc resource to the guest, the entire sgx union should be zero and the SGX feature bit should be hidden.
My intention was to hide physical EPC info for pv_max_policy and hvm_max_policy (recalculate_sgx is also called by calculate_pv_max_policy and calculate_hvm_max_policy), as they are for guest and don't need physical EPC info. But keeping physical EPC info in them does no harm so I think we can simply remove hide_epc.
IMO we cannot check whether EPC is valid and zero sgx union in recalculate_sgx, as it is called for each CPUID. For example, it is called for SGX subleaf 0, and 1, and then 2, and when subleaf 0 and 1 are called, the EPC resource is 0 (hasn't been configured).
Quoted 15 lines
xen/arch/x86/cpuid.c static void recalculate_xstate(struct cpuid_policy *p) · line 174
Quoted 13 lines
xen/arch/x86/cpuid.c +12
-0+{ +if ( !p->feat.sgx ) + { +memset (&p->sgx,0 ,sizeof (p->sgx)); +return ; + } + +if ( !p->sgx.sgx1 ) + { +memset (&p->sgx,0 ,sizeof (p->sgx)); +return ; + }These two clauses can be combined.
Will do.
Quoted 23 lines
Quoted 13 lines
xen/arch/x86/cpuid.c +5
-0+ +/* +* SDM 42.7.2.1 SECS.ATTRIBUTE.XFRM: +* +* Legal value for SECS.ATTRIBUTE.XFRM conform to these requirements:
xen/arch/x86/cpuid.c +2
-0+* - XFRM[1:0] must be set to 0x3; +* - If processor does not support XSAVE, or if the system software has not
xen/arch/x86/cpuid.c +2
-0+* enabled XSAVE, then XFRM[63:2] must be 0. +* - If the processor does support XSAVE, XFRM must contain a value that
xen/arch/x86/cpuid.c static void recalculate_xstate(struct cpuid_policy *p) · line 192
Quoted 9 lines
xen/arch/x86/cpuid.c +9
-0+* would be legal if loaded into XCR0. +*/ + p->sgx.xfrm_low =0x3 ; + p->sgx.xfrm_high =0 ; +if ( p->basic.xsave ) + { + p->sgx.xfrm_low |= p->xstate.xcr0_low; + p->sgx.xfrm_high |= p->xstate.xcr0_high; + }
Quoted 3 lines
There is a bug here, but it will disappear with my CPUID work. At the moment, the job of this function is to sanitise values handed by the toolstack, which includes zeroing all the reserved bits. This is because there is currently no way to signal a failure.
When I fix the toolstack interface, the toolstack will propose a new CPUID policy, and Xen will have a function to check it against the architectural requirements. At that point, we will be applying checks, but not modifying the contents.
I think I need to look at your design first and then I should be able to understand your comment. :)
Quoted 33 lines
Quoted 33 lines
xen/arch/x86/cpuid.c +7
-0+ +if ( hide_epc ) + { +memset (&p->sgx.raw[0x2 ],0 ,sizeof (struct cpuid_leaf )); + } +} +/* * Misc adjustments to the policy. Mostly clobbering reserved fields and
* duplicating shared fields. Intentionally hidden fields are
annotated.xen/arch/x86/cpuid.c +5
-1@@ -239,7 +278,7 @@ static void __init calculate_raw_policy(void) {switch ( i ) { -case 0x4 :case 0x7 :case 0xd : +case 0x4 :case 0x7 :case 0xd :case 0x12 :/* Multi-invocation leaves. Deferred. */ continue ; }@@ -299,6 +338,19 @@ static void __init calculate_raw_policy(void) } } +if ( p->basic.max_leaf >=SGX_CPUID ) + { +/* +* For raw policy we just report native CPUID. For EPC on native it's
+ * possible that we will have multiple EPC sections (meaning
subleaf 3,
+ * 4, ... may also be valid), but as the policy is for guest
Quoted 8 lines
xen/arch/x86/cpuid.c static void __init calculate_raw_policy(void) · line 351
Quoted 6 lines
so we only
xen/arch/x86/cpuid.c +5
-0+* need one EPC section (subleaf 2). +*/ +cpuid_count_leaf (SGX_CPUID ,0 , &p->sgx.raw[0 ]); +cpuid_count_leaf (SGX_CPUID ,0 , &p->sgx.raw[0 ]); +cpuid_count_leaf (SGX_CPUID ,0 , &p->sgx.raw[0 ]);Copy & paste error? I presume you meant to use leaves 1 and 2 here, rather than leaf 0 each time?
Oh sorry. Yes indeed I meant zero out subleaf 1 and 2.
Quoted 42 lines
Quoted 42 lines
xen/arch/x86/cpuid.c +6
-0+ } +/* Extended leaves. */ cpuid_leaf (0x80000000 , &p->extd.raw[0 ]); for ( i = 1; i < min(ARRAY_SIZE(p->extd.raw),@@ -324,6 +376,8 @@ static void __init calculate_host_policy(void)cpuid_featureset_to_policy (boot_cpu_data.x86_capability, p);recalculate_xstate (p);recalculate_misc (p); +/* For host policy we report physical EPC */ +recalculate_sgx (p,0 );if ( p->extd.svm ) {@@ -357,6 +411,11 @@ static void __init calculate_pv_max_policy(void)sanitise_featureset (pv_featureset);cpuid_featureset_to_policy (pv_featureset, p);recalculate_xstate (p); +/* +* For PV policy we don't report physical EPC. Actually for PV policy
xen/arch/x86/cpuid.c +5
-0+* currently SGX will be disabled. +*/ +recalculate_sgx (p,1 ); p->extd.raw[0xa ] =EMPTY_LEAF ;/* No SVM for PV guests. */ }@@ -413,6 +472,13 @@ static void __init calculate_hvm_max_policy(void)sanitise_featureset (hvm_featureset);cpuid_featureset_to_policy (hvm_featureset, p);recalculate_xstate (p); +/* +* For HVM policy we don't report physical EPC. Actually cpuid policy
+ * should report VM's virtual EPC base and size. However VM's
virtualxen/arch/x86/cpuid.c +4
-0+* EPC info will come from toolstack, and only after Xen is notified +* VM's cpuid policy should report invalid EPC. +*/ +recalculate_sgx (p,1 ); }void __init init_guest_cpuid (void )@@ -528,6 +594,12 @@ void recalculate_cpuid_policy(struct domain *d)
Quoted 8 lines
xen/arch/x86/cpuid.c void recalculate_cpuid_policy(struct domain *d) · line 600
Quoted 6 lines
xen/arch/x86/cpuid.c +4
-0if ( p->basic.max_leaf <XSTATE_CPUID )__clear_bit (X86_FEATURE_XSAVE , fs); +if ( p->basic.max_leaf <SGX_CPUID ) + { +__clear_bit (X86_FEATURE_SGX , fs); +__clear_bit (X86_FEATURE_SGX_LAUNCH_CONTROL , fs);Because you filled in the feature dependency graph for
SGX_LAUNCHdepending on SGX, this second clear bit isn't necessary. Clearing SGX will causesanitise_featureset()to automatically clearSGX_LAUNCH(and any future feature bits).
Yes you are right. I'll just clear SGX in next version.
Quoted 28 lines
Quoted 19 lines
xen/arch/x86/cpuid.c +5
-0+ } +sanitise_featureset (fs);/* Fold host's FDP_EXCP_ONLY and NO_FPU_SEL into guest's view. */ @@ -550,6 +622,12 @@ void recalculate_cpuid_policy(struct domain *d)recalculate_xstate (p);recalculate_misc (p); +/* +* recalculate_cpuid_policy is also called for domain's cpuid policy, +* which is from toolstack via XEN_DOMCTL_set_cpuid, therefore we cannot
xen/arch/x86/cpuid.c +3
-0+* hide domain's virtual EPC from toolstack. +*/ +recalculate_sgx (p,0 );for ( i =0 ; i <ARRAY_SIZE (p->cache.raw); ++i ) {xen/arch/x86/cpuid.c void guest_cpuid(const struct vcpu *v, uint32_t · line 728
Quoted 6 lines
xen/arch/x86/cpuid.c +3
-0@@ -645,6 +723,13 @@ void guest_cpuid(const struct vcpu *v, uint32_t leaf, *res = p->xstate.raw[subleaf];break ; +case SGX_CPUID : +if ( !p->feat.sgx ) +return ;|| subleaf >=
ARRAY_SIZE(p->sgx.raw)
Quoted 1 line
Otherwise, a guest CPUID query can walk read off the end of raw[].
Oh yes. Thanks for pointing out.
Quoted 30 lines
Quoted 30 lines
xen/arch/x86/cpuid.c +4
-0+ + *res = p->sgx.raw[subleaf]; +break ; +default : *res = p->basic.raw[leaf];break ;xen/arch/x86/domctl.c +1
-0@@ -53,6 +53,7 @@ static int update_domain_cpuid_info(struct domain *d,struct cpuid_policy *p = d->arch.cpuid;const struct cpuid_leaf leaf = { ctl->eax, ctl->ebx, ctl->ecx, ctl->edx };int old_vendor = p->x86_vendor; +int ret =0 ;/* * Skip update for leaves we don't care about. This avoids the overhead
xen/arch/x86/domctl.c +0
-0@@ -74,6 +75,7 @@ static int update_domain_cpuid_info(struct domain *d,if ( ctl->input[0 ] ==XSTATE_CPUID && ctl->input[1 ] !=1 )/* Everything else automatically calculated. */
return 0;xen/arch/x86/domctl.c +1
-0+break ;case 0x40000000 :case 0x40000100 :@@ -104,6 +106,10 @@ static int update_domain_cpuid_info(struct domain
Quoted 8 lines
xen/arch/x86/domctl.c static int update_domain_cpuid_info(struct domain *d, · line 111
Quoted 6 lines
xen/arch/x86/domctl.c +3
-0*d, p->xstate.raw[ctl->input[1 ]] = leaf;break ; +case SGX_CPUID : + p->sgx.raw[ctl->input[1 ]] = leaf; +break ;You also need to modify the higher switch statement so the toolstack can't cause Xen to write beyond the end of .raw[].
Yes I understand your point now. Will do.
Quoted 25 lines
Quoted 25 lines
xen/arch/x86/domctl.c +17
-0+default : p->basic.raw[ctl->input[0 ]] = leaf;break ;@@ -255,6 +261,45 @@ static int update_domain_cpuid_info(struct domain *d, }break ; +case 0x12 : + { +uint64_t base_pfn, npages; + +if ( boot_cpu_data.x86_vendor !=X86_VENDOR_INTEL ) +break ; + +if ( ctl->input[1 ] !=2 ) +break ; + +/* SGX has not enabled */ +if ( !p->feat.sgx || !p->sgx.sgx1 ) +break ; + +/* +* If SGX is enabled in CPUID, then we are expecting valid
Quoted 8 lines
xen/arch/x86/domctl.c static int update_domain_cpuid_info(struct domain *d, · line 283
Quoted 6 lines
EPC resource
+ * in sub-leaf0x2. Return-EFAULTto notify toolstack that
there'sxen/arch/x86/domctl.c +3
-0+* something wrong. +*/ +if ( !p->sgx.base_valid || !p->sgx.size_valid )Is there any plausible usecase where only one of these is valid? If not, why are they split?
You mean why are they split in SDM? I don't think there's any usecase where only one is valid. In reality, either they both valid, or they both invalid, otherwise there's bug in either CPU ucode or BIOS. This is just the definition in SDM.
Quoted 60 lines
Quoted 49 lines
xen/arch/x86/domctl.c +19
-0+ { + ret = -EINVAL ; +break ; + } + + base_pfn = (((uint64_t )(p->sgx.base_pfn_high)) <<20 ) | + (uint64_t )p->sgx.base_pfn_low; + npages = (((uint64_t )(p->sgx.npages_high)) <<20 ) | + (uint64_t )p->sgx.npages_low; + +if ( !hvm_epc_populated (d) ) + ret =hvm_populate_epc (d, base_pfn, npages); +else +if ( base_pfn !=to_sgx (d)->epc_base_pfn || + npages !=to_sgx (d)->epc_npages ) + ret = -EINVAL ; + +break ; + }case 0x80000001 : if ( is_pv_domain(d) && ((levelling_caps & LCAP_e1cd) ==
LCAP_e1cd) ){xen/arch/x86/domctl.c +1
-1@@ -299,7 +344,7 @@ static int update_domain_cpuid_info(struct domain *d,break ; } -return 0 ; +return ret; } void arch_get_domain_info(const struct domain *d,xen/include/asm-x86/cpuid.h +0
-0@@ -61,10 +61,11 @@ extern struct cpuidmasks cpuidmask_defaults;/* Whether or not cpuid faulting is available for the current domain. */
DECLARE_PER_CPU(bool, cpuid_faulting_enabled);xen/include/asm-x86/cpuid.h +2
-1-#define CPUID_GUEST_NR_BASIC (0xdu + 1) +#define CPUID_GUEST_NR_BASIC (0x12u + 1)#define CPUID_GUEST_NR_FEAT (0u + 1)#define CPUID_GUEST_NR_CACHE (5u + 1)#define CPUID_GUEST_NR_XSTATE (62u + 1) +#define CPUID_GUEST_NR_SGX (0x2u + 1)#define CPUID_GUEST_NR_EXTD_INTEL (0x8u + 1)#define CPUID_GUEST_NR_EXTD_AMD (0x1cu + 1)#define CPUID_GUEST_NR_EXTD MAX(CPUID_GUEST_NR_EXTD_INTEL, \xen/include/asm-x86/cpuid.h struct cpuid_policy · line 177
Quoted 8 lines
xen/include/asm-x86/cpuid.h +6
-0@@ -169,6 +170,29 @@ struct cpuid_policy } comp[CPUID_GUEST_NR_XSTATE ]; } xstate; +union { +struct cpuid_leaf raw[CPUID_GUEST_NR_SGX ]; + +struct { +/* Subleaf 0. */ + uint32_t sgx1:1, sgx2:1, :30;Please use bool bitfields for these.
Quoted 7 lines
Something like:
bool sgx1:1 sgx2:2;
uint32_t:30;should be fine.
OK. Will do. Thanks.
Quoted 6 lines
xen/include/asm-x86/cpuid.h struct cpuid_policy · line 180
Quoted 2 lines
xen/include/asm-x86/cpuid.h +2
-0+ uint32_t miscselect, /* c */ :32; + uint32_t maxenclavesize_n64:8, maxenclavesize_64:8, :16;
uint8_tfor these please, rather than an 8 bit bitfield.Can we use clearer names, such as
maxsize_legacyandmaxsize_long? They will be accessed viap->sgx. anyway, so the "enclave" bit of context is already present.
Sure. I'll change to the name you suggested.
Quoted 8 lines
xen/include/asm-x86/cpuid.h struct cpuid_policy · line 184
Quoted 6 lines
xen/include/asm-x86/cpuid.h +4
-0+ +/* Subleaf 1. */ + uint32_t init:1, debug:1, mode64:1, /*reserve*/:1, provisionkey:1, + einittokenkey:1, :26;bools as well please here.
Will do.
Quoted 7 lines
xen/include/asm-x86/cpuid.h struct cpuid_policy · line 186
Quoted 3 lines
xen/include/asm-x86/cpuid.h +2
-0+ uint32_t /* reserve */:32; +uint32_t xfrm_low, xfrm_high;
uint64_txfrm ?The XSAVE words are apart because they are not adjacent in the architectural layout, but these are.
I think the reason I chose xfrm_low and xfrm_high is in recalculate_sgx, I need to reference them:
if ( p->basic.xsave )
{
p->sgx.xfrm_low |= p->xstate.xcr0_low;
p->sgx.xfrm_high |= p->xstate.xcr0_high;
}But I have no problem change to xfrm.
Quoted 10 lines
xen/include/asm-x86/cpuid.h struct cpuid_policy · line 193
Quoted 8 lines
xen/include/asm-x86/cpuid.h +7
-0+ +/* Subleaf 2. */ + uint32_t base_valid:1, :11, base_pfn_low:20; + uint32_t base_pfn_high:20, :12; + uint32_t size_valid:1, :11, npages_low:20; + uint32_t npages_high:20, :12; + };Are the {base,size}_valid fields correct? The manual says the are 4-bit fields rather than single bit fields.
They are 4 bits in SDM but actually currently only bit 1 is valid (other values are reserved). I think for now bool base_valid should be enough. We can extend when new values come out. What's your suggestion?
Quoted 2 lines
I would also drop the
_pfnfrom the base names. The fields still need shifting to get a sensible value.
OK. Will do.
Quoted 10 lines
~Andrew
xen/include/asm-x86/cpuid.h struct cpuid_policy · line 198
Quoted 5 lines
xen/include/asm-x86/cpuid.h +2
-0+ } sgx; +/* Extended leaves: 0x800000xx */ union {struct cpuid_leaf raw[CPUID_GUEST_NR_EXTD ];
Quoted 4 lines
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 05/15] xen: p2m: new 'p2m_epc' type for EPC mapping
In reply to George DunlapQuoted 4 lines · George Dunlap
Quoted 4 lines · Andrew Cooper
On 09/07/17 10:12, Kai Huang wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
A new '
p2m_epc' type is added for EPC mapping type. Two wrapper functionsset_epc_p2m_entryandclear_epc_p2m_entryare also added for further use.Other groups in Intel have been looking to reduce the number of p2m types we have, so we can use more hardware defined bits in the EPT pagetable entries.
If we need a new type then we will certainly add one, but it is not clear why this type is needed.
Does the hypervisor need to know which pages of a domain’s p2m 1) have valid config set up, but 2) aren’t accessible to itself or any other domain?
Hi Andrew, George,
Actually I haven't thought this thoroughly, but my first glance is there's no existing p2m_type that can be reasonably used for EPC. Probably p2m_ram_rw or p2m_mmio_direct are two potential candidates. For EPC, for *static partitioning* Xen hypervisor just needs to setup mappings and then leave it until guest is destroyed. But for p2m_ram_rw and p2m_mmio_direct there are additional logic when Xen learns about the two types. To me adding 'p2m_epc' is more straightforward and safe. Maybe we can change to a more generic name such as 'p2m_ram_encrypted'? But again I am not sure other encryption technology can also be applied to EPC.
Quoted 7 lines
-George
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 15/15] xen: tools: expose EPC in ACPI table
In reply to Andrew CooperQuoted 7 lines · Andrew Cooper
Quoted 3 lines · Kai Huang
On physical machine EPC is exposed in ACPI table via "INT0E0C". Although EPC can be discovered by CPUID but Windows driver requires EPC to be exposed in ACPI table as well. This patch exposes EPC in ACPI table.
:(
Quoted 13 lines
tools/libacpi/dsdt.asl DefinitionBlock ("DSDT.aml", "DSDT", 2, "Xen", · line 446
Quoted 11 lines
tools/libacpi/dsdt.asl +2
-0@@ -441,6 +441,55 @@ DefinitionBlock ("DSDT.aml", "DSDT", 2, "Xen", "HVM", 0) } } } + + Device (EPC)Would it not be better to put this into an SSDT, and only expose it to the guest if SGX is advertised?
You mean to create dedicated ssdt_epc.asl? I thought about this, but I am not quite sure if we can, because new EPC device will need to refer \_SB.EMIN, and \_SB.ELEN, which are in acpi_info, to get EPC base and size. Can we refer acpi_info in dedicated ssdt_epc.asl?
Thanks,
-Kai
Quoted 7 lines
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 08/15] xen: x86: add SGX cpuid handling support.
In reply to Huang, KaiQuoted 6 lines · Kai
Quoted 3 lines · Andrew Cooper
On 09/07/17 10:10, Kai Huang wrote:
Why do we need this
hide_epcparameter? If we aren't providing any epc resource to the guest, the entire sgx union should be zero and the SGX feature bit should be hidden.My intention was to hide physical EPC info for
pv_max_policyandhvm_max_policy(recalculate_sgxis also called bycalculate_pv_max_policyandcalculate_hvm_max_policy), as they are for guest and don't need physical EPC info. But keeping physical EPC info in them does no harm so I think we can simply removehide_epc.
It is my experience that providing half the information is worse than providing none or all of it, because developers are notorious for taking shortcuts when looking for features.
Patch 1 means that a PV guest will never have p->feat.sgx set. Therefore, we will hit the memset() below, and zero the whole of the SGX union.
Quoted 2 lines
IMO we cannot check whether EPC is valid and zero sgx union in
recalculate_sgx, as it is called for each CPUID. For example, it is called for SGX subleaf 0, and 1, and then 2, and when subleaf 0 and 1 are called, the EPC resource is 0 (hasn't been configured).
recalculate_*() only get called when the toolstack makes updates to the policy. It is an unfortunate side effect of the current implementation, but will be going away with my CPUID work.
The intended flow will be this:
At Xen boot:
* Calculates the raw, host and max policies (as we do today)
At domain create:
* Appropriate policy gets copied to make the default domain policy.
* Toolstack gets the whole policy at one with a new DOMCTL_get_cpuid_policy hypercall.
* Toolstack makes all adjustments (locally) that it wants to, based on configuration, etc.
* Toolstack makes a single DOMCTL_set_cpuid_policy hypercall.
* Xen audits the new policy proposed by the toolstack, resulting in a single yes/no decision.
** If not, the toolstack is told to try again. This will likely result in xl asking the user to modify their .cfg file. ** If yes, the proposed policy becomes the actual policy.
This scheme will fix the current problem we have where the toolstack blindly proposes changes (one leaf at a time), and Xen has to zero the bits it doesn't like (because the toolstack has never traditionally checked the return value of the hypercall :( )
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
xen/include/asm-x86/cpuid.h +7
-0+ +/* Subleaf 2. */ + uint32_t base_valid:1, :11, base_pfn_low:20; + uint32_t base_pfn_high:20, :12; + uint32_t size_valid:1, :11, npages_low:20; + uint32_t npages_high:20, :12; + };
Quoted 3 lines
Quoted 1 line
Are the {base,size}_valid fields correct? The manual says the are 4-bit fields rather than single bit fields.
They are 4 bits in SDM but actually currently only bit 1 is valid (other values are reserved). I think for now bool
base_validshould be enough. We can extend when new values come out. What's your suggestion?
Ok. That can work for now.
Quoted 5 lines
Quoted 3 lines
I would also drop the
_pfnfrom the base names. The fields still need shifting to get a sensible value.OK. Will do.
As a further thought, what about uint64_t base:40 and size:40? That would reduce the complexity of calculating the values.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 08/15] xen: x86: add SGX cpuid handling support.
Quoted 22 lines
Quoted 1 line
Quoted 1 line · On 14.07.17 at 09:37, <[email protected]> wrote:
Quoted 9 lines · Kai
Quoted 8 lines · Andrew Cooper
Quoted 6 lines · Kai Huang
xen/include/asm-x86/cpuid.h +6
-0+/* Subleaf 2. */ + uint32_t base_valid:1, :11, base_pfn_low:20; + uint32_t base_pfn_high:20, :12; + uint32_t size_valid:1, :11, npages_low:20; + uint32_t npages_high:20, :12; + };Quoted 6 lines
Quoted 2 lines
Are the {base,size}_valid fields correct? The manual says the are 4-bit fields rather than single bit fields.
They are 4 bits in SDM but actually currently only bit 1 is valid (other values are reserved). I think for now bool
base_validshould be enough. We can extend when new values come out. What's your suggestion?Ok. That can work for now.
Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I would also drop the
_pfnfrom the base names. The fields still need shifting to get a sensible value.OK. Will do.
As a further thought, what about
uint64_tbase:40 and size:40? That would reduce the complexity of calculating the values.
But that may not really be portable. I've just checked the Intel compiler (on Windows, admittedly), and it then starts the base and size fields each on an 8-byte boundary. Hence all other fields would then better also be uint64_t.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 15/15] xen: tools: expose EPC in ACPI table
In reply to Kai Huangtools/firmware/hvmloader/util.c cpuid(uint32_t idx, uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx) · line 333
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 09.07.17 at 10:16, <[email protected]> wrote:
tools/firmware/hvmloader/util.c +1
-0@@ -330,6 +330,15 @@ cpuid(uint32_t idx, uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx) : "0" (idx) ); } +void cpuid_count(uint32_t idx, uint32_t count, uint32_t *eax,
Please name the first two leaf and subleaf.
tools/firmware/hvmloader/util.c static uint8_t acpi_lapic_id(unsigned cpu) · line 907
Quoted 12 lines
tools/firmware/hvmloader/util.c +8
-0@@ -888,6 +897,18 @@ static uint8_t acpi_lapic_id(unsigned cpu)return LAPIC_ID (cpu); } +static void get_epc_info (struct acpi_config *config) +{ +uint32_t eax, ebx, ecx, edx; + +cpuid_count (0x12 ,0x2 , &eax, &ebx, &ecx, &edx); + + config->epc_base = (((uint64_t )(ebx &0xfffff )) <<32 ) | + (uint64_t )(eax &0xfffff000 );
Pointless cast.
tools/firmware/hvmloader/util.c static uint8_t acpi_lapic_id(unsigned cpu) · line 909
Quoted 2 lines
tools/firmware/hvmloader/util.c +2
-0+ config->epc_size = (((uint64_t )(edx &0xfffff )) <<32 ) | + (uint64_t )(ecx &0xfffff000 );
Again.
Quoted 27 lines
tools/libacpi/dsdt.asl +21
-0@@ -441,6 +441,55 @@ DefinitionBlock ("DSDT.aml", "DSDT", 2, "Xen", "HVM", 0) } } } + + Device (EPC) + { + Name (_HID, EisaId ("INT0E0C")) + Name (_STR, Unicode ("Enclave Page Cache 1.5")) + Name (_MLS, Package (0x01) + { + Package (0x02) + { + "en", + Unicode ("Enclave Page Cache 1.5") + } + }) + Name (RBUF, ResourceTemplate () + { + QWordMemory (ResourceConsumer, PosDecode, MinFixed, MaxFixed, + Cacheable, ReadWrite, + 0x0000000000000000, // Granularity + 0x0000000000000000, // Range Minimum + 0x0000000000000000, // Range Maximum + 0x0000000000000000, // Translation Offset
tools/libacpi/dsdt.asl DefinitionBlock ("DSDT.aml", "DSDT", 2, "Xen", "HVM", 0) · line 474
Quoted 10 lines
tools/libacpi/dsdt.asl +10
-0+ 0x0000000000000001, // Length + ,, _Y03, + AddressRangeMemory, TypeStatic) + }) + + Method(_CRS, 0, NotSerialized) // _CRS: Current Resource Settings + { + CreateQwordField (RBUF, \_SB.EPC._Y03._MIN, EMIN) // _MIN: Minimuum Base Address + CreateQwordField (RBUF, \_SB.EPC._Y03._MAX, EMAX) // _MIN: Maximum Base Address + CreateQwordField (RBUF, \_SB.EPC._Y03._LEN, ELEN) // _LEN: Length
Please see the comment in _SB.PCI0._CRS regarding operations on qword fields. Even if we may not formally support the named Windows versions anymore, we should continue to be careful here. You could have noticed this by seeing that ...
tools/libacpi/dsdt.asl line 27
Quoted 9 lines
tools/libacpi/dsdt.asl +3
-1@@ -21,6 +21,8 @@ LMIN, 32, HMIN, 32, LLEN, 32, - HLEN, 32 + HLEN, 32, + EMIN, 64, + ELEN, 64, }
... there have been no 64-bit fields here so far.
tools/libacpi/dsdt.asl static int init_acpi_config(libxl__gc *gc, · line 160
Quoted 6 lines
tools/libacpi/dsdt.asl +2
-0@@ -156,6 +156,9 @@ static int init_acpi_config(libxl__gc *gc, config->lapic_id = acpi_lapic_id; config->acpi_revision = 5; + config->epc_base = b_info->u.hvm.sgx.epcbase; + config->epc_size = (b_info->u.hvm.sgx.epckb << 10);
Pointless parentheses. Plus I guess the field names could do with an underscore separator in the middle - it took me a moment to realize this is a kB value (explaining the shift by 10).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Hi Andrew,
Thank you very much for comments. Sorry for late reply, and please see my reply below.
Quoted 23 lines · Andrew Cooper
Quoted 3 lines · Kai Huang
Hi all,
This series is RFC Xen SGX virtualization support design and RFC draft patches.
Thankyou very much for this design doc.
Quoted 11 lines
2. SGX Virtualization Design
2.1 High Level Toolstack Changes:
2.1.1 New 'epc' parameter
EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin
alao should be able to get all domain's virtual EPC size.Quoted 4 lines
For this purpose, a new 'epc = <size>' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted.
Quoted 5 lines
How will this interact with multi-package servers? Even though its fine to implement the single-package support first, the design should be extensible to the multi-package case.
First of all, what are the implications of multi-package SGX?
(Somewhere) you mention changes to scheduling. I presume this is because a guest with EPC mappings in EPT must be scheduled on the same package, or ENCLU[EENTER] will fail. I presume also that each package will have separate, unrelated private keys?
The ENCLU[EENTE] will continue to work on multi-package server. Actually I was told all ISA existing behavior documented in SDM won't change for server, as otherwise this would be a bad design :)
Unfortunately I was told I cannot talk about MP server SGX a lot now. Basically I can only talk about staff already documented in SDM (sorry :( ). But I guess multiple EPC in CPUID is designed to cover MP server, at lease mainly (we can do reasonable guess).
In terms of the design, I think we can follow XL config file parameters for memory. 'epc' parameter will always specify totol EPC size that the domain has. And we can use existing NUMA related parameters, such as setting cpus='...' to physically pin vcpu to specific pCPUs, so that EPC will be mostly allocated from related node. If that node runs out of EPC, we can decide whether to allocate EPC from other node, or fail to create domain. I know Linux supports NUMA policy which can specify whether to allow allocating memory from other nodes, does Xen has such policy? Sorry I haven't checked this. If Xen has such policy, we need to choose whether to use memory policy, or introduce new policy for EPC.
If we are going to support vNUAM EPC in the future. We can also use similar way to config vNUMA EPC in XL config.
Sorry I mentioned scheduling. I should say *potentially* :). My thinking was as SGX is per-thread, then SGX info reported by different CPU package may be different (ex, whether SGX2 is supported), then we may need scheduler to be aware of SGX. But I think we don't have to consider this now.
What's your comments?
Quoted 2 lines
I presume there is no sensible way (even on native) for a single logical process to use multiple different enclaves? By extension, does it make sense to try and offer parts of multiple enclaves to a single VM?
The native machine allows running multiple enclaves, even signed by multiple authors. SGX only has limit that before launching any other enclave, Launch Enclave (LE) must be launched. LE is the only enclave that doesn't require EINITTOKEN in EINIT. For LE, its signer (SHA256(sigstruct->modulus)) must be equal to the value in IA32_SGXLEPUBKEYHASHn MSRs. LE will generates EINITTOKEN for other enclaves (EINIT for other enclaves requires EINITTOKEN). For other enclaves, there's no such limitation that enclave's signer must match IA32_SGXLEPUBKEYHASHn so the signer can be anybody. But for other enclaves, before running EINIT, the LE's signer (which is equal to IA32_SGXLEPUBKEYHASHn as explained above) needs to be updated to IA32_SGXLEPUBKEYHASHn (MSRs can be changed, for example, when there's multiple LEs running in OS). This is because EINIT needs to perform EINITTOKEN integrity check (EINITTOKEN contains MAC info that calculated by LE, and EINIT needs LE's IA32_SGXLEPUBKEYHASHn to derive the key to verify MAC).
SGX in VM doesn't change those behaviors, so in VM, the enclaves can also be signed by anyone, but Xen needs to emulate IA32_SGXLEPUBKEYHASHn so that when one VM is running, the correct IA32_SGXLEPUBKEYHASHn are already in physical MSRs.
Quoted 9 lines
Quoted 5 lines
2.1.3 Notify domain's virtual EPC base and size to Xen
Xen needs to know guest's EPC base and size in order to populate EPC pages for it. Toolstack notifies EPC base and size to Xen via
XEN_DOMCTL_set_cpuid.I am currently in the process of reworking the Xen/Toolstack interface when it comes to CPUID handling. The latest design is available here: https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg00378.html but the end result will be the toolstack expressing its CPUID policy in terms of the architectural layout.
Therefore, I would expect that, however the setting is represented in the configuration file, xl/libxl would configure it with the hypervisor by setting CPUID.0x12[2] with the appropriate base and size.
I agree. I saw you are planning to introduce new XEN_DOMCTL_get{set}_cpuid_policy, which will allow toolstack to query/set cpuid policy in single hypercall (if I understand correctly), so I think we should definitely use the new hypercalls.
I also saw you are planning to introduce new hypercall to query raw/host/pv_max/hvm_max cpuid policy (not just featureset), so I think 'xl sgxinfo' (or xl info -sgx) can certainly use that to get physical SGX info (EPC info). And 'xl sgxlist' (or xl list -sgx) can use XEN_DOMCTL_get{set}_cpuid_policy to display domain's SGX info (EPC info).
Btw, do you think we need 'xl sgxinfo' and 'xl sgxlist'? If we do, which is better? New 'xl sgxinfo' and 'xl sgxlist', or extending existing 'xl info' and 'xl list' to support SGX, such as 'xl info -sgx' and 'xl list -sgx' above?
Quoted 33 lines
Quoted 33 lines
2.1.4 Launch Control Support (?)
Xen Launch Control Support is about to support running multiple domains with each running its own LE signed by different owners (if HW allows, explained below). As explained in 1.4 SGX Launch Control, EINIT for LE (Launch Enclave) only succeeds when SHA256(SIGSTRUCT.modulus) matches
IA32_SGXLEPUBKEYHASHn, and EINIT for other enclaves will derive EINITTOKEN key according toIA32_SGXLEPUBKEYHASHn. Therefore, to support this, guest's virtualIA32_SGXLEPUBKEYHASHnmust be updated to phyiscal MSRs before EINIT (which also means the physicalIA32_SGXLEPUBKEYHASHnneed to be *unlocked* in BIOS before booting to OS).For physical machine, it is BIOS's writer's decision that whether BIOS would provide interface for user to specify customerized
IA32_SGXLEPUBKEYHASHn(it is default to digest of Intel's signing key after reset). In reality, OS's SGX driver may require BIOS to make MSRs *unlocked* and actively write the hash value to MSRs in order to run EINIT successfully, as in this case, the driver will not depend on BIOS's capability (whether it allows user to customerizeIA32_SGXLEPUBKEYHASHnvalue).The problem is for Xen, do we need a new parameter, such as 'lehash=<SHA256>' to specify the default value of guset's virtual
IA32_SGXLEPUBKEYHASHn? And do we need a new parameter, such as 'lewr' to specify whether guest's virtual MSRs are locked or not before handling to guest's OS?I tends to not introduce 'lehash', as it seems SGX driver would actively update the MSRs. And new parameter would add additional changes for upper layer software (such as openstack). And 'lewr' is not needed either as Xen can always *unlock* the MSRs to guest.
Please give comments?
Quoted 8 lines
Quoted 4 lines
Currently in my RFC patches above two parameters are not implemented. Xen hypervisor will always *unlock* the MSRs. Whether there is 'lehash' parameter or not doesn't impact Xen hypervisor's emulation of
IA32_SGXLEPUBKEYHASHn. See below Xen hypervisor changes for details.Reading around, am I correct with the following?
1) Some processors have no launch control. There is no restriction on which enclaves can boot.
Yes that some processors have no launch control. However it doesn't mean there's no restriction on which enclaves can boot. Contrary, on those machines only Intel's Launch Enclave (LE) can run, as on those machine, IA32_SGXLEPUBKEYHASHn either doesn't exist, or equal to digest of Intel's signing RSA pubkey. However although only Intel's LE can be run, we can still run other enclaves from other signers. Please see my reply above.
Quoted 2 lines
2) Some Skylake client processors claim to have launch control, but the MSRs are unavailable (is this an erratum?). These are limited to booting enclaves matching the Intel public key.
Sorry I don't know whether this is an erratum. I will get back to you after confirming internally.
Quoted 2 lines
3) Launch control may be locked by the BIOS. There may be a custom hash, or it might be the Intel default. Xen can't adjust it at all, but can support running any number of VMs with matching enclaves.
Yes Launch control may be locked by BIOS, although this depends on whether BIOS provides interface for user to configure. I was told that typically BIOS will unlock Launch Control, as SGX driver is expecting such behavior. But I am not sure we can always assume this.
Whether there will be custom hash also depends on BIOS. BIOS may or may not provide interface for user to configure custom hash. So on physical machine, I think we need to consider all the cases. On machine that with Launch control *unlocked*, Xen is able to dynamically change IA32_SGXLEKEYHASHn so that Xen is able to run multiple VM with each running LE from different signer. However if launch control is *locked* in BIOS, then Xen is still able to run multiple VM, but all VM can only run LE from the signer that matches the IA32_SGXLEPUBKEYHASHn (which in most case should be Intel default, but can be custom hash if BIOS allows user to configure).
Sorry I am not quite sure the typical implementation of BIOS. I think I can reach out internally and get back to you if I have something.
Quoted 2 lines
4) Launch control may be unlocked by the BIOS. In this case, Xen can context switch a hash per domain, and run all enclaves.
Yes. With enclave == LE I think you meant.
Quoted 2 lines
The eventual plans for CPUID and MSR levelling should allow all of these to be expressed in sensible ways, and I don't forsee any issues with supporting all of these scenarios.
So do you think we should have 'lehash' and 'lewr' parameters in XL config file? The former provides custom hash, and the latter provides whether unlock guest's Launch control.
My thinking is SGX driver needs to *actively* write LE's pubkey hash to IA32_SGXLEPUBKEYHASHn in *unlocked* mode, so 'lehash' alone is not needed. 'lehash' only has meaning when 'lewr' is needed to provide a default hash value in locked mode, as if we always use *unlocked* mode for guest, 'lehash' is not necessary.
Quoted 41 lines
Quoted 41 lines
2.2 High Level Xen Hypervisor Changes:
2.2.1 EPC Management (?)
Xen hypervisor needs to detect SGX, discover EPC, and manage EPC before supporting SGX to guest. EPC is detected via SGX CPUID 0x12.0x2. It's possible that there are multiple EPC sections (enumerated via sub-leaves
0x3and so on, until invaid EPC is reported), but this is only true on multiple-socket server machines. For server machines there are additional things also needs to be done, such as NUMA EPC, scheduling, etc. We will support server machine in the future but currently we only support one EPC.EPC is reported as reserved memory (so it is not reported as normal memory). EPC must be managed in 4K pages. CPU hardware uses EPCM to track status of each EPC pages. Xen needs to manage EPC and provide functions to, ie, alloc and free EPC pages for guest.
There are two ways to manage EPC: Manage EPC separately; or Integrate it to existing memory management framework.
It is easy to manage EPC separately, as currently EPC is pretty small (~100MB), and we can even put them in a single list. However it is not flexible, for example, you will have to write new algorithms when EPC becomes larger, ex, GB. And you have to write new code to support NUMA EPC (although this will not come in short time).
Integrating EPC to existing memory management framework seems more reasonable, as in this way we can resume memory management data structures/algorithms, and it will be more flexible to support larger EPC and potentially NUMA EPC. But modifying MM framework has a higher risk to break existing memory management code (potentially more bugs).
In my RFC patches currently we choose to manage EPC separately. A new structure
epc_pageis added to represent a single 4K EPC page. A whole array of structepc_pagewill be allocated during EPC initialization, so that given the other, one of PFN of EPC page and 'structepc_page' can be got by adding offset.
Quoted 7 lines
Quoted 3 lines
But maybe integrating EPC to MM framework is more reasonable. Comments?
2.2.2 EPC Virtualization (?)
It looks like managing the EPC is very similar to managing the NVDIMM ranges. We have a (set of) physical address ranges which need 4k ownership granularity to different domains.
I think integrating this into struct
page_structis the better way to go.
Will do. So I assume we will introduce new MEMF_epc, and use existing alloc_domheap/xenheap_pages to allocate EPC? MEMF_epc can also be used if we need to support ballooning in the future (using existing XENMEM_{decrease/increase}_reservation.
Quoted 79 lines
Quoted 79 lines
This part is how to populate EPC for guests. We have 3 choices:
- Static Partitioning
- Oversubscription
- BallooningStatic Partitioning means all EPC pages will be allocated and mapped to guest when it is created, and there's no runtime change of page table mappings for EPC pages. Oversubscription means Xen hypervisor supports EPC page swapping between domains, meaning Xen is able to evict EPC page from another domain and assign it to the domain that needs the EPC. With oversubscription, EPC can be assigned to domain on demand, when EPT violation happens. Ballooning is similar to memory ballooning. It is basically "Static Partitioning" + "Balloon driver" in guest.
Static Partitioning is the easiest way in terms of implementation, and there will be no hypervisor overhead (except EPT overhead of course), because in "Static partitioning", there is no EPT violation for EPC, and Xen doesn't need to turn on ENCLS VMEXIT for guest as ENCLS runs perfectly in non-root mode.
Ballooning is "Static Partitioning" + "Balloon driver" in guest. Like "Static Paratitioning", ballooning doesn't need to turn on ENCLS VMEXIT, and doesn't have EPT violation for EPC either. To support ballooning, we need ballooning driver in guest to issue hypercall to give up or reclaim EPC pages. In terms of hypercall, we have two choices: 1) Add new hypercall for EPC ballooning; 2) Using existing XENMEM_{increase/decrease}_reservation with new memory flag, ie,
XENMEMF_epc. I'll discuss more regarding to adding dedicated hypercall or not later.Oversubscription looks nice but it requires more complicated implemetation. Firstly, as explained in 1.3.3 EPC Eviction & Reload, we need to follow specific steps to evict EPC pages, and in order to do that, basically Xen needs to trap ENCLS from guest and keep track of EPC page status and enclave info from all guest. This is because:
- To evict regular EPC page, Xen needs to know SECS location
- Xen needs to know EPC page type: evicting regular EPC and
evicting SECS,
VA page have different steps.
- Xen needs to know EPC page status: whether the page is blocked
or not.Those info can only be got by trapping ENCLS from guest, and parsing its parameters (to identify SECS page, etc). Parsing ENCLS parameters means we need to know which ENCLS leaf is being trapped, and we need to translate guest's virtual address to get physical address in order to locate EPC page. And once ENCLS is trapped, we have to emulate ENCLS in Xen, which means we need to reconstruct ENCLS parameters by remapping all guest's virtual address to Xen's virtual address (
gva->gpa->pa->xen_va), as ENCLS always use *effective address* which is able to be traslated by processor when running ENCLS.--------------------------------------------------------------
| ENCLS |
--------------------------------------------------------------
| /|\
ENCLS VMEXIT| | VMENTRY
| |
\|/ |1) parse ENCLS parameters
2) reconstruct(remap) guest's ENCLS parameters
3) run ENCLS on behalf of guest (and skip ENCLS)
4) on success, update EPC/enclave info, or inject errorAnd Xen needs to maintain each EPC page's status (type, blocked or not, in enclave or not, etc). Xen also needs to maintain all Enclave's info from all guests, in order to find the correct SECS for regular EPC page, and enclave's linear address as well.
So in general, "Static Partitioning" has simplest implementation, but obviously not the best way to use EPC efficiently; "Ballooning" has all pros of Static Partitioning but requies guest balloon driver; "Oversubscription" is best in terms of flexibility but requires complicated hypervisor implemetation.
We have implemented "Static Partitioning" in RFC patches, but needs your feedback on whether it is enough. If not, which one should we do at next stage -- Ballooning or Oversubscription. IMO Ballooning may be good enough, given fact
that currently memory is also "Static Partitioning" + "Ballooning".
Quoted 5 lines
Quoted 1 line
Comments?
Definitely go for static partitioning to begin with. This is far simpler to implement.
I can't see a pressing usecase for oversubscription or ballooning. Any datacenter work will be using exclusively static, and I expect static will fine for all (or at least, most) client usecases.
Thanks. So for the first stage I will focus on static partitioning.
Quoted 10 lines
Quoted 10 lines
2.2.3 Populate EPC for Guest
Toolstack notifies Xen about domain's EPC base and size by
XEN_DOMCTL_set_cpuid, so currently Xen populates all EPC pages for guest inXEN_DOMCTL_set_cpuid, particularly, in handlingXEN_DOMCTL_set_cpuidfor CPUID.0x12.0x2. Once Xen checks the values passed from toolstack is valid, Xen will allocate all EPC pages and setup EPT mappings for guest.
Quoted 3 lines
Quoted 1 line
2.2.4 New Dedicated Hypercall (?)
All this information should (eventually) be available via the appropriate SYSCTL_get_{cpuid,msr}_policy hypercalls. I don't see any need for dedicated hypercalls.
Yes I agree. Originally I had concern that without dedicated hypercall, it is hard to implement 'xl sgxinfo' and 'xl sgxlist', but according to your new CPUID enhancement plan, the two can be done via the new hypercalls to query Xen's and domain's cpuid policy. See my reply above regarding to "Notify Xen about guest's EPC info".
Quoted 21 lines
Quoted 21 lines
2.2.9 Guest Suspend & Resume
On hardware, EPC is destroyed when power goes to S3-S5. So Xen will destroy guest's EPC when guest's power goes into S3-S5. Currently Xen is notified by Qemu in terms of S State change via
HVM_PARAM_ACPI_S_STATE, where Xen will destroy EPC if S State is S3-S5.Specifically, Xen will run EREMOVE for guest's each EPC page, as guest may not handle EPC suspend & resume correctly, in which case physically guest's EPC pages may still be valid, so Xen needs to run EREMOVE to make sure all EPC pages are becoming invalid. Otherwise further operation in guest on EPC may fault as it assumes all EPC pages are invalid after guest is resumed.
For SECS page, EREMOVE may fault with
SGX_CHILD_PRESENT, in which case Xen will keep this SECS page into a list, and call EREMOVE for them again after all EPC pages have been called with EREMOVE. This time the EREMOVE on SECS will succeed as all children (regular EPC pages) have already been removed.2.2.10 Destroying Domain
Quoted 9 lines
Quoted 6 lines
Normally Xen just frees all EPC pages for domain when it is destroyed. But Xen will also do EREMOVE on all guest's EPC pages (described in above 2.2.7) before free them, as guest may shutdown unexpected (ex, user kills guest), and in this case, guest's EPC may still be valid.
2.3 Additional Point: Live Migration, Snapshot Support (?)
How big is the EPC? If we are talking MB rather than GB, movement of the EPC could be after the pause, which would add some latency to live migration but should work. I expect that people would prefer to have the flexibility of migration even at the cost of extra latency.
The EPC is typically ~100MB at maximum (as I observed). The EPC is typically reserved with EPCM (EPC map, which is invisible to SW) together by BIOS as processor reserved memory (RPM). On real machine, for both our internal develop machines, and some machines that from Dell, HP, Lenovo (that you can buy from market now), BIOS always provides 3 choices in terms RPM: 32M, 64M, and 128M. And with 128M RPM, EPC is slightly less than 100M.
The problem is EPC cannot be moved. I think you were saying moving EPC by evicting EPC out at last stage and copy evicted content to remote, and then reload. However I don't think this will work, as EPC eviction itself needs to use a VA slot (which itslef is EPC), so you can image that the VA slots cannot be moved to remote. Even if they can, they cannot be used to reload EPC in remote, as info in VA slot is bound to platform and cannot be used on remote.
To support live migration, we can only choose to ignore EPC during live migration and let guest SGX driver/user SW stack to handle restoring enclave (which is actually a lot simpler in hypervisor/toolstack's implementation) . Guest SGX driver needs to handle lose EPC anyway, as EPC is destroyed in S3-S5. The only difference is to support live migration, guest SGX driver needs to support *sudden* lose of EPC, which is not HW behavior, and I was told that currently both Windows & Linux SGX driver already support *sudden* lose of EPC, which leaves us a question whether we need to support SGX live migration (and snapshot).
Quoted 6 lines
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 15/15] xen: tools: expose EPC in ACPI table
In reply to Jan BeulichQuoted 10 lines · Jan Beulich
tools/firmware/hvmloader/util.c cpuid(uint32_t idx, uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx) · line 333
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 09.07.17 at 10:16, <[email protected]> wrote:
tools/firmware/hvmloader/util.c +1
-0@@ -330,6 +330,15 @@ cpuid(uint32_t idx, uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx) : "0" (idx) ); } +void cpuid_count(uint32_t idx, uint32_t count, uint32_t *eax,Please name the first two leaf and subleaf.
Sure will do.
Quoted 16 lines
tools/firmware/hvmloader/util.c static uint8_t acpi_lapic_id(unsigned cpu) · line 907
Quoted 13 lines
tools/firmware/hvmloader/util.c +8
-0@@ -888,6 +897,18 @@ static uint8_t acpi_lapic_id(unsigned cpu)return LAPIC_ID (cpu); } +static void get_epc_info (struct acpi_config *config) +{ +uint32_t eax, ebx, ecx, edx; + +cpuid_count (0x12 ,0x2 , &eax, &ebx, &ecx, &edx); + + config->epc_base = (((uint64_t )(ebx &0xfffff )) <<32 ) | + (uint64_t )(eax &0xfffff000 );Pointless cast.
Quoted 4 lines
tools/firmware/hvmloader/util.c static uint8_t acpi_lapic_id(unsigned cpu) · line 909
Quoted 2 lines
tools/firmware/hvmloader/util.c +2
-0+ config->epc_size = (((uint64_t )(edx &0xfffff )) <<32 ) | + (uint64_t )(ecx &0xfffff000 );Again.
Will do.
Quoted 44 lines
Quoted 28 lines
tools/libacpi/dsdt.asl +21
-0@@ -441,6 +441,55 @@ DefinitionBlock ("DSDT.aml", "DSDT", 2, "Xen", "HVM", 0) } } } + + Device (EPC) + { + Name (_HID, EisaId ("INT0E0C")) + Name (_STR, Unicode ("Enclave Page Cache 1.5")) + Name (_MLS, Package (0x01) + { + Package (0x02) + { + "en", + Unicode ("Enclave Page Cache 1.5") + } + }) + Name (RBUF, ResourceTemplate () + { + QWordMemory (ResourceConsumer, PosDecode, MinFixed, MaxFixed, + Cacheable, ReadWrite, + 0x0000000000000000, // Granularity + 0x0000000000000000, // Range Minimum + 0x0000000000000000, // Range Maximum + 0x0000000000000000, // Translation Offsettools/libacpi/dsdt.asl DefinitionBlock ("DSDT.aml", "DSDT", 2, "Xen", "HVM", 0) · line 474
Quoted 10 lines
tools/libacpi/dsdt.asl +10
-0+ 0x0000000000000001, // Length + ,, _Y03, + AddressRangeMemory, TypeStatic) + }) + + Method(_CRS, 0, NotSerialized) // _CRS: Current Resource Settings + { + CreateQwordField (RBUF, \_SB.EPC._Y03._MIN, EMIN) // _MIN: Minimuum Base Address + CreateQwordField (RBUF, \_SB.EPC._Y03._MAX, EMAX) // _MIN: Maximum Base Address + CreateQwordField (RBUF, \_SB.EPC._Y03._LEN, ELEN) // _LEN: LengthPlease see the comment in _SB.PCI0._CRS regarding operations on qword fields. Even if we may not formally support the named Windows versions anymore, we should continue to be careful here. You could have noticed this by seeing that ...
Quoted 11 lines
tools/libacpi/dsdt.asl line 27
Quoted 9 lines
tools/libacpi/dsdt.asl +3
-1@@ -21,6 +21,8 @@ LMIN, 32, HMIN, 32, LLEN, 32, - HLEN, 32 + HLEN, 32, + EMIN, 64, + ELEN, 64, }... there have been no 64-bit fields here so far.
Thank you for pointing this out. I'll take a look.
Quoted 11 lines
tools/libacpi/dsdt.asl static int init_acpi_config(libxl__gc *gc, · line 160
Quoted 7 lines
tools/libacpi/dsdt.asl +2
-0@@ -156,6 +156,9 @@ static int init_acpi_config(libxl__gc *gc, config->lapic_id = acpi_lapic_id; config->acpi_revision = 5; + config->epc_base = b_info->u.hvm.sgx.epcbase; + config->epc_size = (b_info->u.hvm.sgx.epckb << 10);Pointless parentheses. Plus I guess the field names could do with an underscore separator in the middle - it took me a moment to realize this is a kB value (explaining the shift by 10).
Sure. will change to epc_kb and epc_base :)
Thanks,
-Kai
Quoted 9 lines
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 08/15] xen: x86: add SGX cpuid handling support.
In reply to Andrew CooperQuoted 11 lines · Andrew Cooper
Quoted 6 lines · Kai
Quoted 3 lines · Andrew Cooper
On 09/07/17 10:10, Kai Huang wrote:
Why do we need this
hide_epcparameter? If we aren't providing any epc resource to the guest, the entire sgx union should be zero and the SGX feature bit should be hidden.My intention was to hide physical EPC info for
pv_max_policyandhvm_max_policy(recalculate_sgxis also called bycalculate_pv_max_policyandcalculate_hvm_max_policy), as they are for guest and don't need physical EPC info. But keeping physical EPC info in them does no harm so I think we can simply removehide_epc.It is my experience that providing half the information is worse than providing none or all of it, because developers are notorious for taking shortcuts when looking for features.
Patch 1 means that a PV guest will never have
p->feat.sgxset. Therefore, we will hit thememset()below, and zero the whole of the SGX union.
Yes I'll remove hide_epc. It is not absolutely needed.
Quoted 20 lines
Quoted 3 lines
IMO we cannot check whether EPC is valid and zero sgx union in
recalculate_sgx, as it is called for each CPUID. For example, it is called for SGX subleaf 0, and 1, and then 2, and when subleaf 0 and 1 are called, the EPC resource is 0 (hasn't been configured).recalculate_*() only get called when the toolstack makes updates to the policy. It is an unfortunate side effect of the current implementation, but will be going away with my CPUID work.
The intended flow will be this:
At Xen boot:
* Calculates the raw, host and max policies (as we do today)At domain create:
* Appropriate policy gets copied to make the default domain policy.
* Toolstack gets the whole policy at one with a newDOMCTL_get_cpuid_policyhypercall.
* Toolstack makes all adjustments (locally) that it wants to, based on configuration, etc.
* Toolstack makes a singleDOMCTL_set_cpuid_policyhypercall.
* Xen audits the new policy proposed by the toolstack, resulting in a single yes/no decision.
** If not, the toolstack is told to try again. This will likely result in xl asking the user to modify their .cfg file. ** If yes, the proposed policy becomes the actual policy.
Quoted 1 line
This scheme will fix the current problem we have where the toolstack blindly proposes changes (one leaf at a time), and Xen has to zero the bits it doesn't like (because the toolstack has never traditionally checked the return value of the hypercall :( )
This is actually what I was looking for when implementing CPUID support for SGX. I think I'll wait for your work to be merged to Xen and then do my work above your work. :)
Thanks,
-Kai
Quoted 24 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
xen/include/asm-x86/cpuid.h +7
-0+ +/* Subleaf 2. */ + uint32_t base_valid:1, :11, base_pfn_low:20; + uint32_t base_pfn_high:20, :12; + uint32_t size_valid:1, :11, npages_low:20; + uint32_t npages_high:20, :12; + };Quoted 3 lines
Quoted 1 line
Are the {base,size}_valid fields correct? The manual says the are 4-bit fields rather than single bit fields.
They are 4 bits in SDM but actually currently only bit 1 is valid (other values are reserved). I think for now bool
base_validshould be enough. We can extend when new values come out. What's your suggestion?Ok. That can work for now.
Quoted 5 lines
Quoted 3 lines
I would also drop the
_pfnfrom the base names. The fields still need shifting to get a sensible value.OK. Will do.
Quoted 8 lines
As a further thought, what about
uint64_tbase:40 and size:40? That would reduce the complexity of calculating the values.~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 01/15] xen: x86: expose SGX to HVM domain in CPU featureset
In reply to Andrew CooperQuoted 8 lines · Andrew Cooper
Quoted 7 lines · Kai Huang
Expose SGX in CPU featureset for HVM domain. SGX will not be supported for PV domain, as ENCLS (which SGX driver in guest essentially runs) must run in ring 0, while PV kernel runs in ring 3. Theoretically we can support SGX in PV domain via either emulating #GP caused by ENCLS running in ring 3, or by PV ENCLS but it is really not necessary at this stage. And currently SGX is only exposed to HAP HVM domain (we can add for shadow in the future).
Quoted 6 lines
Quoted 4 lines
SGX Launch Control is also exposed in CPU featureset for HVM domain. SGX Launch Control depends on SGX.
Signed-off-by Kai Huang <[email protected]>I think its perfectly reasonable to restrict to HVM guests to start with, although I don't see how shadow vs HAP has any impact at this stage? All that matters is that the EPC pages appear in the guests p2m.
Hmm it seems I forgot replying this one. Sorry. Actually there's no difference between shadow and HAP SGX, as currently SGX functionality is not depending on EPT. I didn't expose SGX to shadow as I haven't got chance to implement and test shadow part. I will add shadow support in next version.
Thanks,
-Kai
Quoted 7 lines
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Hi Kai
Thanks for this nice write-up.
Some comments and questions below.
Quoted 2 lines · Kai Huang
Hi all,
[...]
Quoted 35 lines
2. SGX Virtualization Design
2.1 High Level Toolstack Changes:
2.1.1 New 'epc' parameter
EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin
alao should be able to get all domain's virtual EPC size.For this purpose, a new 'epc = <size>' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted.
2.1.2 New XL commands (?)
Administrator should be able to get physical EPC size, and all domain's virtual EPC size. For this purpose, we can introduce 2 additional commands:
# xl sgxinfoWhich will print out physical EPC size, and other SGX info (such as SGX1, SGX2, etc) if necessary.
# xl sgxlist <did>Which will print out particular domain's virtual EPC size, or list all virtual EPC sizes for all supported domains.
Alternatively, we can also extend existing XL commands by adding new option
# xl info -sgx
Quoted 8 lines
Which will print out physical EPC size along with other physinfo. And
# xl list <did> -sgxWhich will print out domain's virtual EPC size.
Comments?
Can a guest have multiple EPC? If so, the proposed parameter is not good enough.
Can a guest with EPC enabled be migrated? The answer to this question can lead to multiple other questions.
Another question, is EPC going to be backed by normal memory? This is related to memory accounting of the guest.
Is EPC going to be modeled as a device or another type of memory? This is related to how we manage it in the toolstack.
Finally why do you not allow the users to specify the base address?
Quoted 12 lines
In my RFC patches I didn't implement the commands as I don't know which is better. In the github repo I mentioned at the beginning, there's an old branch in which I implemented 'xl sgxinfo' and 'xl sgxlist', but they are implemented via dedicated hypercall for SGX, which I am not sure whether is a good option so I didn't include it in my RFC patches.
2.1.3 Notify domain's virtual EPC base and size to Xen
Xen needs to know guest's EPC base and size in order to populate EPC pages for it. Toolstack notifies EPC base and size to Xen via
XEN_DOMCTL_set_cpuid.2.1.4 Launch Control Support (?)
[...]
Quoted 10 lines
But maybe integrating EPC to MM framework is more reasonable. Comments?
2.2.2 EPC Virtualization (?)
This part is how to populate EPC for guests. We have 3 choices:
- Static Partitioning
- Oversubscription
- Ballooning
IMHO static partitioning is good enough as a starting point.
Ballooning is nice to have but please don't make it mandatory. Not all guests have balloon driver -- imagine a unikernel style secure domain running with EPC.
Quoted 3 lines
2.3 Additional Point: Live Migration, Snapshot Support (?)
Oh, here it is. Nice.
Quoted 59 lines
Actually from hardware's point of view, SGX is not migratable. There are two reasons:
- SGX key architecture cannot be virtualized.
For example, some keys are bound to CPU. For example, Sealing key, EREPORT key, etc. If VM is migrated to another machine, the same enclave will derive the different keys. Taking Sealing key as an example, Sealing key is typically used by enclave (enclave can get sealing key by EGETKEY) to *seal* its secrets to outside (ex, persistent storage) for further use. If Sealing key changes after VM migration, then the enclave can never get the sealed secrets back by using sealing key, as it has changed, and old sealing key cannot be got back.
- There's no ENCLS to evict EPC page to normal memory, but at the meaning time, still keep content in EPC. Currently once EPC page is evicted, the EPC page becomes invalid. So technically, we are unable to implement live migration (or check pointing, or snapshot) for enclave.
But, with some workaround, and some facts of existing SGX driver, technically we are able to support Live migration (or even check pointing, snapshot). This is because:
- Changing key (which is bound to CPU) is not a problem in reality
Take Sealing key as an example. Losing sealed data is not a problem, because sealing key is only supposed to encrypt secrets that can be provisioned again. The typical work model is, enclave gets secrets provisioned from remote (service provider), and use sealing key to store it for further use. When enclave tries to *unseal* use sealing key, if the sealing key is changed, enclave will find the data is some kind of corrupted (integrity check failure), so it will ask secrets to be provisioned again from remote. Another reason is, in data center, VM's typically share lots of data, and as sealing key is bound to CPU, it means the data encrypted by one enclave on one machine cannot be shared by another enclave on another mahcine. So from SGX app writer's point of view, developer should treat Sealing key as a changeable key, and should handle lose of sealing data anyway. Sealing key should only be used to seal secrets that can be easily provisioned again.
For other keys such as EREPORT key and provisioning key, which are used for local attestation and remote attestation, due to the second reason below, losing them is not a problem either.
- Sudden lose of EPC is not a problem.
On hardware, EPC will be lost if system goes to S3-S5, or reset, or shutdown, and SGX driver need to handle lose of EPC due to power transition. This is done by cooperation between SGX driver and userspace SGX SDK/apps. However during live migration, there may not be power transition in guest, so there may not be EPC lose during live migration. And technically we cannot *really* live migrate enclave (explained above), so looks it's not feasible. But the fact is that both Linux SGX driver and Windows SGX driver have already supported *sudden* lose of EPC (not EPC lose during power transition), which means both driver are able to recover in case EPC is lost at any runtime. With this, technically we are able to support live migration by simply ignoring EPC. After VM is migrated, the destination VM will only suffer *sudden* lose of EPC, which both Windows SGX driver and Linux SGX driver are already able to handle.
Quoted 4 lines
But we must point out such *sudden* lose of EPC is not hardware behavior, and other SGX driver for other OSes (such as FreeBSD) may not implement this, so for those guests, destination VM will behavior in unexpected manner. But I am not sure we need to care about other OSes.
Presumably it wouldn't be too hard for FreeBSD to replicate the behaviour of Linux and Windows.
Quoted 4 lines
For the same reason, we are able to support check pointing for SGX guest (only Linux and Windows);
Quoted 9 lines
For snapshot, we can support snapshot SGX guest by either:
- Suspend guest before snapshot (s3-s5). This works for all guests but requires user to manually susppend guest.
- Issue an hypercall to destroy guest's EPC insave_vm. This only works for Linux and Windows but doesn't require user intervention.What's your comments?
IMHO it is of course good to have migration and snapshot support for such guests.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 15/15] xen: tools: expose EPC in ACPI table
In reply to Kai HuangQuoted 8 lines · Kai Huang
On physical machine EPC is exposed in ACPI table via "INT0E0C". Although EPC can be discovered by CPUID but Windows driver requires EPC to be exposed in ACPI table as well. This patch exposes EPC in ACPI table.
Signed-off-by Kai Huang <[email protected]>--- tools/firmware/hvmloader/util.c | 23 +++++++++++++++++++ tools/firmware/hvmloader/util.h | 3 +++
Is there any reason this needs to be done in hvmloader instead of libacpi? I'm mostly asking this because PVH guests can also get ACPI tables, so it would be good to be able to expose EPC to them using ACPI.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Hi Wei,
Thank you very much for comments. Please see my reply below.
Quoted 49 lines · Wei Liu
Hi Kai
Thanks for this nice write-up.
Some comments and questions below.
Quoted 2 lines · Kai Huang
Hi all,
[...]
Quoted 39 lines
2. SGX Virtualization Design
2.1 High Level Toolstack Changes:
2.1.1 New 'epc' parameter
EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin
alao should be able to get all domain's virtual EPC size.For this purpose, a new 'epc = <size>' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted.
2.1.2 New XL commands (?)
Administrator should be able to get physical EPC size, and all domain's virtual EPC size. For this purpose, we can introduce 2 additional commands:
# xl sgxinfoWhich will print out physical EPC size, and other SGX info (such as SGX1, SGX2, etc) if necessary.
# xl sgxlist <did>Which will print out particular domain's virtual EPC size, or list all virtual EPC sizes for all supported domains.
Alternatively, we can also extend existing XL commands by adding new option
# xl info -sgxWhich will print out physical EPC size along with other physinfo. And
# xl list <did> -sgx
Quoted 7 lines
Quoted 4 lines
Which will print out domain's virtual EPC size.
Comments?
Can a guest have multiple EPC? If so, the proposed parameter is not good enough.
According to SDM a machine may have multiple EPC, but it may have doesn't mean it must have. EPC is typically reserved by BIOS as Processor Reserved Memory (PRM), and in my understanding, client machine
doesn't need to have multiple EPC. Currently, I don't see why we need
to expose multiple EPC to guest. Even physical machine reports multiple EPC, exposing one EPC to guest is enough. Currently SGX should not be supported with virtual NUMA simultaneously for a single domain.
Quoted 3 lines
Can a guest with EPC enabled be migrated? The answer to this question can lead to multiple other questions.
See the last section of my design. I saw you've already seen it. :)
Quoted 3 lines
Another question, is EPC going to be backed by normal memory? This is related to memory accounting of the guest.
Although SDM says typically EPC is allocated by BIOS as PRM, but I think we can just treat EPC as PRM, so I believe yes, physically EPC is backed by normal memory. But EPC is reported as reserved memory in e820 table.
Quoted 3 lines
Is EPC going to be modeled as a device or another type of memory? This is related to how we manage it in the toolstack.
I think we'd better to treat EPC as another type of memory. I am not sure whether it should be modeled as device, as on real machine, EPC is also exposed in ACPI table via "INT0E0C" device under \_SB (however it is not modeled as PCIE device for sure).
Quoted 2 lines
Finally why do you not allow the users to specify the base address?
I don't see any reason why user needs to specify base address. If we do, then specify what address? On real machine, BIOS set the base address, and for VM, I think toolstack/Xen should do this.
Quoted 25 lines
Quoted 13 lines
In my RFC patches I didn't implement the commands as I don't know which is better. In the github repo I mentioned at the beginning, there's an old branch in which I implemented 'xl sgxinfo' and 'xl sgxlist', but they are implemented via dedicated hypercall for SGX, which I am not sure whether is a good option so I didn't include it in my RFC patches.
2.1.3 Notify domain's virtual EPC base and size to Xen
Xen needs to know guest's EPC base and size in order to populate EPC pages for it. Toolstack notifies EPC base and size to Xen via
XEN_DOMCTL_set_cpuid.2.1.4 Launch Control Support (?)
[...]
Quoted 10 lines
But maybe integrating EPC to MM framework is more reasonable. Comments?
2.2.2 EPC Virtualization (?)
This part is how to populate EPC for guests. We have 3 choices:
- Static Partitioning
- Oversubscription
- Ballooning
Quoted 5 lines
IMHO static partitioning is good enough as a starting point.
Ballooning is nice to have but please don't make it mandatory. Not all guests have balloon driver -- imagine a unikernel style secure domain running with EPC.
That's good point. Thanks.
Quoted 67 lines
Quoted 5 lines
2.3 Additional Point: Live Migration, Snapshot Support (?)
Oh, here it is. Nice.
Quoted 59 lines
Actually from hardware's point of view, SGX is not migratable. There are two reasons:
- SGX key architecture cannot be virtualized.
For example, some keys are bound to CPU. For example, Sealing key, EREPORT key, etc. If VM is migrated to another machine, the same enclave will derive the different keys. Taking Sealing key as an example, Sealing key is typically used by enclave (enclave can get sealing key by EGETKEY) to *seal* its secrets to outside (ex, persistent storage) for further use. If Sealing key changes after VM migration, then the enclave can never get the sealed secrets back by using sealing key, as it has changed, and old sealing key cannot be got back.
- There's no ENCLS to evict EPC page to normal memory, but at the meaning time, still keep content in EPC. Currently once EPC page is evicted, the EPC page becomes invalid. So technically, we are unable to implement live migration (or check pointing, or snapshot) for enclave.
But, with some workaround, and some facts of existing SGX driver, technically we are able to support Live migration (or even check pointing, snapshot). This is because:
- Changing key (which is bound to CPU) is not a problem in reality
Take Sealing key as an example. Losing sealed data is not a problem, because sealing key is only supposed to encrypt secrets that can be provisioned again. The typical work model is, enclave gets secrets provisioned from remote (service provider), and use sealing key to store it for further use. When enclave tries to *unseal* use sealing key, if the sealing key is changed, enclave will find the data is some kind of corrupted (integrity check failure), so it will ask secrets to be provisioned again from remote. Another reason is, in data center, VM's typically share lots of data, and as sealing key is bound to CPU, it means the data encrypted by one enclave on one machine cannot be shared by another enclave on another mahcine. So from SGX app writer's point of view, developer should treat Sealing key as a changeable key, and should handle lose of sealing data anyway. Sealing key should only be used to seal secrets that can be easily provisioned again.
For other keys such as EREPORT key and provisioning key, which are used for local attestation and remote attestation, due to the second reason below, losing them is not a problem either.
- Sudden lose of EPC is not a problem.
On hardware, EPC will be lost if system goes to S3-S5, or reset, or shutdown, and SGX driver need to handle lose of EPC due to power transition. This is done by cooperation between SGX driver and userspace SGX SDK/apps. However during live migration, there may not be power transition in guest, so there may not be EPC lose during live migration. And technically we cannot *really* live migrate enclave (explained above), so looks it's not feasible. But the fact is that both Linux SGX driver and Windows SGX driver have already supported *sudden* lose of EPC (not EPC lose during power transition), which means both driver are able to recover in case EPC is lost at any runtime. With this, technically we are able to support live migration by simply ignoring EPC. After VM is migrated, the destination VM will only suffer *sudden* lose of EPC, which both Windows SGX driver and Linux SGX driver are already able to handle.
Quoted 7 lines
Quoted 4 lines
But we must point out such *sudden* lose of EPC is not hardware behavior, and other SGX driver for other OSes (such as FreeBSD) may not implement this, so for those guests, destination VM will behavior in unexpected manner. But I am not sure we need to care about other OSes.
Presumably it wouldn't be too hard for FreeBSD to replicate the behaviour of Linux and Windows.
The problem is this is not hardware behavior. If FreeBSD guys just look at the SDM then they may not expect such sudden lose of EPC. But I guess maybe they will just port existing driver. :)
Quoted 12 lines
Quoted 12 lines
For the same reason, we are able to support check pointing for SGX guest (only Linux and Windows);
For snapshot, we can support snapshot SGX guest by either:
- Suspend guest before snapshot (s3-s5). This works for all guests but requires user to manually susppend guest.
- Issue an hypercall to destroy guest's EPC insave_vm. This only works for Linux and Windows but doesn't require user intervention.
Quoted 5 lines
Quoted 2 lines
What's your comments?
IMHO it is of course good to have migration and snapshot support for such guests.
Thanks. I have no problem supporting migration and snapshot if no one opposes.
Thanks,
-Kai
Quoted 6 lines
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 15/15] xen: tools: expose EPC in ACPI table
In reply to Roger Pau MonnéQuoted 10 lines · Roger Pau Monné
Quoted 8 lines · Kai Huang
On physical machine EPC is exposed in ACPI table via "INT0E0C". Although EPC can be discovered by CPUID but Windows driver requires EPC to be exposed in ACPI table as well. This patch exposes EPC in ACPI table.
Signed-off-by Kai Huang <[email protected]>--- tools/firmware/hvmloader/util.c | 23 +++++++++++++++++++ tools/firmware/hvmloader/util.h | 3 +++
Quoted 4 lines
Is there any reason this needs to be done in hvmloader instead of libacpi? I'm mostly asking this because PVH guests can also get ACPI tables, so it would be good to be able to expose EPC to them using ACPI.
Hi Roger,
Thanks for comments. I didn't deliberately choose to do in hvmloader instead of libacpi. It seems libxl only builds ACPI table when guest is HVM, and it doesn't use any device model, and I think I have covered this part (see changes to init_acpi_config). Is there anything that I missed?
Thanks,
-Kai
Quoted 8 lines
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 01/15] xen: x86: expose SGX to HVM domain in CPU featureset
In reply to Kai HuangQuoted 30 lines · Kai Huang
Expose SGX in CPU featureset for HVM domain. SGX will not be supported for PV domain, as ENCLS (which SGX driver in guest essentially runs) must run in ring 0, while PV kernel runs in ring 3. Theoretically we can support SGX in PV domain via either emulating #GP caused by ENCLS running in ring 3, or by PV ENCLS but it is really not necessary at this stage. And currently SGX is only exposed to HAP HVM domain (we can add for shadow in the future).
SGX Launch Control is also exposed in CPU featureset for HVM domain. SGX Launch Control depends on SGX.
Signed-off-by Kai Huang <[email protected]>--- xen/include/public/arch-x86/cpufeatureset.h | 3 ++- xen/tools/gen-cpuid.py | 3 +++ 2 files changed, 5 insertions(+), 1 deletion(-)xen/include/public/arch-x86/cpufeatureset.h +1
-1@@ -193,7 +193,7 @@ XEN_CPUFEATURE(XSAVES, 4*32+ 3) /*S XSAVES/XRSTORS instructions *//* Intel-defined CPU features, CPUID level 0x00000007:0.ebx, word 5 */ XEN_CPUFEATURE(FSGSBASE, 5*32+ 0) /*A {RD,WR}{FS,GS}BASE instructions */ XEN_CPUFEATURE(TSC_ADJUST, 5*32+ 1) /*S TSC_ADJUST MSR available */ -XEN_CPUFEATURE(SGX, 5*32+ 2) /* Software Guard extensions */ +XEN_CPUFEATURE(SGX, 5*32+ 2) /*H Intel Software Guard extensions */ XEN_CPUFEATURE(BMI1, 5*32+ 3) /*A 1st bit manipulation extensions */ XEN_CPUFEATURE(HLE, 5*32+ 4) /*A Hardware Lock Elision */XEN_CPUFEATURE (AVX2 ,5 *32 +5 )/*A AVX2 instructions */
xen/include/public/arch-x86/cpufeatureset.h XEN_CPUFEATURE(PKU, 6*32+ 3) /*H Protection Keys for Userspace */ · line 232
Quoted 4 lines
xen/include/public/arch-x86/cpufeatureset.h +1
-0@@ -229,6 +229,7 @@ XEN_CPUFEATURE(PKU, 6*32+ 3) /*H Protection Keys for Userspace */ XEN_CPUFEATURE(OSPKE, 6*32+ 4) /*! OS Protection Keys Enable */ XEN_CPUFEATURE(AVX512_VPOPCNTDQ, 6*32+14) /*A POPCNT for vectors of DW/QW */ XEN_CPUFEATURE(RDPID, 6*32+22) /*A RDPID instruction */ +XEN_CPUFEATURE (SGX_LAUNCH_CONTROL ,6 *32 +30 )/*H Intel SGX Launch Control */
Could we abbreviate this to SGX_LC ? It is certainly rather shorter to write, and appears to be used elsewhere.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 15/15] xen: tools: expose EPC in ACPI table
In reply to Huang, KaiQuoted 18 lines · Kai
Quoted 10 lines · Roger Pau Monné
Quoted 8 lines · Kai Huang
On physical machine EPC is exposed in ACPI table via "INT0E0C". Although EPC can be discovered by CPUID but Windows driver requires EPC to be exposed in ACPI table as well. This patch exposes EPC in ACPI table.
Signed-off-by Kai Huang <[email protected]>--- tools/firmware/hvmloader/util.c | 23 +++++++++++++++++++ tools/firmware/hvmloader/util.h | 3 +++Quoted 4 lines
Is there any reason this needs to be done in hvmloader instead of libacpi? I'm mostly asking this because PVH guests can also get ACPI tables, so it would be good to be able to expose EPC to them using ACPI.
Quoted 6 lines
Hi Roger,
Thanks for comments. I didn't deliberately choose to do in hvmloader instead of libacpi. It seems libxl only builds ACPI table when guest is HVM, and it doesn't use any device model, and I think I have covered this part (see changes to
init_acpi_config). Is there anything that I missed?
dsdt.asl is only used for HVM guests, PVH guests basically get an empty dsdt + dsdt_acpi_info + processor objects populated by make_dsdt (see Makefile in libacpi), so they end up without the EPC Device block.
It would be good if a new empty dsdt is created, that contains the Device EPC block, or a ssdt is used, and it's added to both HVM/PVH guests.
Alternatively you could also code the EPC Device block in mk_dsdt, but that's going to be cumbersome IMHO.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 01/15] xen: x86: expose SGX to HVM domain in CPU featureset
In reply to Andrew CooperQuoted 30 lines · Andrew Cooper
Quoted 29 lines · Kai Huang
Expose SGX in CPU featureset for HVM domain. SGX will not be supported for PV domain, as ENCLS (which SGX driver in guest essentially runs) must run in ring 0, while PV kernel runs in ring 3. Theoretically we can support SGX in PV domain via either emulating #GP caused by ENCLS running in ring 3, or by PV ENCLS but it is really not necessary at this stage. And currently SGX is only exposed to HAP HVM domain (we can add for shadow in the future).
SGX Launch Control is also exposed in CPU featureset for HVM domain. SGX Launch Control depends on SGX.
Signed-off-by Kai Huang <[email protected]>--- xen/include/public/arch-x86/cpufeatureset.h | 3 ++- xen/tools/gen-cpuid.py | 3 +++ 2 files changed, 5 insertions(+), 1 deletion(-)xen/include/public/arch-x86/cpufeatureset.h +1
-1@@ -193,7 +193,7 @@ XEN_CPUFEATURE(XSAVES, 4*32+ 3) /*S XSAVES/XRSTORS instructions *//* Intel-defined CPU features, CPUID level 0x00000007:0.ebx, word 5 */ XEN_CPUFEATURE(FSGSBASE, 5*32+ 0) /*A {RD,WR}{FS,GS}BASE instructions */ XEN_CPUFEATURE(TSC_ADJUST, 5*32+ 1) /*S TSC_ADJUST MSR available */ -XEN_CPUFEATURE(SGX, 5*32+ 2) /* Software Guard extensions */ +XEN_CPUFEATURE(SGX, 5*32+ 2) /*H Intel Software Guard extensions */ XEN_CPUFEATURE(BMI1, 5*32+ 3) /*A 1st bit manipulation extensions */ XEN_CPUFEATURE(HLE, 5*32+ 4) /*A Hardware Lock Elision */XEN_CPUFEATURE (AVX2 ,5 *32 +5 )/*A AVX2 instructions */
Quoted 8 lines
xen/include/public/arch-x86/cpufeatureset.h XEN_CPUFEATURE(PKU, 6*32+ 3) /*H Protection Keys for Userspace */ · line 232
Quoted 5 lines
xen/include/public/arch-x86/cpufeatureset.h +1
-0@@ -229,6 +229,7 @@ XEN_CPUFEATURE(PKU, 6*32+ 3) /*H Protection Keys for Userspace */ XEN_CPUFEATURE(OSPKE, 6*32+ 4) /*! OS Protection Keys Enable */ XEN_CPUFEATURE(AVX512_VPOPCNTDQ, 6*32+14) /*A POPCNT for vectors of DW/QW */ XEN_CPUFEATURE(RDPID, 6*32+22) /*A RDPID instruction */ +XEN_CPUFEATURE (SGX_LAUNCH_CONTROL ,6 *32 +30 )/*H Intel SGX Launch Control */ Could we abbreviate this to
SGX_LC? It is certainly rather shorter to write, and appears to be used elsewhere.
Sure. Will do.
Thanks,
-Kai
Quoted 8 lines
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 15/15] xen: tools: expose EPC in ACPI table
In reply to Roger Pau MonnéQuoted 31 lines · Roger Pau Monné
Quoted 18 lines · Kai
Quoted 10 lines · Roger Pau Monné
Quoted 8 lines · Kai Huang
On physical machine EPC is exposed in ACPI table via "INT0E0C". Although EPC can be discovered by CPUID but Windows driver requires EPC to be exposed in ACPI table as well. This patch exposes EPC in ACPI table.
Signed-off-by Kai Huang <[email protected]>--- tools/firmware/hvmloader/util.c | 23 +++++++++++++++++++ tools/firmware/hvmloader/util.h | 3 +++Quoted 4 lines
Is there any reason this needs to be done in hvmloader instead of libacpi? I'm mostly asking this because PVH guests can also get ACPI tables, so it would be good to be able to expose EPC to them using ACPI.
Quoted 6 lines
Hi Roger,
Thanks for comments. I didn't deliberately choose to do in hvmloader instead of libacpi. It seems libxl only builds ACPI table when guest is HVM, and it doesn't use any device model, and I think I have covered this part (see changes to
init_acpi_config). Is there anything that I missed?dsdt.asl is only used for HVM guests, PVH guests basically get an empty dsdt +
dsdt_acpi_info+ processor objects populated bymake_dsdt(seeMakefilein libacpi), so they end up without the EPC Device block.
Quoted 6 lines
It would be good if a new empty dsdt is created, that contains the Device EPC block, or a ssdt is used, and it's added to both HVM/PVH guests.
Alternatively you could also code the EPC Device block in
mk_dsdt, but that's going to be cumbersome IMHO.
Hi Roger,
I got your point. I think it's definitely better to cover PVH if we can. Let me see whether it is possible to do.
Thanks,
-Kai
Quoted 3 lines
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 03/15] xen: x86: add early stage SGX feature detection
In reply to Kai HuangQuoted 12 lines · Kai Huang
This patch adds early stage SGX feature detection via SGX CPUID
0x12. Functiondetect_sgxis added to detect SGX info on each CPU (called fromvmx_cpu_up). SDM says SGX info returned by CPUID is per-thread, and we cannot assume all threads will return the same SGX info, so we have to detect SGX for each CPU. For simplicity, currently SGX is only supported when all CPUs reports the same SGX info.SDM also says it's possible to have multiple EPC sections but this is only for multiple-socket server, which we don't support now (there are other things need to be done, ex, NUMA EPC, scheduling, etc, as well), so currently only one EPC is supported.
Quoted 5 lines
Dedicated files
sgx.candsgx.hare added (under vmx directory as SGX is Intel specific) for bulk of above SGX detection code detection code, and for further SGX code as well.Signed-off-by Kai Huang <[email protected]>
I am not sure putting this under hvm/ is a sensible move. Almost everything in this patch is currently common, and I can forsee us wanting to introduce PV support, so it would be good to introduce this in a guest-neutral location to begin with.
xen/arch/x86/hvm/vmx/sgx.c
Quoted 24 lines
--- xen/arch/x86/hvm/vmx/Makefile | 1 + xen/arch/x86/hvm/vmx/sgx.c | 208 ++++++++++++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmcs.c | 4 + xen/include/asm-x86/cpufeature.h | 1 + xen/include/asm-x86/hvm/vmx/sgx.h | 45 +++++++++ 5 files changed, 259 insertions(+) create mode 100644 xen/arch/x86/hvm/vmx/sgx.c create mode 100644 xen/include/asm-x86/hvm/vmx/sgx.hxen/arch/x86/hvm/vmx/Makefile +1
-0@@ -4,3 +4,4 @@ obj-y += realmode.o obj-y += vmcs.o obj-y += vmx.o obj-y += vvmx.o +obj-y += sgx.oxen/arch/x86/hvm/vmx/sgx.c new file +0
-0
This file looks like it should be arch/x86/sgx.c, given its current content.
xen/arch/x86/hvm/vmx/sgx.c line 2
Quoted 3 lines
xen/arch/x86/hvm/vmx/sgx.c new file +2
-0@@ -0,0 +1,208 @@ +/* +* Intel Software Guard Extensions support
Please include a GPLv2 header.
xen/arch/x86/hvm/vmx/sgx.c line 14
Quoted 12 lines
xen/arch/x86/hvm/vmx/sgx.c new file +12
-0+* +* Author: Kai Huang <[email protected]> +*/ + +#include <asm/cpufeature.h> +#include <asm/msr-index.h> +#include <asm/msr.h> +#include <asm/hvm/vmx/sgx.h> +#include <asm/hvm/vmx/vmcs.h> + +static struct sgx_cpuinfo __read_mostly sgx_cpudata[NR_CPUS ]; +static struct sgx_cpuinfo __read_mostly boot_sgx_cpudata;
I don't think any of this is necessary. The description says that all EPCs across the server will be reported in CPUID subleaves, and our implementation gives up if the data are non-identical across CPUs.
Therefore, we only need to keep one copy of the data, and check check APs against the master copy.
Let me see about splitting up a few bits of the existing CPUID infrastructure, so we can use the host cpuid policy more effectively for Xen related things.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 09/15] xen: vmx: handle SGX related MSRs
In reply to Kai HuangQuoted 74 lines · Kai Huang
This patch handles
IA32_FEATURE_CONTROLandIA32_SGXLEPUBKEYHASHnMSRs.For
IA32_FEATURE_CONTROL, if SGX is exposed to domain, thenSGX_ENABLEbit is always set. If SGX launch control is also exposed to domain, and physicalIA32_SGXLEPUBKEYHASHnare writable, thenSGX_LAUNCH_CONTROL_ENABLEbit is also always set. Write toIA32_FEATURE_CONTROLis ignored.For
IA32_SGXLEPUBKEYHASHn, a new 'structsgx_vcpu' is added for per-vcpu SGX staff, and currently it has vcpu's virtual ia32_sgxlepubkeyhash[0-3]. Two boolean 'readable' and 'writable' are also added to indicate whether virtualIA32_SGXLEPUBKEYHASHnare readable and writable.During vcpu is initialized, virtual
ia32_sgxlepubkeyhashare also initialized. If physicalIA32_SGXLEPUBKEYHASHnare writable, thenia32_sgxlepubkeyhashare set to Intel's default value, as for physical machine, those MSRs will have Intel's default value. If physical MSRs are not writable (it is *locked* by BIOS before handling to Xen), then we try to read those MSRs and use physical values as defult value for virtual MSRs. One thing isrdmsr_safeis used, as although SDM says if SGX is present,IA32_SGXLEPUBKEYHASHnare available for read, but in reality, skylake client (at least some, depending on BIOS) doesn't have those MSRs available, so we userdmsr_safeand set readable to false if it returns error code.For
IA32_SGXLEPUBKEYHASHnMSR read from guest, if physical MSRs are not readable, guest is not allowed to read either, otherwise vcpu's virtual MSR value is returned.For
IA32_SGXLEPUBKEYHASHnMSR write from guest, we allow guest to write if both physical MSRs are writable and SGX launch control is exposed to domain, otherwise error is injected.To make EINIT run successfully in guest, vcpu's virtual
IA32_SGXLEPUBKEYHASHnwill be update to physical MSRs when vcpu is scheduled in.Signed-off-by Kai Huang <[email protected]>--- xen/arch/x86/hvm/vmx/sgx.c | 194 +++++++++++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmx.c | 24 +++++ xen/include/asm-x86/cpufeature.h | 3 + xen/include/asm-x86/hvm/vmx/sgx.h | 22 +++++ xen/include/asm-x86/hvm/vmx/vmcs.h | 2 + xen/include/asm-x86/msr-index.h | 6 ++ 6 files changed, 251 insertions(+)xen/arch/x86/hvm/vmx/sgx.c +22
-0@@ -405,6 +405,200 @@ void hvm_destroy_epc(struct domain *d)hvm_reset_epc (d, true); } +/* Whether IA32_SGXLEPUBKEYHASHn are physically *unlocked* by BIOS */ +bool_t sgx_ia32_sgxlepubkeyhash_writable (void ) +{ +uint64_t sgx_lc_enabled =IA32_FEATURE_CONTROL_SGX_ENABLE | +IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE | +IA32_FEATURE_CONTROL_LOCK ; +uint64_t val; + +rdmsrl (MSR_IA32_FEATURE_CONTROL , val); + +return (val & sgx_lc_enabled) == sgx_lc_enabled; +} + +bool_t domain_has_sgx (struct domain *d) +{ +/* hvm_epc_populated(d) implies CPUID has SGX */ +return hvm_epc_populated (d); +} + +bool_t domain_has_sgx_launch_control (struct domain *d) +{ +struct cpuid_policy *p = d->arch.cpuid;
xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 439
Quoted 10 lines
xen/arch/x86/hvm/vmx/sgx.c +10
-0+ +if ( !domain_has_sgx (d) ) +return false; + +/* Unnecessary but check anyway */ +if ( !cpu_has_sgx_launch_control ) +return false; + +return !!p->feat.sgx_launch_control; +}
Both of these should be d->arch.cpuid->feat.{sgx,sgx_lc} only, and not from having individual helpers.
The CPUID setup during host boot and domain construction should take care of setting everything up properly, or hiding the features from the guest. The point of the work I've been doing is to prevent situations where the guest can see SGX but something doesn't work because of Xen using nested checks like this.
Quoted 27 lines
xen/arch/x86/hvm/vmx/sgx.c +27
-0+ +/* Digest of Intel signing key. MSR's default value after reset. */ +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH0 0xa6053e051270b7ac +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH1 0x6cfbe8ba8b3b413d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH2 0xc4916d99f2b3735d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH3 0xd4f8c05909f9bb3b + +void sgx_vcpu_init (struct vcpu *v) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); + +memset (sgxv,0 ,sizeof (*sgxv)); + +if (sgx_ia32_sgxlepubkeyhash_writable () ) + { +/* +* If physical MSRs are writable, set vcpu's default value to Intel's +* default value. For real machine, after reset, MSRs contain Intel's +* default value. +*/ + sgxv->ia32_sgxlepubkeyhash[0 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH0 ; + sgxv->ia32_sgxlepubkeyhash[1 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH1 ; + sgxv->ia32_sgxlepubkeyhash[2 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH2 ; + sgxv->ia32_sgxlepubkeyhash[3 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH3 ; + + sgxv->readable =1 ; + sgxv->writable =domain_has_sgx_launch_control (v->domain);
xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 476
Quoted 10 lines
xen/arch/x86/hvm/vmx/sgx.c +10
-0+ } +else + { +uint64_t v; +/* +* Although SDM says if SGX is present, then IA32_SGXLEPUBKEYHASHn are +* available for read, but in reality for SKYLAKE client machines, +* those MSRs are not available if SGX is present, so we cannot rely on +* cpu_has_sgx to determine whether to we are able to read MSRs, +* instead, we always use rdmsr_safe.
Talking with Jun at XenSummit, I got the impression that the availability of these has MSRs is based on SGX_LC, not SGX.
Furthermore, that is my reading of 41.2.2 "Intel SGX Launch Control Configuration", although the logic is expressed in terms of checking SGX before SGX_LC.
Quoted 8 lines
xen/arch/x86/hvm/vmx/sgx.c +8
-0+*/ + sgxv->readable =rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH0 , v) ?0 :1 ; + +if ( !sgxv->readable ) +return ; + +rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH0 , sgxv->ia32_sgxlepubkeyhash[0 ]); +rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH1 , sgxv->ia32_sgxlepubkeyhash[1 ]);
xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 494
Quoted 10 lines
xen/arch/x86/hvm/vmx/sgx.c +10
-0+rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH2 , sgxv->ia32_sgxlepubkeyhash[2 ]); +rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH3 , sgxv->ia32_sgxlepubkeyhash[3 ]); + } +} + +void sgx_ctxt_switch_to (struct vcpu *v) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); + +if ( sgxv->writable &&sgx_ia32_sgxlepubkeyhash_writable () )
This causes a read of FEATURE_CONTROL on every context switch path, which is inefficient.
Just like with CPUID policy, we will (eventually) have a generic MSR policy for the guest to use. In particular, I can forsee a usecase where hardware has LC unlocked, but the host administrator wishes LC to be locked from the guests point of view.
Quoted 23 lines
xen/arch/x86/hvm/vmx/sgx.c +23
-0+ { +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 , sgxv->ia32_sgxlepubkeyhash[0 ]); +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH1 , sgxv->ia32_sgxlepubkeyhash[1 ]); +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH2 , sgxv->ia32_sgxlepubkeyhash[2 ]); +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH3 , sgxv->ia32_sgxlepubkeyhash[3 ]); + } +} + +int sgx_msr_read_intercept (struct vcpu *v,unsigned int msr,u64 *msr_content) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); +u64 data; +int r =1 ; + +if ( !domain_has_sgx (v->domain) ) +return 0 ; + +switch ( msr ) + { +case MSR_IA32_FEATURE_CONTROL : + data = (IA32_FEATURE_CONTROL_LOCK | +IA32_FEATURE_CONTROL_SGX_ENABLE ); +/*
xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 527
Quoted 10 lines
xen/arch/x86/hvm/vmx/sgx.c +10
-0+* If physical IA32_SGXLEPUBKEYHASHn are writable, then we always +* allow guest to be able to change IA32_SGXLEPUBKEYHASHn at runtime. +*/ +if (sgx_ia32_sgxlepubkeyhash_writable () && +domain_has_sgx_launch_control (v->domain) ) + data |=IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE ; + + *msr_content = data; + +break ;
Newline here please.
xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 528
Quoted 1 line
xen/arch/x86/hvm/vmx/sgx.c +1
-0+ case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3:
Spaces around ... please. (it is only because of the #defines that this isn't a syntax error).
Quoted 31 lines
xen/arch/x86/hvm/vmx/sgx.c +31
-0+/* +* SDM 35.1 Model-Specific Registers, table 35-2. +* +* IA32_SGXLEPUBKEYHASH[0..3]: +* +* Read permitted if CPUID.0x12.0:EAX[0] = 1. +* +* In reality, MSRs may not be readable even SGX is present, in which +* case guest is not allowed to read either. +*/ +if ( !sgxv->readable ) + { + r =0 ; +break ; + } + + data = sgxv->ia32_sgxlepubkeyhash[msr -MSR_IA32_SGXLEPUBKEYHASH0 ]; + + *msr_content = data; + +break ; +default : + r =0 ; +break ; + } + +return r; +} + +int sgx_msr_write_intercept (struct vcpu *v,unsigned int msr,u64 msr_content) +{
xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 569
Quoted 10 lines
xen/arch/x86/hvm/vmx/sgx.c +10
-0+struct sgx_vcpu *sgxv =to_sgx_vcpu (v); +int r =1 ; + +if ( !domain_has_sgx (v->domain) ) +return 0 ; + +switch ( msr ) + { +case MSR_IA32_FEATURE_CONTROL : +/* sliently drop */
Silently dropping is not ok. This change needs rebasing over c/s 46c3acb308 where I have fixed up the writeability of FEATURE_CONTROL.
Quoted 59 lines
xen/arch/x86/hvm/vmx/sgx.c +32
-0+break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +/* +* SDM 35.1 Model-Specific Registers, table 35-2. +* +* IA32_SGXLEPUBKEYHASH[0..3]: +* +* - If CPUID.0x7.0:ECX[30] = 1, FEATURE_CONTROL[17] is available. +* - Write permitted if CPUID.0x12.0:EAX[0] = 1 && +* FEATURE_CONTROL[17] = 1 && FEATURE_CONTROL[0] = 1. +* +* sgxv->writable == 1 means sgx_ia32_sgxlepubkeyhash_writable() and +* domain_has_sgx_launch_control(d) both are true. +*/ +if ( !sgxv->writable ) + { + r =0 ; +break ; + } + + sgxv->ia32_sgxlepubkeyhash[msr -MSR_IA32_SGXLEPUBKEYHASH0 ] = + msr_content; + +break ; +default : + r =0 ; +break ; + } + +return r; +} +static bool_t sgx_enabled_in_bios (void ) { uint64_t val, sgx_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE |xen/arch/x86/hvm/vmx/vmx.c +5
-0@@ -470,6 +470,8 @@ static int vmx_vcpu_initialise(struct vcpu *v)if ( v->vcpu_id ==0 ) v->arch.user_regs.rax =1 ; +sgx_vcpu_init (v); +return 0 ; }@@ -1048,6 +1050,9 @@ static void vmx_ctxt_switch_to(struct vcpu *v)if ( v->domain->arch.hvm_domain.pi_ops.switch_to ) v->domain->arch.hvm_domain.pi_ops.switch_to (v); + +if (domain_has_sgx (v->domain) ) +sgx_ctxt_switch_to (v); }
xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_read_intercept(unsigned int msr, uint64_t *msr_content) · line 2888
Quoted 8 lines
xen/arch/x86/hvm/vmx/vmx.c +5
-0@@ -2876,10 +2881,20 @@ static int vmx_msr_read_intercept(unsigned int msr, uint64_t *msr_content)__vmread (GUEST_IA32_DEBUGCTL , msr_content);break ;case MSR_IA32_FEATURE_CONTROL : +/* If neither SGX nor nested is supported, this MSR should not be +* touched */ +if ( !sgx_msr_read_intercept (current, msr, msr_content) && + !nvmx_msr_read_intercept (msr, msr_content) ) + goto gp_fault;
Unfortunately, this logic is broken. In the case that both SMX and VMX are configured, the VMX handler will clobber the values set up by the SGX handler. Sergey has a VMX-policy series (v1 posted, v2 in the works) to start addressing some of the issues on the VMX side, but fundamentally, all reads like this need serving out of a single policy, rather than having different subsystems fighting for control of the values. (The Xen MSR code is terrible for this at the moment.)
Quoted 13 lines
xen/arch/x86/hvm/vmx/vmx.c +5
-0+break ; case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_VMFUNC:if ( !nvmx_msr_read_intercept (msr, msr_content) ) goto gp_fault;break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +if ( !sgx_msr_read_intercept (current, msr, msr_content) ) + goto gp_fault; +break ;case MSR_IA32_MISC_ENABLE :rdmsrl (MSR_IA32_MISC_ENABLE , *msr_content);/* Debug Trace Store is not supported. */
xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content) · line 3139
Quoted 6 lines
xen/arch/x86/hvm/vmx/vmx.c +3
-0@@ -3119,10 +3134,19 @@ static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content)break ; }case MSR_IA32_FEATURE_CONTROL : +/* See vmx_msr_read_intercept */ +if ( !sgx_msr_write_intercept (current, msr, msr_content) && + !nvmx_msr_write_intercept (msr, msr_content) )
Definitely needs a rebase. nvmx_msr_write_intercept() has been removed.
~Andrew
Quoted 78 lines
xen/arch/x86/hvm/vmx/vmx.c +6
-0+ goto gp_fault; +break ; case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_TRUE_ENTRY_CTLS:if ( !nvmx_msr_write_intercept (msr, msr_content) ) goto gp_fault;break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +if ( !sgx_msr_write_intercept (current, msr, msr_content) ) + goto gp_fault; +break ; case MSR_P6_PERFCTR(0)...MSR_P6_PERFCTR(7): case MSR_P6_EVNTSEL(0)...MSR_P6_EVNTSEL(7): case MSR_CORE_PERF_FIXED_CTR0...MSR_CORE_PERF_FIXED_CTR2:xen/include/asm-x86/cpufeature.h +3
-0@@ -98,6 +98,9 @@#define cpu_has_smap boot_cpu_has(X86_FEATURE_SMAP)#define cpu_has_sha boot_cpu_has(X86_FEATURE_SHA) +/* CPUID level 0x00000007:0.ecx */ +#define cpu_has_sgx_launch_control boot_cpu_has(X86_FEATURE_SGX_LAUNCH_CONTROL) +/* CPUID level 0x80000007.edx */ #define cpu_has_itsc boot_cpu_has(X86_FEATURE_ITSC)xen/include/asm-x86/hvm/vmx/sgx.h +22
-0@@ -75,4 +75,26 @@ int hvm_populate_epc(struct domain *d, unsigned long epc_base_pfn,int hvm_reset_epc (struct domain *d,bool_t free_epc);void hvm_destroy_epc (struct domain *d); +/* Per-vcpu SGX structure */ +struct sgx_vcpu { +uint64_t ia32_sgxlepubkeyhash[4 ]; +/* +* Although SDM says if SGX is present, then IA32_SGXLEPUBKEYHASHn are +* available for read, but in reality for SKYLAKE client machines, those +* those MSRs are not available if SGX is present. +*/ +bool_t readable; +bool_t writable; +}; +#define to_sgx_vcpu (v) (&(v->arch.hvm_vmx.sgx)) + +bool_t sgx_ia32_sgxlepubkeyhash_writable (void ); +bool_t domain_has_sgx (struct domain *d); +bool_t domain_has_sgx_launch_control (struct domain *d); + +void sgx_vcpu_init (struct vcpu *v); +void sgx_ctxt_switch_to (struct vcpu *v); +int sgx_msr_read_intercept (struct vcpu *v,unsigned int msr,u64 *msr_content); +int sgx_msr_write_intercept (struct vcpu *v,unsigned int msr,u64 msr_content); +#endif /* __ASM_X86_HVM_VMX_SGX_H__ */ xen/include/asm-x86/hvm/vmx/vmcs.h +2
-0@@ -160,6 +160,8 @@ struct arch_vmx_struct {* pCPU and wakeup the related vCPU. */ struct pi_blocking_vcpu pi_blocking; + +struct sgx_vcpu sgx; };int vmx_create_vmcs (struct vcpu *v);xen/include/asm-x86/msr-index.h +0
-0@@ -296,6 +296,12 @@#define IA32_FEATURE_CONTROL_SENTER_PARAM_CTL 0x7f00#define IA32_FEATURE_CONTROL_ENABLE_SENTER 0x8000
xen/include/asm-x86/msr-index.h line 307
Quoted 10 lines
xen/include/asm-x86/msr-index.h +6
-0#define IA32_FEATURE_CONTROL_SGX_ENABLE 0x40000 +#define IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE 0x20000 + +#define MSR_IA32_SGXLEPUBKEYHASH0 0x0000008c +#define MSR_IA32_SGXLEPUBKEYHASH1 0x0000008d +#define MSR_IA32_SGXLEPUBKEYHASH2 0x0000008e +#define MSR_IA32_SGXLEPUBKEYHASH3 0x0000008f#define MSR_IA32_TSC_ADJUST 0x0000003b
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 111 lines · Kai
Hi Andrew,
Thank you very much for comments. Sorry for late reply, and please see my reply below.
Quoted 23 lines · Andrew Cooper
Quoted 3 lines · Kai Huang
Hi all,
This series is RFC Xen SGX virtualization support design and RFC draft patches.
Thankyou very much for this design doc.
Quoted 11 lines
2. SGX Virtualization Design
2.1 High Level Toolstack Changes:
2.1.1 New 'epc' parameter
EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin
alao should be able to get all domain's virtual EPC size.Quoted 4 lines
For this purpose, a new 'epc = <size>' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted.
Quoted 5 lines
How will this interact with multi-package servers? Even though its fine to implement the single-package support first, the design should be extensible to the multi-package case.
First of all, what are the implications of multi-package SGX?
(Somewhere) you mention changes to scheduling. I presume this is because a guest with EPC mappings in EPT must be scheduled on the same package, or ENCLU[EENTER] will fail. I presume also that each package will have separate, unrelated private keys?
The ENCLU[EENTE] will continue to work on multi-package server. Actually I was told all ISA existing behavior documented in SDM won't change for server, as otherwise this would be a bad design :)
Unfortunately I was told I cannot talk about MP server SGX a lot now. Basically I can only talk about staff already documented in SDM (sorry :( ). But I guess multiple EPC in CPUID is designed to cover MP server, at lease mainly (we can do reasonable guess).
In terms of the design, I think we can follow XL config file parameters for memory. 'epc' parameter will always specify totol EPC size that the domain has. And we can use existing NUMA related parameters, such as setting cpus='...' to physically pin vcpu to specific pCPUs, so that EPC will be mostly allocated from related node. If that node runs out of EPC, we can decide whether to allocate EPC from other node, or fail to create domain. I know Linux supports NUMA policy which can specify whether to allow allocating memory from other nodes, does Xen has such policy? Sorry I haven't checked this. If Xen has such policy, we need to choose whether to use memory policy, or introduce new policy for EPC.
If we are going to support vNUAM EPC in the future. We can also use similar way to config vNUMA EPC in XL config.
Sorry I mentioned scheduling. I should say *potentially* :). My thinking was as SGX is per-thread, then SGX info reported by different CPU package may be different (ex, whether SGX2 is supported), then we may need scheduler to be aware of SGX. But I think we don't have to consider this now.
What's your comments?
Quoted 2 lines
I presume there is no sensible way (even on native) for a single logical process to use multiple different enclaves? By extension, does it make sense to try and offer parts of multiple enclaves to a single VM?
The native machine allows running multiple enclaves, even signed by multiple authors. SGX only has limit that before launching any other enclave, Launch Enclave (LE) must be launched. LE is the only enclave that doesn't require EINITTOKEN in EINIT. For LE, its signer (
SHA256(sigstruct->modulus)) must be equal to the value inIA32_SGXLEPUBKEYHASHnMSRs. LE will generates EINITTOKEN for other enclaves (EINIT for other enclaves requires EINITTOKEN). For other enclaves, there's no such limitation that enclave's signer must matchIA32_SGXLEPUBKEYHASHnso the signer can be anybody. But for other enclaves, before running EINIT, the LE's signer (which is equal toIA32_SGXLEPUBKEYHASHnas explained above) needs to be updated toIA32_SGXLEPUBKEYHASHn(MSRs can be changed, for example, when there's multiple LEs running in OS). This is because EINIT needs to perform EINITTOKEN integrity check (EINITTOKEN contains MAC info that calculated by LE, and EINIT needs LE'sIA32_SGXLEPUBKEYHASHnto derive the key to verify MAC).SGX in VM doesn't change those behaviors, so in VM, the enclaves can also be signed by anyone, but Xen needs to emulate
IA32_SGXLEPUBKEYHASHnso that when one VM is running, the correctIA32_SGXLEPUBKEYHASHnare already in physical MSRs.Quoted 9 lines
Quoted 5 lines
2.1.3 Notify domain's virtual EPC base and size to Xen
Xen needs to know guest's EPC base and size in order to populate EPC pages for it. Toolstack notifies EPC base and size to Xen via
XEN_DOMCTL_set_cpuid.I am currently in the process of reworking the Xen/Toolstack interface when it comes to CPUID handling. The latest design is available here: https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg00378.html but the end result will be the toolstack expressing its CPUID policy in terms of the architectural layout.
Therefore, I would expect that, however the setting is represented in the configuration file, xl/libxl would configure it with the hypervisor by setting CPUID.0x12[2] with the appropriate base and size.
I agree. I saw you are planning to introduce new XEN_DOMCTL_get{set}_cpuid_policy, which will allow toolstack to query/set cpuid policy in single hypercall (if I understand correctly), so I think we should definitely use the new hypercalls.
I also saw you are planning to introduce new hypercall to query raw/host/pv_max/hvm_max cpuid policy (not just featureset), so I think 'xl sgxinfo' (or xl info -sgx) can certainly use that to get physical SGX info (EPC info). And 'xl sgxlist' (or xl list -sgx) can use XEN_DOMCTL_get{set}_cpuid_policy to display domain's SGX info (EPC info).
Btw, do you think we need 'xl sgxinfo' and 'xl sgxlist'? If we do, which is better? New 'xl sgxinfo' and 'xl sgxlist', or extending existing 'xl info' and 'xl list' to support SGX, such as 'xl info -sgx' and 'xl list -sgx' above?
Quoted 40 lines
Quoted 31 lines
2.1.4 Launch Control Support (?)
Xen Launch Control Support is about to support running multiple domains with each running its own LE signed by different owners (if HW allows, explained below). As explained in 1.4 SGX Launch Control, EINIT for LE (Launch Enclave) only succeeds when SHA256(SIGSTRUCT.modulus) matches
IA32_SGXLEPUBKEYHASHn, and EINIT for other enclaves will derive EINITTOKEN key according toIA32_SGXLEPUBKEYHASHn. Therefore, to support this, guest's virtualIA32_SGXLEPUBKEYHASHnmust be updated to phyiscal MSRs before EINIT (which also means the physicalIA32_SGXLEPUBKEYHASHnneed to be *unlocked* in BIOS before booting to OS).For physical machine, it is BIOS's writer's decision that whether BIOS would provide interface for user to specify customerized
IA32_SGXLEPUBKEYHASHn(it is default to digest of Intel's signing key after reset). In reality, OS's SGX driver may require BIOS to make MSRs *unlocked* and actively write the hash value to MSRs in order to run EINIT successfully, as in this case, the driver will not depend on BIOS's capability (whether it allows user to customerizeIA32_SGXLEPUBKEYHASHnvalue).The problem is for Xen, do we need a new parameter, such as 'lehash=<SHA256>' to specify the default value of guset's virtual
IA32_SGXLEPUBKEYHASHn? And do we need a new parameter, such as 'lewr' to specify whether guest's virtual MSRs are locked or not before handling to guest's OS?I tends to not introduce 'lehash', as it seems SGX driver would actively update the MSRs. And new parameter would add additional changes for upper layer software (such as openstack). And 'lewr' is not needed either as Xen can always *unlock* the MSRs to guest.
Quoted 6 lines
Please give comments?
Currently in my RFC patches above two parameters are not implemented. Xen hypervisor will always *unlock* the MSRs. Whether there is 'lehash' parameter or not doesn't impact Xen hypervisor's emulation of
IA32_SGXLEPUBKEYHASHn. See below Xen hypervisor changes for details.Reading around, am I correct with the following?
Quoted 8 lines
Quoted 1 line
1) Some processors have no launch control. There is no restriction on which enclaves can boot.
Yes that some processors have no launch control. However it doesn't mean there's no restriction on which enclaves can boot. Contrary, on those machines only Intel's Launch Enclave (LE) can run, as on those machine,
IA32_SGXLEPUBKEYHASHneither doesn't exist, or equal to digest of Intel's signing RSA pubkey. However although only Intel's LE can be run, we can still run other enclaves from other signers. Please see my reply above.Quoted 2 lines
2) Some Skylake client processors claim to have launch control, but the MSRs are unavailable (is this an erratum?). These are limited to booting enclaves matching the Intel public key.
Sorry I don't know whether this is an erratum. I will get back to you after confirming internally.
Hi Andrew,
I raised this internally, and it turns out that in the latest SDM Intel has fixed the statement, so that IA32_SGXLEPUBKEYHASHn MSRs are only available when both SGX and SGX_LC is present in CPUID. When I was writing the design and patches, I was referring to old SDM, and the old one doesn't mention SGX_LC in CPUID as condition. So it is my fault and this statement has been fixed in latest SDM (41.2.2 Intel SGX Launch Control Configuration):
https://software.intel.com/sites/default/files/managed/7c/f1/332831-sdm-vol-3d.pdf
However in latest SDM volume 4: Model-Specific Registers:
https://software.intel.com/sites/default/files/managed/22/0d/335592-sdm-vol-4.pdf
You can still see that for IA32_SGXLEPUBKEYHASHn (table 2-2, register address 8CH): "Read permitted If CPUID.(EAX=12H,ECX=0H):EAX[0]=1". So there's still error in SDM.
I don't think this will be an erratum. Intel will fix the error in vol 4 in next version SDM. We should refer to 41.2.2 as it has accurate description.
Quoted 10 lines
Quoted 4 lines
3) Launch control may be locked by the BIOS. There may be a custom hash, or it might be the Intel default. Xen can't adjust it at all, but can support running any number of VMs with matching enclaves.
Yes Launch control may be locked by BIOS, although this depends on whether BIOS provides interface for user to configure. I was told that typically BIOS will unlock Launch Control, as SGX driver is expecting such behavior. But I am not sure we can always assume this.
Whether there will be custom hash also depends on BIOS. BIOS may or may not provide interface for user to configure custom hash. So on physical machine, I think we need to consider all the cases. On machine that with Launch control *unlocked*, Xen is able to dynamically change
IA32_SGXLEKEYHASHnso that Xen is able to run multiple VM with each running LE from different signer. However if launch control is *locked* in BIOS, then Xen is still able to run multiple VM, but all VM can only run LE from the signer that matches theIA32_SGXLEPUBKEYHASHn(which in most case should be Intel default, but can be custom hash if BIOS allows user to configure).Sorry I am not quite sure the typical implementation of BIOS. I think I can reach out internally and get back to you if I have something.
I also reached out internally to find the typical BIOS implementation in terms of SGX LC. Typically BIOS will neither provide configuration options for user to set custom hash, nor select whether MSRs are locked or not. Typically for client machine, MSRs are locked with Intel default, and for server machine, MSRs are unlocked. But we cannot rule out 3rd party to provide different BIOS that may provide options for user to choose locked/unlocked mode, and/or for user to specify custom hash. Custom hash + locked mode may be useful for some special purpose (ex, IT management) as it provides most secure option -- that even kernel/VMM can only launch LE signed with particular signer. In case of VM, custom hash + locked mode may be even more useful than bare-metal as VM is usually supposed to run some particular purpose appliance.
So I think it is better to keep 'lehash' and 'lewr' XL parameters. They both are optional -- the former provides custom hash, and the latter set VM to be in unlocked mode. If neither is specified, then VM will be in locked mode, and VM's virtual IA32_SGXLEPUBKEYHASHn either have Intel's default value (when physical machine is unlocked), or have machine's MSR values (when machine is in locked mode). And when physical machine is in locked mode, specifying either 'lehash' or 'lewr' will result in creating VM failure.
So we have 3 XL parameters for SGX: 'epc', 'lehash' and 'lewr', probably we should consolidate them into one XL parameter, such sgx=['epc=<size>', 'lehash=<sha256>', 'lewr=[on|off]'] ?
Thanks,
-Kai
Quoted 211 lines
Quoted 3 lines
4) Launch control may be unlocked by the BIOS. In this case, Xen can context switch a hash per domain, and run all enclaves.
Yes. With enclave == LE I think you meant.
Quoted 2 lines
The eventual plans for CPUID and MSR levelling should allow all of these to be expressed in sensible ways, and I don't forsee any issues with supporting all of these scenarios.
So do you think we should have 'lehash' and 'lewr' parameters in XL config file? The former provides custom hash, and the latter provides whether unlock guest's Launch control.
My thinking is SGX driver needs to *actively* write LE's pubkey hash to
IA32_SGXLEPUBKEYHASHnin *unlocked* mode, so 'lehash' alone is not needed. 'lehash' only has meaning when 'lewr' is needed to provide a default hash value in locked mode, as if we always use *unlocked* mode for guest, 'lehash' is not necessary.Quoted 41 lines
Quoted 41 lines
2.2 High Level Xen Hypervisor Changes:
2.2.1 EPC Management (?)
Xen hypervisor needs to detect SGX, discover EPC, and manage EPC before supporting SGX to guest. EPC is detected via SGX CPUID 0x12.0x2. It's possible that there are multiple EPC sections (enumerated via sub-leaves
0x3and so on, until invaid EPC is reported), but this is only true on multiple-socket server machines. For server machines there are additional things also needs to be done, such as NUMA EPC, scheduling, etc. We will support server machine in the future but currently we only support one EPC.EPC is reported as reserved memory (so it is not reported as normal memory). EPC must be managed in 4K pages. CPU hardware uses EPCM to track status of each EPC pages. Xen needs to manage EPC and provide functions to, ie, alloc and free EPC pages for guest.
There are two ways to manage EPC: Manage EPC separately; or Integrate it to existing memory management framework.
It is easy to manage EPC separately, as currently EPC is pretty small (~100MB), and we can even put them in a single list. However it is not flexible, for example, you will have to write new algorithms when EPC becomes larger, ex, GB. And you have to write new code to support NUMA EPC (although this will not come in short time).
Integrating EPC to existing memory management framework seems more reasonable, as in this way we can resume memory management data structures/algorithms, and it will be more flexible to support larger EPC and potentially NUMA EPC. But modifying MM framework has a higher risk to break existing memory management code (potentially more bugs).
In my RFC patches currently we choose to manage EPC separately. A new structure
epc_pageis added to represent a single 4K EPC page. A whole array of structepc_pagewill be allocated during EPC initialization, so that given the other, one of PFN of EPC page and 'structepc_page' can be got by adding offset.Quoted 7 lines
Quoted 3 lines
But maybe integrating EPC to MM framework is more reasonable. Comments?
2.2.2 EPC Virtualization (?)
It looks like managing the EPC is very similar to managing the NVDIMM ranges. We have a (set of) physical address ranges which need 4k ownership granularity to different domains.
I think integrating this into struct
page_structis the better way to go.Will do. So I assume we will introduce new
MEMF_epc, and use existing alloc_domheap/xenheap_pages to allocate EPC?MEMF_epccan also be used if we need to support ballooning in the future (using existing XENMEM_{decrease/increase}_reservation.Quoted 79 lines
Quoted 79 lines
This part is how to populate EPC for guests. We have 3 choices:
- Static Partitioning
- Oversubscription
- BallooningStatic Partitioning means all EPC pages will be allocated and mapped to guest when it is created, and there's no runtime change of page table mappings for EPC pages. Oversubscription means Xen hypervisor supports EPC page swapping between domains, meaning Xen is able to evict EPC page from another domain and assign it to the domain that needs the EPC. With oversubscription, EPC can be assigned to domain on demand, when EPT violation happens. Ballooning is similar to memory ballooning. It is basically "Static Partitioning" + "Balloon driver" in guest.
Static Partitioning is the easiest way in terms of implementation, and there will be no hypervisor overhead (except EPT overhead of course), because in "Static partitioning", there is no EPT violation for EPC, and Xen doesn't need to turn on ENCLS VMEXIT for guest as ENCLS runs perfectly in non-root mode.
Ballooning is "Static Partitioning" + "Balloon driver" in guest. Like "Static Paratitioning", ballooning doesn't need to turn on ENCLS VMEXIT, and doesn't have EPT violation for EPC either. To support ballooning, we need ballooning driver in guest to issue hypercall to give up or reclaim EPC pages. In terms of hypercall, we have two choices: 1) Add new hypercall for EPC ballooning; 2) Using existing XENMEM_{increase/decrease}_reservation with new memory flag, ie,
XENMEMF_epc. I'll discuss more regarding to adding dedicated hypercall or not later.Oversubscription looks nice but it requires more complicated implemetation. Firstly, as explained in 1.3.3 EPC Eviction & Reload, we need to follow specific steps to evict EPC pages, and in order to do that, basically Xen needs to trap ENCLS from guest and keep track of EPC page status and enclave info from all guest. This is because:
- To evict regular EPC page, Xen needs to know SECS location
- Xen needs to know EPC page type: evicting regular EPC and
evicting SECS,
VA page have different steps.
- Xen needs to know EPC page status: whether the page is blocked
or not.Those info can only be got by trapping ENCLS from guest, and parsing its parameters (to identify SECS page, etc). Parsing ENCLS parameters means we need to know which ENCLS leaf is being trapped, and we need to translate guest's virtual address to get physical address in order to locate EPC page. And once ENCLS is trapped, we have to emulate ENCLS in Xen, which means we need to reconstruct ENCLS parameters by remapping all guest's virtual address to Xen's virtual address (
gva->gpa->pa->xen_va), as ENCLS always use *effective address* which is able to be traslated by processor when running ENCLS.--------------------------------------------------------------
| ENCLS |
--------------------------------------------------------------
| /|\
ENCLS VMEXIT| | VMENTRY
| |
\|/ |1) parse ENCLS parameters
2) reconstruct(remap) guest's ENCLS parameters
3) run ENCLS on behalf of guest (and skip ENCLS)
4) on success, update EPC/enclave info, or inject errorAnd Xen needs to maintain each EPC page's status (type, blocked or not, in enclave or not, etc). Xen also needs to maintain all Enclave's info from all guests, in order to find the correct SECS for regular EPC page, and enclave's linear address as well.
So in general, "Static Partitioning" has simplest implementation, but obviously not the best way to use EPC efficiently; "Ballooning" has all pros of Static Partitioning but requies guest balloon driver; "Oversubscription" is best in terms of flexibility but requires complicated hypervisor implemetation.
We have implemented "Static Partitioning" in RFC patches, but needs your feedback on whether it is enough. If not, which one should we do at next stage -- Ballooning or Oversubscription. IMO Ballooning may be good enough, given fact
that currently memory is also "Static Partitioning" + "Ballooning".Quoted 5 lines
Quoted 1 line
Comments?
Definitely go for static partitioning to begin with. This is far simpler to implement.
I can't see a pressing usecase for oversubscription or ballooning. Any datacenter work will be using exclusively static, and I expect static will fine for all (or at least, most) client usecases.
Thanks. So for the first stage I will focus on static partitioning.
Quoted 10 lines
Quoted 10 lines
2.2.3 Populate EPC for Guest
Toolstack notifies Xen about domain's EPC base and size by
XEN_DOMCTL_set_cpuid, so currently Xen populates all EPC pages for guest inXEN_DOMCTL_set_cpuid, particularly, in handlingXEN_DOMCTL_set_cpuidfor CPUID.0x12.0x2. Once Xen checks the values passed from toolstack is valid, Xen will allocate all EPC pages and setup EPT mappings for guest.Quoted 3 lines
Quoted 1 line
2.2.4 New Dedicated Hypercall (?)
All this information should (eventually) be available via the appropriate SYSCTL_get_{cpuid,msr}_policy hypercalls. I don't see any need for dedicated hypercalls.
Yes I agree. Originally I had concern that without dedicated hypercall, it is hard to implement 'xl sgxinfo' and 'xl sgxlist', but according to your new CPUID enhancement plan, the two can be done via the new hypercalls to query Xen's and domain's cpuid policy. See my reply above regarding to "Notify Xen about guest's EPC info".
Quoted 21 lines
Quoted 21 lines
2.2.9 Guest Suspend & Resume
On hardware, EPC is destroyed when power goes to S3-S5. So Xen will destroy guest's EPC when guest's power goes into S3-S5. Currently Xen is notified by Qemu in terms of S State change via
HVM_PARAM_ACPI_S_STATE, where Xen will destroy EPC if S State is S3-S5.Specifically, Xen will run EREMOVE for guest's each EPC page, as guest may not handle EPC suspend & resume correctly, in which case physically guest's EPC pages may still be valid, so Xen needs to run EREMOVE to make sure all EPC pages are becoming invalid. Otherwise further operation in guest on EPC may fault as it assumes all EPC pages are invalid after guest is resumed.
For SECS page, EREMOVE may fault with
SGX_CHILD_PRESENT, in which case Xen will keep this SECS page into a list, and call EREMOVE for them again after all EPC pages have been called with EREMOVE. This time the EREMOVE on SECS will succeed as all children (regular EPC pages) have already been removed.2.2.10 Destroying Domain
Quoted 9 lines
Quoted 6 lines
Normally Xen just frees all EPC pages for domain when it is destroyed. But Xen will also do EREMOVE on all guest's EPC pages (described in above 2.2.7) before free them, as guest may shutdown unexpected (ex, user kills guest), and in this case, guest's EPC may still be valid.
2.3 Additional Point: Live Migration, Snapshot Support (?)
How big is the EPC? If we are talking MB rather than GB, movement of the EPC could be after the pause, which would add some latency to live migration but should work. I expect that people would prefer to have the flexibility of migration even at the cost of extra latency.
The EPC is typically ~100MB at maximum (as I observed). The EPC is typically reserved with EPCM (EPC map, which is invisible to SW) together by BIOS as processor reserved memory (RPM). On real machine, for both our internal develop machines, and some machines that from Dell, HP, Lenovo (that you can buy from market now), BIOS always provides 3 choices in terms RPM: 32M, 64M, and 128M. And with 128M RPM, EPC is slightly less than 100M.
The problem is EPC cannot be moved. I think you were saying moving EPC by evicting EPC out at last stage and copy evicted content to remote, and then reload. However I don't think this will work, as EPC eviction itself needs to use a VA slot (which itslef is EPC), so you can image that the VA slots cannot be moved to remote. Even if they can, they cannot be used to reload EPC in remote, as info in VA slot is bound to platform and cannot be used on remote.
To support live migration, we can only choose to ignore EPC during live migration and let guest SGX driver/user SW stack to handle restoring enclave (which is actually a lot simpler in hypervisor/toolstack's implementation) . Guest SGX driver needs to handle lose EPC anyway, as EPC is destroyed in S3-S5. The only difference is to support live migration, guest SGX driver needs to support *sudden* lose of EPC, which is not HW behavior, and I was told that currently both Windows & Linux SGX driver already support *sudden* lose of EPC, which leaves us a question whether we need to support SGX live migration (and snapshot).
Quoted 6 lines
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 4 lines
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 03/15] xen: x86: add early stage SGX feature detection
In reply to Andrew CooperQuoted 74 lines · Andrew Cooper
Quoted 12 lines · Kai Huang
This patch adds early stage SGX feature detection via SGX CPUID
0x12. Functiondetect_sgxis added to detect SGX info on each CPU (called fromvmx_cpu_up). SDM says SGX info returned by CPUID is per-thread, and we cannot assume all threads will return the same SGX info, so we have to detect SGX for each CPU. For simplicity, currently SGX is only supported when all CPUs reports the same SGX info.SDM also says it's possible to have multiple EPC sections but this is only for multiple-socket server, which we don't support now (there are other things need to be done, ex, NUMA EPC, scheduling, etc, as well), so currently only one EPC is supported.
Quoted 5 lines
Dedicated files
sgx.candsgx.hare added (under vmx directory as SGX is Intel specific) for bulk of above SGX detection code detection code, and for further SGX code as well.Signed-off-by Kai Huang <[email protected]>I am not sure putting this under hvm/ is a sensible move. Almost everything in this patch is currently common, and I can forsee us wanting to introduce PV support, so it would be good to introduce this in a guest-neutral location to begin with.
xen/arch/x86/hvm/vmx/sgx.c
Quoted 24 lines
--- xen/arch/x86/hvm/vmx/Makefile | 1 + xen/arch/x86/hvm/vmx/sgx.c | 208 ++++++++++++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmcs.c | 4 + xen/include/asm-x86/cpufeature.h | 1 + xen/include/asm-x86/hvm/vmx/sgx.h | 45 +++++++++ 5 files changed, 259 insertions(+)create mode 100644
xen/arch/x86/hvm/vmx/sgx.ccreate mode 100644xen/include/asm-x86/hvm/vmx/sgx.hxen/arch/x86/hvm/vmx/Makefile +1
-0@@ -4,3 +4,4 @@ obj-y += realmode.o obj-y += vmcs.o obj-y += vmx.o obj-y += vvmx.o +obj-y += sgx.oxen/arch/x86/hvm/vmx/sgx.c new file +0
-0This file looks like it should be
arch/x86/sgx.c, given its current content.xen/arch/x86/hvm/vmx/sgx.c line 2
Quoted 3 lines
xen/arch/x86/hvm/vmx/sgx.c new file +2
-0@@ -0,0 +1,208 @@ +/* +* Intel Software Guard Extensions support Please include a GPLv2 header.
xen/arch/x86/hvm/vmx/sgx.c line 14
Quoted 12 lines
xen/arch/x86/hvm/vmx/sgx.c new file +12
-0+* +* Author: Kai Huang <[email protected]> +*/ + +#include <asm/cpufeature.h> +#include <asm/msr-index.h> +#include <asm/msr.h> +#include <asm/hvm/vmx/sgx.h> +#include <asm/hvm/vmx/vmcs.h> + +static struct sgx_cpuinfo __read_mostly sgx_cpudata[NR_CPUS ]; +static struct sgx_cpuinfo __read_mostly boot_sgx_cpudata;I don't think any of this is necessary. The description says that all EPCs across the server will be reported in CPUID subleaves, and our implementation gives up if the data are non-identical across CPUs.
Quoted 2 lines
Therefore, we only need to keep one copy of the data, and check check APs against the master copy.
Right. boot_sgx_cpudata is what we need. Currently detect_sgx is called from vmx_cpu_up. How about changing to calling it from identify_cpu, and something like below ?
if ( c == &boot_cpu_data )
detect_sgx(&boot_sgx_cpudata);
else {
struct sgx_cpuinfo tmp;
detect_sgx(&tmp);
if ( memcmp(&boot_sgx_cpudata, &tmp, sizeof (tmp)) )
//disable SGX
}Thanks,
-Kai
Quoted 6 lines
Let me see about splitting up a few bits of the existing CPUID infrastructure, so we can use the host cpuid policy more effectively for Xen related things.
Quoted 7 lines
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 09/15] xen: vmx: handle SGX related MSRs
In reply to Andrew CooperQuoted 89 lines · Andrew Cooper
Quoted 74 lines · Kai Huang
This patch handles
IA32_FEATURE_CONTROLandIA32_SGXLEPUBKEYHASHnMSRs.For
IA32_FEATURE_CONTROL, if SGX is exposed to domain, thenSGX_ENABLEbit is always set. If SGX launch control is also exposed to domain, and physicalIA32_SGXLEPUBKEYHASHnare writable, thenSGX_LAUNCH_CONTROL_ENABLEbit is also always set. Write toIA32_FEATURE_CONTROLis ignored.For
IA32_SGXLEPUBKEYHASHn, a new 'structsgx_vcpu' is added for per-vcpu SGX staff, and currently it has vcpu's virtual ia32_sgxlepubkeyhash[0-3]. Two boolean 'readable' and 'writable' are also added to indicate whether virtualIA32_SGXLEPUBKEYHASHnare readable and writable.During vcpu is initialized, virtual
ia32_sgxlepubkeyhashare also initialized. If physicalIA32_SGXLEPUBKEYHASHnare writable, thenia32_sgxlepubkeyhashare set to Intel's default value, as for physical machine, those MSRs will have Intel's default value. If physical MSRs are not writable (it is *locked* by BIOS before handling to Xen), then we try to read those MSRs and use physical values as defult value for virtual MSRs. One thing isrdmsr_safeis used, as although SDM says if SGX is present,IA32_SGXLEPUBKEYHASHnare available for read, but in reality, skylake client (at least some, depending on BIOS) doesn't have those MSRs available, so we userdmsr_safeand set readable to false if it returns error code.For
IA32_SGXLEPUBKEYHASHnMSR read from guest, if physical MSRs are not readable, guest is not allowed to read either, otherwise vcpu's virtual MSR value is returned.For
IA32_SGXLEPUBKEYHASHnMSR write from guest, we allow guest to write if both physical MSRs are writable and SGX launch control is exposed to domain, otherwise error is injected.To make EINIT run successfully in guest, vcpu's virtual
IA32_SGXLEPUBKEYHASHnwill be update to physical MSRs when vcpu is scheduled in.Signed-off-by Kai Huang <[email protected]>--- xen/arch/x86/hvm/vmx/sgx.c | 194 +++++++++++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmx.c | 24 +++++ xen/include/asm-x86/cpufeature.h | 3 + xen/include/asm-x86/hvm/vmx/sgx.h | 22 +++++ xen/include/asm-x86/hvm/vmx/vmcs.h | 2 + xen/include/asm-x86/msr-index.h | 6 ++ 6 files changed, 251 insertions(+)xen/arch/x86/hvm/vmx/sgx.c +22
-0@@ -405,6 +405,200 @@ void hvm_destroy_epc(struct domain *d)hvm_reset_epc (d, true); } +/* Whether IA32_SGXLEPUBKEYHASHn are physically *unlocked* by BIOS */ +bool_t sgx_ia32_sgxlepubkeyhash_writable (void ) +{ +uint64_t sgx_lc_enabled =IA32_FEATURE_CONTROL_SGX_ENABLE | +IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE | +IA32_FEATURE_CONTROL_LOCK ; +uint64_t val; + +rdmsrl (MSR_IA32_FEATURE_CONTROL , val); + +return (val & sgx_lc_enabled) == sgx_lc_enabled; +} + +bool_t domain_has_sgx (struct domain *d) +{ +/* hvm_epc_populated(d) implies CPUID has SGX */ +return hvm_epc_populated (d); +} + +bool_t domain_has_sgx_launch_control (struct domain *d) +{ +struct cpuid_policy *p = d->arch.cpuid;xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 439
Quoted 10 lines
xen/arch/x86/hvm/vmx/sgx.c +10
-0+ +if ( !domain_has_sgx (d) ) +return false; + +/* Unnecessary but check anyway */ +if ( !cpu_has_sgx_launch_control ) +return false; + +return !!p->feat.sgx_launch_control; +}Both of these should be d->arch.cpuid->feat.{sgx,sgx_lc} only, and not from having individual helpers.
Quoted 5 lines
The CPUID setup during host boot and domain construction should take care of setting everything up properly, or hiding the features from the guest. The point of the work I've been doing is to prevent situations where the guest can see SGX but something doesn't work because of Xen using nested checks like this.
Thanks for comments. Will change to simple check against d->arch.cpuid->feat.{sgx,sgx_lc}.
Quoted 42 lines
Quoted 28 lines
xen/arch/x86/hvm/vmx/sgx.c +27
-0+ +/* Digest of Intel signing key. MSR's default value after reset. */ +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH0 0xa6053e051270b7ac +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH1 0x6cfbe8ba8b3b413d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH2 0xc4916d99f2b3735d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH3 0xd4f8c05909f9bb3b + +void sgx_vcpu_init (struct vcpu *v) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); + +memset (sgxv,0 ,sizeof (*sgxv)); + +if (sgx_ia32_sgxlepubkeyhash_writable () ) + { +/* +* If physical MSRs are writable, set vcpu's default value to Intel's +* default value. For real machine, after reset, MSRs contain Intel's +* default value. +*/ + sgxv->ia32_sgxlepubkeyhash[0 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH0 ; + sgxv->ia32_sgxlepubkeyhash[1 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH1 ; + sgxv->ia32_sgxlepubkeyhash[2 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH2 ; + sgxv->ia32_sgxlepubkeyhash[3 ] =SGX_INTEL_DEFAULT_LEPUBKEYHASH3 ; + + sgxv->readable =1 ; + sgxv->writable =domain_has_sgx_launch_control (v->domain);xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 476
Quoted 10 lines
xen/arch/x86/hvm/vmx/sgx.c +10
-0+ } +else + { +uint64_t v; +/* +* Although SDM says if SGX is present, then IA32_SGXLEPUBKEYHASHn are +* available for read, but in reality for SKYLAKE client machines, +* those MSRs are not available if SGX is present, so we cannot rely on +* cpu_has_sgx to determine whether to we are able to read MSRs, +* instead, we always use rdmsr_safe. Talking with Jun at XenSummit, I got the impression that the availability of these has MSRs is based on
SGX_LC, not SGX.
Quoted 3 lines
Furthermore, that is my reading of 41.2.2 "Intel SGX Launch Control Configuration", although the logic is expressed in terms of checking SGX before
SGX_LC.
Yes you are correct indeed. When I was writing the code I was reading the old SDM, which has bug and doesn't mention SGX_LC in CPUID as a condition. Please see my reply to your question whether this is erratum.
We should add cpu_has_sgx_lc as additional check, and I think we can use rdmsr if both SGX and SGX_LC are present (probably using rdmsr_safe is still better?).
Quoted 23 lines
Quoted 9 lines
xen/arch/x86/hvm/vmx/sgx.c +8
-0+*/ + sgxv->readable =rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH0 , v) ?0 :1 ; + +if ( !sgxv->readable ) +return ; + +rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH0 , sgxv->ia32_sgxlepubkeyhash[0 ]); +rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH1 , sgxv->ia32_sgxlepubkeyhash[1 ]);xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 494
Quoted 10 lines
xen/arch/x86/hvm/vmx/sgx.c +10
-0+rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH2 , sgxv->ia32_sgxlepubkeyhash[2 ]); +rdmsr_safe (MSR_IA32_SGXLEPUBKEYHASH3 , sgxv->ia32_sgxlepubkeyhash[3 ]); + } +} + +void sgx_ctxt_switch_to (struct vcpu *v) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); + +if ( sgxv->writable &&sgx_ia32_sgxlepubkeyhash_writable () )This causes a read of
FEATURE_CONTROLon every context switch path, which is inefficient.
Quoted 4 lines
Just like with CPUID policy, we will (eventually) have a generic MSR policy for the guest to use. In particular, I can forsee a usecase where hardware has LC unlocked, but the host administrator wishes LC to be locked from the guests point of view.
We can remove sgx_ia32_sgxlepubkeyhash_writable, as if sgxv->writable is true, then sgx_ia32_sgxlepubkeyhash_writable is always true. I am not sure whether we should leave guest in locked mode in most cases but I think we can add 'lewr' XL parameter to explicitly set guest to unlocked mode (otherwise guest is locked). Please see my latest reply to design of Launch Control.
Quoted 28 lines
Quoted 28 lines
xen/arch/x86/hvm/vmx/sgx.c +27
-0+ { +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH0 , sgxv->ia32_sgxlepubkeyhash[0 ]); +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH1 , sgxv->ia32_sgxlepubkeyhash[1 ]); +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH2 , sgxv->ia32_sgxlepubkeyhash[2 ]); +wrmsrl (MSR_IA32_SGXLEPUBKEYHASH3 , sgxv->ia32_sgxlepubkeyhash[3 ]); + } +} + +int sgx_msr_read_intercept (struct vcpu *v,unsigned int msr,u64 *msr_content) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); +u64 data; +int r =1 ; + +if ( !domain_has_sgx (v->domain) ) +return 0 ; + +switch ( msr ) + { +case MSR_IA32_FEATURE_CONTROL : + data = (IA32_FEATURE_CONTROL_LOCK | +IA32_FEATURE_CONTROL_SGX_ENABLE ); +/* +* If physical IA32_SGXLEPUBKEYHASHn are writable, then we always +* allow guest to be able to change IA32_SGXLEPUBKEYHASHn at runtime. +*/ + if ( sgx_ia32_sgxlepubkeyhash_writable() &&
Quoted 8 lines
xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 527
Quoted 6 lines
xen/arch/x86/hvm/vmx/sgx.c +6
-0+ domain_has_sgx_launch_control(v->domain) ) + data |=IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE ; + + *msr_content = data; + +break ;Newline here please.
Sure.
Quoted 5 lines
xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 528
Quoted 2 lines
xen/arch/x86/hvm/vmx/sgx.c +1
-0+ case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3:Spaces around ... please. (it is only because of the #defines that this isn't a syntax error).
Will do.
Quoted 37 lines
Quoted 37 lines
xen/arch/x86/hvm/vmx/sgx.c +36
-0+/* +* SDM 35.1 Model-Specific Registers, table 35-2. +* +* IA32_SGXLEPUBKEYHASH[0..3]: +* +* Read permitted if CPUID.0x12.0:EAX[0] = 1. +* +* In reality, MSRs may not be readable even SGX is present, in which +* case guest is not allowed to read either. +*/ +if ( !sgxv->readable ) + { + r =0 ; +break ; + } + + data = sgxv->ia32_sgxlepubkeyhash[msr -MSR_IA32_SGXLEPUBKEYHASH0 ]; + + *msr_content = data; + +break ; +default : + r =0 ; +break ; + } + +return r; +} + +int sgx_msr_write_intercept (struct vcpu *v,unsigned int msr,u64 msr_content) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); +int r =1 ; + +if ( !domain_has_sgx (v->domain) ) +return 0 ;
Quoted 8 lines
xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 569
Quoted 5 lines
xen/arch/x86/hvm/vmx/sgx.c +5
-0+ +switch ( msr ) + { +case MSR_IA32_FEATURE_CONTROL : +/* sliently drop */ Silently dropping is not ok. This change needs rebasing over c/s
46c3acb308where I have fixed up the writeability ofFEATURE_CONTROL.
Thanks. I'll take a look and change accordingly.
Quoted 69 lines
Quoted 60 lines
xen/arch/x86/hvm/vmx/sgx.c +32
-0+break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +/* +* SDM 35.1 Model-Specific Registers, table 35-2. +* +* IA32_SGXLEPUBKEYHASH[0..3]: +* +* - If CPUID.0x7.0:ECX[30] = 1, FEATURE_CONTROL[17] is available. +* - Write permitted if CPUID.0x12.0:EAX[0] = 1 && +* FEATURE_CONTROL[17] = 1 && FEATURE_CONTROL[0] = 1. +* +* sgxv->writable == 1 means sgx_ia32_sgxlepubkeyhash_writable() and +* domain_has_sgx_launch_control(d) both are true. +*/ +if ( !sgxv->writable ) + { + r =0 ; +break ; + } + + sgxv->ia32_sgxlepubkeyhash[msr -MSR_IA32_SGXLEPUBKEYHASH0 ] = + msr_content; + +break ; +default : + r =0 ; +break ; + } + +return r; +} +static bool_t sgx_enabled_in_bios (void ) { uint64_t val, sgx_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE |xen/arch/x86/hvm/vmx/vmx.c +5
-0@@ -470,6 +470,8 @@ static int vmx_vcpu_initialise(struct vcpu *v)if ( v->vcpu_id ==0 ) v->arch.user_regs.rax =1 ; +sgx_vcpu_init (v); +return 0 ; }@@ -1048,6 +1050,9 @@ static void vmx_ctxt_switch_to(struct vcpu *v)if ( v->domain->arch.hvm_domain.pi_ops.switch_to ) v->domain->arch.hvm_domain.pi_ops.switch_to (v); + +if (domain_has_sgx (v->domain) ) +sgx_ctxt_switch_to (v); }xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_read_intercept(unsigned int msr, uint64_t *msr_content) · line 2888
Quoted 8 lines
xen/arch/x86/hvm/vmx/vmx.c +5
-0@@ -2876,10 +2881,20 @@ static int vmx_msr_read_intercept(unsigned int msr, uint64_t *msr_content)__vmread (GUEST_IA32_DEBUGCTL , msr_content);break ;case MSR_IA32_FEATURE_CONTROL : +/* If neither SGX nor nested is supported, this MSR should not be +* touched */ +if ( !sgx_msr_read_intercept (current, msr, msr_content) && + !nvmx_msr_read_intercept (msr, msr_content) ) + goto gp_fault;
Quoted 7 lines
Unfortunately, this logic is broken. In the case that both SMX and VMX are configured, the VMX handler will clobber the values set up by the SGX handler. Sergey has a VMX-policy series (v1 posted, v2 in the works) to start addressing some of the issues on the VMX side, but fundamentally, all reads like this need serving out of a single policy, rather than having different subsystems fighting for control of the values. (The Xen MSR code is terrible for this at the moment.)
Thanks for pointing out. I have located vmx-policy series. I'll look into it to see how should I change this logic.
Quoted 14 lines
Quoted 14 lines
xen/arch/x86/hvm/vmx/vmx.c +5
-0+break ; case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_VMFUNC:if ( !nvmx_msr_read_intercept (msr, msr_content) ) goto gp_fault;break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +if ( !sgx_msr_read_intercept (current, msr, msr_content) ) + goto gp_fault; +break ;case MSR_IA32_MISC_ENABLE :rdmsrl (MSR_IA32_MISC_ENABLE , *msr_content);/* Debug Trace Store is not supported. */ @@ -3119,10 +3134,19 @@ static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content)
Quoted 8 lines
xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content) · line 3139
Quoted 6 lines
xen/arch/x86/hvm/vmx/vmx.c +3
-0break ; }case MSR_IA32_FEATURE_CONTROL : +/* See vmx_msr_read_intercept */ +if ( !sgx_msr_write_intercept (current, msr, msr_content) && + !nvmx_msr_write_intercept (msr, msr_content) )Definitely needs a rebase.
nvmx_msr_write_intercept()has been removed.
Yeah. The code base of this series is about 3-4 weeks ago unfortunately. Will do.
Thanks,
-Kai
Quoted 89 lines
~Andrew
Quoted 86 lines
xen/arch/x86/hvm/vmx/vmx.c +6
-0+ goto gp_fault; +break ; case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_TRUE_ENTRY_CTLS:if ( !nvmx_msr_write_intercept (msr, msr_content) ) goto gp_fault;break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +if ( !sgx_msr_write_intercept (current, msr, msr_content) ) + goto gp_fault; +break ; case MSR_P6_PERFCTR(0)...MSR_P6_PERFCTR(7): case MSR_P6_EVNTSEL(0)...MSR_P6_EVNTSEL(7): case MSR_CORE_PERF_FIXED_CTR0...MSR_CORE_PERF_FIXED_CTR2:xen/include/asm-x86/cpufeature.h +3
-0@@ -98,6 +98,9 @@#define cpu_has_smap boot_cpu_has(X86_FEATURE_SMAP)#define cpu_has_sha boot_cpu_has(X86_FEATURE_SHA) +/* CPUID level 0x00000007:0.ecx */ +#define cpu_has_sgx_launch_control boot_cpu_has(X86_FEATURE_SGX_LAUNCH_CONTROL) +/* CPUID level 0x80000007.edx */ #define cpu_has_itsc boot_cpu_has(X86_FEATURE_ITSC)xen/include/asm-x86/hvm/vmx/sgx.h +22
-0@@ -75,4 +75,26 @@ int hvm_populate_epc(struct domain *d, unsigned long epc_base_pfn,int hvm_reset_epc (struct domain *d,bool_t free_epc);void hvm_destroy_epc (struct domain *d); +/* Per-vcpu SGX structure */ +struct sgx_vcpu { +uint64_t ia32_sgxlepubkeyhash[4 ]; +/* +* Although SDM says if SGX is present, then IA32_SGXLEPUBKEYHASHn are +* available for read, but in reality for SKYLAKE client machines, those +* those MSRs are not available if SGX is present. +*/ +bool_t readable; +bool_t writable; +}; +#define to_sgx_vcpu (v) (&(v->arch.hvm_vmx.sgx)) + +bool_t sgx_ia32_sgxlepubkeyhash_writable (void ); +bool_t domain_has_sgx (struct domain *d); +bool_t domain_has_sgx_launch_control (struct domain *d); + +void sgx_vcpu_init (struct vcpu *v); +void sgx_ctxt_switch_to (struct vcpu *v); +int sgx_msr_read_intercept (struct vcpu *v,unsigned int msr,u64 *msr_content); +int sgx_msr_write_intercept (struct vcpu *v,unsigned int msr,u64 msr_content); +#endif /* __ASM_X86_HVM_VMX_SGX_H__ */ xen/include/asm-x86/hvm/vmx/vmcs.h +2
-0@@ -160,6 +160,8 @@ struct arch_vmx_struct {* pCPU and wakeup the related vCPU. */ struct pi_blocking_vcpu pi_blocking; + +struct sgx_vcpu sgx; };int vmx_create_vmcs (struct vcpu *v);xen/include/asm-x86/msr-index.h +6
-0@@ -296,6 +296,12 @@#define IA32_FEATURE_CONTROL_SENTER_PARAM_CTL 0x7f00#define IA32_FEATURE_CONTROL_ENABLE_SENTER 0x8000#define IA32_FEATURE_CONTROL_SGX_ENABLE 0x40000 +#define IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE 0x20000 + +#define MSR_IA32_SGXLEPUBKEYHASH0 0x0000008c +#define MSR_IA32_SGXLEPUBKEYHASH1 0x0000008d +#define MSR_IA32_SGXLEPUBKEYHASH2 0x0000008e +#define MSR_IA32_SGXLEPUBKEYHASH3 0x0000008f
Quoted 9 lines
xen/include/asm-x86/msr-index.h line 307
Quoted 2 lines
xen/include/asm-x86/msr-index.h +0
-0#define MSR_IA32_TSC_ADJUST 0x0000003b_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 03/15] xen: x86: add early stage SGX feature detection
In reply to Huang, KaiQuoted 20 lines · Kai
Quoted 17 lines · Andrew Cooper
Quoted 16 lines · Kai Huang
This patch adds early stage SGX feature detection via SGX CPUID
0x12. Functiondetect_sgxis added to detect SGX info on each CPU (called fromvmx_cpu_up). SDM says SGX info returned by CPUID is per-thread, and we cannot assume all threads will return the same SGX info, so we have to detect SGX for each CPU. For simplicity, currently SGX is only supported when all CPUs reports the same SGX info.SDM also says it's possible to have multiple EPC sections but this is only for multiple-socket server, which we don't support now (there are other things need to be done, ex, NUMA EPC, scheduling, etc, as well), so currently only one EPC is supported.
Dedicated files
sgx.candsgx.hare added (under vmx directory as SGX is Intel specific) for bulk of above SGX detection code detection code, and for further SGX code as well.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Signed-off-by Kai Huang <[email protected]>I am not sure putting this under hvm/ is a sensible move. Almost everything in this patch is currently common, and I can forsee us wanting to introduce PV support, so it would be good to introduce this in a guest-neutral location to begin with.
Sorry I forgot to response to this in my last reply. I looked at code again and yes I think we can make the code to common place. I will move current sgx.c to arch/x86/sgx.c. Thanks for comments.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
--- xen/arch/x86/hvm/vmx/Makefile | 1 + xen/arch/x86/hvm/vmx/sgx.c | 208 ++++++++++++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmcs.c | 4 + xen/include/asm-x86/cpufeature.h | 1 + xen/include/asm-x86/hvm/vmx/sgx.h | 45 +++++++++ 5 files changed, 259 insertions(+)create mode 100644
xen/arch/x86/hvm/vmx/sgx.ccreate mode 100644xen/include/asm-x86/hvm/vmx/sgx.h
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/x86/hvm/vmx/Makefile +0
-0@@ -4,3 +4,4 @@ obj-y += realmode.o obj-y += vmcs.o obj-y += vmx.o obj-y += vvmx.oQuoted 8 lines
xen/arch/x86/hvm/vmx/sgx.c
Quoted 6 lines
xen/arch/x86/hvm/vmx/Makefile +1
-0+obj-y += sgx.oxen/arch/x86/hvm/vmx/sgx.c new file +0
-0This file looks like it should be
arch/x86/sgx.c, given its current content.
Will do.
Quoted 6 lines
Quoted 6 lines
xen/arch/x86/hvm/vmx/sgx.c line 2
Quoted 4 lines
xen/arch/x86/hvm/vmx/sgx.c new file +2
-0@@ -0,0 +1,208 @@ +/* +* Intel Software Guard Extensions support Please include a GPLv2 header.
Yes will do.
Thanks,
-Kai
Quoted 48 lines
Quoted 18 lines
xen/arch/x86/hvm/vmx/sgx.c line 14
Quoted 13 lines
xen/arch/x86/hvm/vmx/sgx.c new file +12
-0+* +* Author: Kai Huang <[email protected]> +*/ + +#include <asm/cpufeature.h> +#include <asm/msr-index.h> +#include <asm/msr.h> +#include <asm/hvm/vmx/sgx.h> +#include <asm/hvm/vmx/vmcs.h> + +static struct sgx_cpuinfo __read_mostly sgx_cpudata[NR_CPUS ]; +static struct sgx_cpuinfo __read_mostly boot_sgx_cpudata;I don't think any of this is necessary. The description says that all EPCs across the server will be reported in CPUID subleaves, and our implementation gives up if the data are non-identical across CPUs.
Quoted 2 lines
Therefore, we only need to keep one copy of the data, and check check APs against the master copy.
Right.
boot_sgx_cpudatais what we need. Currentlydetect_sgxis called fromvmx_cpu_up. How about changing to calling it fromidentify_cpu, and something like below ?if ( c == &boot_cpu_data ) detect_sgx(&boot_sgx_cpudata); else { struct sgx_cpuinfo tmp; detect_sgx(&tmp); if ( memcmp(&boot_sgx_cpudata, &tmp, sizeof (tmp)) ) //disable SGX }Thanks,
-KaiQuoted 6 lines
Let me see about splitting up a few bits of the existing CPUID infrastructure, so we can use the host cpuid policy more effectively for Xen related things.
Quoted 7 lines
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 4 lines
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Re: [PATCH 09/15] xen: vmx: handle SGX related MSRs
In reply to Huang, KaiQuoted 189 lines · Kai
Quoted 90 lines · Andrew Cooper
Quoted 75 lines · Kai Huang
This patch handles
IA32_FEATURE_CONTROLandIA32_SGXLEPUBKEYHASHnMSRs.For
IA32_FEATURE_CONTROL, if SGX is exposed to domain, thenSGX_ENABLEbit is always set. If SGX launch control is also exposed to domain, and physicalIA32_SGXLEPUBKEYHASHnare writable, thenSGX_LAUNCH_CONTROL_ENABLEbit is also always set. Write toIA32_FEATURE_CONTROLis ignored.For
IA32_SGXLEPUBKEYHASHn, a new 'structsgx_vcpu' is added for per-vcpu SGX staff, and currently it has vcpu's virtual ia32_sgxlepubkeyhash[0-3]. Two boolean 'readable' and 'writable' are also added to indicate whether virtualIA32_SGXLEPUBKEYHASHnare readable and writable.During vcpu is initialized, virtual
ia32_sgxlepubkeyhashare also initialized. If physicalIA32_SGXLEPUBKEYHASHnare writable, thenia32_sgxlepubkeyhashare set to Intel's default value, as for physical machine, those MSRs will have Intel's default value. If physical MSRs are not writable (it is *locked* by BIOS before handling to Xen), then we try to read those MSRs and use physical values as defult value for virtual MSRs. One thing isrdmsr_safeis used, as although SDM says if SGX is present,IA32_SGXLEPUBKEYHASHnare available for read, but in reality, skylake client (at least some, depending on BIOS) doesn't have those MSRs available, so we userdmsr_safeand set readable to false if it returns error code.For
IA32_SGXLEPUBKEYHASHnMSR read from guest, if physical MSRs are not readable, guest is not allowed to read either, otherwise vcpu's virtual MSR value is returned.For
IA32_SGXLEPUBKEYHASHnMSR write from guest, we allow guest to write if both physical MSRs are writable and SGX launch control is exposed to domain, otherwise error is injected.To make EINIT run successfully in guest, vcpu's virtual
IA32_SGXLEPUBKEYHASHnwill be update to physical MSRs when vcpu is scheduled in.Signed-off-by Kai Huang <[email protected]>--- xen/arch/x86/hvm/vmx/sgx.c | 194 +++++++++++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmx.c | 24 +++++ xen/include/asm-x86/cpufeature.h | 3 + xen/include/asm-x86/hvm/vmx/sgx.h | 22 +++++ xen/include/asm-x86/hvm/vmx/vmcs.h | 2 + xen/include/asm-x86/msr-index.h | 6 ++ 6 files changed, 251 insertions(+)xen/arch/x86/hvm/vmx/sgx.c +5
-0@@ -405,6 +405,200 @@ void hvm_destroy_epc(struct domain *d)hvm_reset_epc (d, true); } +/* Whether IA32_SGXLEPUBKEYHASHn are physically *unlocked* by BIOS */ +bool_t sgx_ia32_sgxlepubkeyhash_writable (void ) +{ + uint64_t sgx_lc_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE | +
IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE|xen/arch/x86/hvm/vmx/sgx.c +17
-0+IA32_FEATURE_CONTROL_LOCK ; +uint64_t val; + +rdmsrl (MSR_IA32_FEATURE_CONTROL , val); + +return (val & sgx_lc_enabled) == sgx_lc_enabled; +} + +bool_t domain_has_sgx (struct domain *d) +{ +/* hvm_epc_populated(d) implies CPUID has SGX */ +return hvm_epc_populated (d); +} + +bool_t domain_has_sgx_launch_control (struct domain *d) +{ +struct cpuid_policy *p = d->arch.cpuid;xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 439
Quoted 10 lines
xen/arch/x86/hvm/vmx/sgx.c +10
-0+ +if ( !domain_has_sgx (d) ) +return false; + +/* Unnecessary but check anyway */ +if ( !cpu_has_sgx_launch_control ) +return false; + +return !!p->feat.sgx_launch_control; +}Both of these should be d->arch.cpuid->feat.{sgx,sgx_lc} only, and not from having individual helpers.
Quoted 5 lines
The CPUID setup during host boot and domain construction should take care of setting everything up properly, or hiding the features from the guest. The point of the work I've been doing is to prevent situations where the guest can see SGX but something doesn't work because of Xen using nested checks like this.
Thanks for comments. Will change to simple check against d->arch.cpuid->feat.{sgx,sgx_lc}.
Quoted 52 lines
Quoted 40 lines
xen/arch/x86/hvm/vmx/sgx.c +17
-0+ +/* Digest of Intel signing key. MSR's default value after reset. */ +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH0 0xa6053e051270b7ac +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH1 0x6cfbe8ba8b3b413d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH2 0xc4916d99f2b3735d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH3 0xd4f8c05909f9bb3b + +void sgx_vcpu_init (struct vcpu *v) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); + +memset (sgxv,0 ,sizeof (*sgxv)); + +if (sgx_ia32_sgxlepubkeyhash_writable () ) + { +/* +* If physical MSRs are writable, set vcpu's default value to Intel's
+ * default value. For real machine, after reset, MSRs
contain Intel'sxen/arch/x86/hvm/vmx/sgx.c +3
-0+* default value. +*/ + sgxv->ia32_sgxlepubkeyhash[0] =
SGX_INTEL_DEFAULT_LEPUBKEYHASH0;
+ sgxv->ia32_sgxlepubkeyhash[1] =SGX_INTEL_DEFAULT_LEPUBKEYHASH1;
+ sgxv->ia32_sgxlepubkeyhash[2] =SGX_INTEL_DEFAULT_LEPUBKEYHASH2;
+ sgxv->ia32_sgxlepubkeyhash[3] =SGX_INTEL_DEFAULT_LEPUBKEYHASH3;xen/arch/x86/hvm/vmx/sgx.c +9
-0+ + sgxv->readable =1 ; + sgxv->writable =domain_has_sgx_launch_control (v->domain); + } +else + { +uint64_t v; +/* +* Although SDM says if SGX is present, then Quoted 8 lines
IA32_SGXLEPUBKEYHASHnare
+ * available for read, but in reality for SKYLAKE client
machines,
+ * those MSRs are not available if SGX is present, so we
cannot rely on
+ *cpu_has_sgxto determine whether to we are able to read
MSRs,
+ * instead, we always userdmsr_safe.Talking with Jun at XenSummit, I got the impression that the availability of these has MSRs is based on
SGX_LC, not SGX.Quoted 3 lines
Furthermore, that is my reading of 41.2.2 "Intel SGX Launch Control Configuration", although the logic is expressed in terms of checking SGX before
SGX_LC.Yes you are correct indeed. When I was writing the code I was reading the old SDM, which has bug and doesn't mention
SGX_LCin CPUID as a condition. Please see my reply to your question whether this is erratum.We should add
cpu_has_sgx_lcas additional check, and I think we can use rdmsr if both SGX andSGX_LCare present (probably usingrdmsr_safeis still better?).Quoted 28 lines
Quoted 16 lines
xen/arch/x86/hvm/vmx/sgx.c +2
-0+*/ + sgxv->readable = rdmsr_safe(MSR_IA32_SGXLEPUBKEYHASH0, v) ?0 : 1;
xen/arch/x86/hvm/vmx/sgx.c +5
-0+ +if ( !sgxv->readable ) +return ; + + rdmsr_safe(MSR_IA32_SGXLEPUBKEYHASH0,sgxv->ia32_sgxlepubkeyhash[0]);
+ rdmsr_safe(MSR_IA32_SGXLEPUBKEYHASH1,
sgxv->ia32_sgxlepubkeyhash[1]);
+ rdmsr_safe(MSR_IA32_SGXLEPUBKEYHASH2,
sgxv->ia32_sgxlepubkeyhash[2]);
+ rdmsr_safe(MSR_IA32_SGXLEPUBKEYHASH3,
sgxv->ia32_sgxlepubkeyhash[3]);xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 494
Quoted 8 lines
xen/arch/x86/hvm/vmx/sgx.c +8
-0+ } +} + +void sgx_ctxt_switch_to (struct vcpu *v) +{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); + +if ( sgxv->writable &&sgx_ia32_sgxlepubkeyhash_writable () )This causes a read of
FEATURE_CONTROLon every context switch path, which is inefficient.
Quoted 6 lines
Quoted 4 lines
Just like with CPUID policy, we will (eventually) have a generic MSR policy for the guest to use. In particular, I can forsee a usecase where hardware has LC unlocked, but the host administrator wishes LC to be locked from the guests point of view.
We can remove
sgx_ia32_sgxlepubkeyhash_writable, as ifsgxv->writableis true, thensgx_ia32_sgxlepubkeyhash_writableis always true. I am not sure whether we should leave guest in locked mode in most cases but I think we can add 'lewr' XL parameter to explicitly set guest to unlocked mode (otherwise guest is locked). Please see my latest reply to design of Launch Control.
Hi Andrew,
I'd like to add something regarding to performance optimization of updating the MSRs. There are two optimizations that we can do:
- We can add per_cpu variable for the MSRs, and keep the per_vcpu variable always being equal to physical MSRs, then we only need to update the MSRs when the value we want to update is not equal to the value of per_vcpu variable. This can reduce some physical MSR write.
- Thanks to current SGX implementation, IA32_SGXLEPUBKEYHASHn MSRs are only used by EINIT. Once EINIT is done, EGETKEY won't depend on IA32_SGXLEPUBKEYHASHn. So we can also trap guest's EINIT, and update MSRs in EINIT VMEXIT. However if we trap EINIT, Xen needs to run EINIT on behalf of guest, meaning Xen needs to remap guest's EINIT parameters (we probably don't need to reconstruct EINIT parameters in Xen as both SIGSTRUCT and EINITTOKEN don't contain guest's virtual address internally that needs to be remapped by Xen), run EINIT, and emulate EINIT return value to guest.
The first optimization is pretty straightforward, and I think we should do it (and I'll do in next version). The second optimization requires trapping EINIT from guest (thus more complicated implementation) but can eliminate unnecessary MSR updates during context switch. Do you think we should do both the optimizations?
Thanks,
-Kai
Quoted 312 lines
Quoted 36 lines
Quoted 36 lines
xen/arch/x86/hvm/vmx/sgx.c +2
-0+ { + wrmsrl(MSR_IA32_SGXLEPUBKEYHASH0,sgxv->ia32_sgxlepubkeyhash[0]);
+ wrmsrl(MSR_IA32_SGXLEPUBKEYHASH1,
sgxv->ia32_sgxlepubkeyhash[1]);
+ wrmsrl(MSR_IA32_SGXLEPUBKEYHASH2,
sgxv->ia32_sgxlepubkeyhash[2]);
+ wrmsrl(MSR_IA32_SGXLEPUBKEYHASH3,
sgxv->ia32_sgxlepubkeyhash[3]);xen/arch/x86/hvm/vmx/sgx.c +4
-0+ } +} + +int sgx_msr_read_intercept(struct vcpu *v, unsigned int msr, u64
*msr_content)xen/arch/x86/hvm/vmx/sgx.c +15
-0+{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); +u64 data; +int r =1 ; + +if ( !domain_has_sgx (v->domain) ) +return 0 ; + +switch ( msr ) + { +case MSR_IA32_FEATURE_CONTROL : + data = (IA32_FEATURE_CONTROL_LOCK | +IA32_FEATURE_CONTROL_SGX_ENABLE ); +/* +* If physical IA32_SGXLEPUBKEYHASHn are writable, then we always
+ * allow guest to be able to changeIA32_SGXLEPUBKEYHASHnat
runtime.xen/arch/x86/hvm/vmx/sgx.c +2
-0+*/ + if ( sgx_ia32_sgxlepubkeyhash_writable() &&Quoted 8 lines
xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 527
Quoted 6 lines
xen/arch/x86/hvm/vmx/sgx.c +6
-0+ domain_has_sgx_launch_control(v->domain) ) + data |=IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE ; + + *msr_content = data; + +break ;Newline here please.
Sure.
Quoted 5 lines
Quoted 2 lines
+ case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3:
Spaces around ... please. (it is only because of the #defines that this isn't a syntax error).
Will do.
Quoted 40 lines
Quoted 40 lines
xen/arch/x86/hvm/vmx/sgx.c +8
-0+/* +* SDM 35.1 Model-Specific Registers, table 35-2. +* +* IA32_SGXLEPUBKEYHASH[0..3]: +* +* Read permitted if CPUID.0x12.0:EAX[0] = 1. +* +* In reality, MSRs may not be readable even SGX is present, in which
xen/arch/x86/hvm/vmx/sgx.c +9
-0+* case guest is not allowed to read either. +*/ +if ( !sgxv->readable ) + { + r =0 ; +break ; + } + + data = sgxv->ia32_sgxlepubkeyhash[msr -
MSR_IA32_SGXLEPUBKEYHASH0];xen/arch/x86/hvm/vmx/sgx.c +13
-0+ + *msr_content = data; + +break ; +default : + r =0 ; +break ; + } + +return r; +} + +int sgx_msr_write_intercept(struct vcpu *v, unsigned int msr, u64
msr_content)xen/arch/x86/hvm/vmx/sgx.c +6
-0+{ +struct sgx_vcpu *sgxv =to_sgx_vcpu (v); +int r =1 ; + +if ( !domain_has_sgx (v->domain) ) +return 0 ;Quoted 8 lines
xen/arch/x86/hvm/vmx/sgx.c void hvm_destroy_epc(struct domain *d) · line 569
Quoted 5 lines
xen/arch/x86/hvm/vmx/sgx.c +5
-0+ +switch ( msr ) + { +case MSR_IA32_FEATURE_CONTROL : +/* sliently drop */ Silently dropping is not ok. This change needs rebasing over c/s
46c3acb308where I have fixed up the writeability ofFEATURE_CONTROL.Thanks. I'll take a look and change accordingly.
Quoted 68 lines
Quoted 63 lines
xen/arch/x86/hvm/vmx/sgx.c +8
-0+break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +/* +* SDM 35.1 Model-Specific Registers, table 35-2. +* +* IA32_SGXLEPUBKEYHASH[0..3]: +* +* - If CPUID.0x7.0:ECX[30] = 1, FEATURE_CONTROL[17] is available.
xen/arch/x86/hvm/vmx/sgx.c +4
-0+* - Write permitted if CPUID.0x12.0:EAX[0] = 1 && +* FEATURE_CONTROL[17] = 1 && FEATURE_CONTROL[0] = 1. +* +* sgxv->writable == 1 means
sgx_ia32_sgxlepubkeyhash_writable()andxen/arch/x86/hvm/vmx/sgx.c +20
-0+* domain_has_sgx_launch_control(d) both are true. +*/ +if ( !sgxv->writable ) + { + r =0 ; +break ; + } + + sgxv->ia32_sgxlepubkeyhash[msr -MSR_IA32_SGXLEPUBKEYHASH0 ] = + msr_content; + +break ; +default : + r =0 ; +break ; + } + +return r; +} +static bool_t sgx_enabled_in_bios (void ) { uint64_t val, sgx_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE |xen/arch/x86/hvm/vmx/vmx.c +6
-0@@ -470,6 +470,8 @@ static int vmx_vcpu_initialise(struct vcpu *v)if ( v->vcpu_id ==0 ) v->arch.user_regs.rax =1 ; +sgx_vcpu_init (v); +return 0 ; }@@ -1048,6 +1050,9 @@ static void vmx_ctxt_switch_to(struct vcpu *v)if ( v->domain->arch.hvm_domain.pi_ops.switch_to ) v->domain->arch.hvm_domain.pi_ops.switch_to (v); + +if (domain_has_sgx (v->domain) ) +sgx_ctxt_switch_to (v); }@@ -2876,10 +2881,20 @@ static int vmx_msr_read_intercept(unsigned int msr, uint64_t *msr_content)__vmread (GUEST_IA32_DEBUGCTL ,msr_content );break ;case MSR_IA32_FEATURE_CONTROL : +/* If neither SGX nor nested is supported, this MSR should not be
xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_read_intercept(unsigned int msr, uint64_t *msr_content) · line 2888
Quoted 4 lines
xen/arch/x86/hvm/vmx/vmx.c +4
-0+* touched */ +if ( !sgx_msr_read_intercept (current, msr, msr_content) && + !nvmx_msr_read_intercept (msr, msr_content) ) + goto gp_fault;Quoted 7 lines
Unfortunately, this logic is broken. In the case that both SMX and VMX are configured, the VMX handler will clobber the values set up by the SGX handler. Sergey has a VMX-policy series (v1 posted, v2 in the works) to start addressing some of the issues on the VMX side, but fundamentally, all reads like this need serving out of a single policy, rather than having different subsystems fighting for control of the values. (The Xen MSR code is terrible for this at the moment.)
Thanks for pointing out. I have located vmx-policy series. I'll look into it to see how should I change this logic.
Quoted 15 lines
Quoted 15 lines
xen/arch/x86/hvm/vmx/vmx.c +5
-0+break ; case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_VMFUNC:if ( !nvmx_msr_read_intercept (msr, msr_content) ) goto gp_fault;break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +if ( !sgx_msr_read_intercept (current, msr, msr_content) ) + goto gp_fault; +break ;case MSR_IA32_MISC_ENABLE :rdmsrl (MSR_IA32_MISC_ENABLE , *msr_content);/* Debug Trace Store is not supported. */ @@ -3119,10 +3134,19 @@ static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content)Quoted 8 lines
xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content) · line 3139
Quoted 6 lines
xen/arch/x86/hvm/vmx/vmx.c +3
-0break ; }case MSR_IA32_FEATURE_CONTROL : +/* See vmx_msr_read_intercept */ +if ( !sgx_msr_write_intercept (current, msr, msr_content) && + !nvmx_msr_write_intercept (msr, msr_content) )Definitely needs a rebase.
nvmx_msr_write_intercept()has been removed.Yeah. The code base of this series is about 3-4 weeks ago unfortunately. Will do.
Thanks,
-KaiQuoted 89 lines
~Andrew
Quoted 86 lines
xen/arch/x86/hvm/vmx/vmx.c +6
-0+ goto gp_fault; +break ; case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_TRUE_ENTRY_CTLS:if ( !nvmx_msr_write_intercept (msr, msr_content) ) goto gp_fault;break ; + case MSR_IA32_SGXLEPUBKEYHASH0...MSR_IA32_SGXLEPUBKEYHASH3: +if ( !sgx_msr_write_intercept (current, msr, msr_content) ) + goto gp_fault; +break ; case MSR_P6_PERFCTR(0)...MSR_P6_PERFCTR(7): case MSR_P6_EVNTSEL(0)...MSR_P6_EVNTSEL(7):case MSR_CORE_PERF_FIXED_CTR0...MSR_CORE_PERF_FIXED_CTR2:
xen/include/asm-x86/cpufeature.h +3
-0@@ -98,6 +98,9 @@#define cpu_has_smap boot_cpu_has(X86_FEATURE_SMAP)#define cpu_has_sha boot_cpu_has(X86_FEATURE_SHA) +/* CPUID level 0x00000007:0.ecx */ +#define cpu_has_sgx_launch_control boot_cpu_has(X86_FEATURE_SGX_LAUNCH_CONTROL) +/* CPUID level 0x80000007.edx */ #define cpu_has_itsc boot_cpu_has(X86_FEATURE_ITSC)xen/include/asm-x86/hvm/vmx/sgx.h +5
-0@@ -75,4 +75,26 @@ int hvm_populate_epc(struct domain *d, unsignedlong epc_base_pfn, int hvm_reset_epc(struct domain *d, bool_t free_epc);void hvm_destroy_epc (struct domain *d); +/* Per-vcpu SGX structure */ +struct sgx_vcpu { +uint64_t ia32_sgxlepubkeyhash[4 ]; +/* +* Although SDM says if SGX is present, then
IA32_SGXLEPUBKEYHASHnare
+ * available for read, but in reality for SKYLAKE client
machines, thosexen/include/asm-x86/hvm/vmx/sgx.h +14
-0+* those MSRs are not available if SGX is present. +*/ +bool_t readable; +bool_t writable; +}; +#define to_sgx_vcpu (v) (&(v->arch.hvm_vmx.sgx)) + +bool_t sgx_ia32_sgxlepubkeyhash_writable (void ); +bool_t domain_has_sgx (struct domain *d); +bool_t domain_has_sgx_launch_control (struct domain *d); + +void sgx_vcpu_init (struct vcpu *v); +void sgx_ctxt_switch_to (struct vcpu *v); +int sgx_msr_read_intercept(struct vcpu *v, unsigned int msr, u64
*msr_content);
+intsgx_msr_write_intercept(struct vcpu *v, unsigned int msr, u64 msr_content);xen/include/asm-x86/hvm/vmx/sgx.h +1
-0+#endif /* __ASM_X86_HVM_VMX_SGX_H__ */ xen/include/asm-x86/hvm/vmx/vmcs.h +2
-0@@ -160,6 +160,8 @@ struct arch_vmx_struct {* pCPU and wakeup the related vCPU. */ struct pi_blocking_vcpu pi_blocking; + +struct sgx_vcpu sgx; };int vmx_create_vmcs (struct vcpu *v);xen/include/asm-x86/msr-index.h +5
-0@@ -296,6 +296,12 @@#define IA32_FEATURE_CONTROL_SENTER_PARAM_CTL 0x7f00#define IA32_FEATURE_CONTROL_ENABLE_SENTER 0x8000#define IA32_FEATURE_CONTROL_SGX_ENABLE 0x40000 +#define IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE 0x20000 + +#define MSR_IA32_SGXLEPUBKEYHASH0 0x0000008c +#define MSR_IA32_SGXLEPUBKEYHASH1 0x0000008d +#define MSR_IA32_SGXLEPUBKEYHASH2 0x0000008eQuoted 9 lines
xen/include/asm-x86/msr-index.h line 306
Quoted 2 lines
xen/include/asm-x86/msr-index.h +1
-0+#define MSR_IA32_SGXLEPUBKEYHASH3 0x0000008f#define MSR_IA32_TSC_ADJUST 0x0000003b_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 4 lines
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 62 lines · Kai
Hi Wei,
Thank you very much for comments. Please see my reply below.
Quoted 49 lines · Wei Liu
Hi Kai
Thanks for this nice write-up.
Some comments and questions below.
Quoted 2 lines · Kai Huang
Hi all,
[...]
Quoted 39 lines
2. SGX Virtualization Design
2.1 High Level Toolstack Changes:
2.1.1 New 'epc' parameter
EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin
alao should be able to get all domain's virtual EPC size.For this purpose, a new 'epc = <size>' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted.
2.1.2 New XL commands (?)
Administrator should be able to get physical EPC size, and all domain's virtual EPC size. For this purpose, we can introduce 2 additional commands:
# xl sgxinfoWhich will print out physical EPC size, and other SGX info (such as SGX1, SGX2, etc) if necessary.
# xl sgxlist <did>Which will print out particular domain's virtual EPC size, or list all virtual EPC sizes for all supported domains.
Alternatively, we can also extend existing XL commands by adding new option
# xl info -sgxWhich will print out physical EPC size along with other physinfo. And
# xl list <did> -sgxQuoted 7 lines
Quoted 4 lines
Which will print out domain's virtual EPC size.
Comments?
Can a guest have multiple EPC? If so, the proposed parameter is not good enough.
Quoted 8 lines
According to SDM a machine may have multiple EPC, but it may have doesn't mean it must have. EPC is typically reserved by BIOS as Processor Reserved Memory (PRM), and in my understanding, client machine doesn't need to have multiple EPC. Currently, I don't see why we need to expose multiple EPC to guest. Even physical machine reports multiple EPC, exposing one EPC to guest is enough. Currently SGX should not be supported with virtual NUMA simultaneously for a single domain.
When you say "is enough", do you mean Intel doesn't recommend users to use more than one? I don't think from reading this doc precludes using more then one technically.
Quoted 24 lines
Quoted 3 lines
Can a guest with EPC enabled be migrated? The answer to this question can lead to multiple other questions.
See the last section of my design. I saw you've already seen it. :)
Quoted 3 lines
Another question, is EPC going to be backed by normal memory? This is related to memory accounting of the guest.
Although SDM says typically EPC is allocated by BIOS as PRM, but I think we can just treat EPC as PRM, so I believe yes, physically EPC is backed by normal memory. But EPC is reported as reserved memory in e820 table.
Quoted 3 lines
Is EPC going to be modeled as a device or another type of memory? This is related to how we manage it in the toolstack.
I think we'd better to treat EPC as another type of memory. I am not sure whether it should be modeled as device, as on real machine, EPC is also exposed in ACPI table via "INT0E0C" device under \_SB (however it is not modeled as PCIE device for sure).
Quoted 1 line
Quoted 5 lines
Quoted 1 line
Finally why do you not allow the users to specify the base address?
I don't see any reason why user needs to specify base address. If we do, then specify what address? On real machine, BIOS set the base address, and for VM, I think toolstack/Xen should do this.
We can expose an option for user to control that if they want to and at the same time provide the logic to calculate the base address internally. I'm not sure if that's going to be very useful, but I'm not convinced it is entirely useless either.
Thinking a bit more we can always extend the syntax and API to support that if need be, so I'm fine with not providing such mechanism at early stage.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Hi Wei,
Thanks for your comments. Please see my reply below.
Quoted 72 lines · Wei Liu
Quoted 62 lines · Kai
Hi Wei,
Thank you very much for comments. Please see my reply below.
Quoted 49 lines · Wei Liu
Hi Kai
Thanks for this nice write-up.
Some comments and questions below.
Quoted 2 lines · Kai Huang
Hi all,
[...]
Quoted 39 lines
2. SGX Virtualization Design
2.1 High Level Toolstack Changes:
2.1.1 New 'epc' parameter
EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin
alao should be able to get all domain's virtual EPC size.For this purpose, a new 'epc = <size>' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted.
2.1.2 New XL commands (?)
Administrator should be able to get physical EPC size, and all domain's virtual EPC size. For this purpose, we can introduce 2 additional commands:
# xl sgxinfoWhich will print out physical EPC size, and other SGX info (such as SGX1, SGX2, etc) if necessary.
# xl sgxlist <did>Which will print out particular domain's virtual EPC size, or list all virtual EPC sizes for all supported domains.
Alternatively, we can also extend existing XL commands by adding new option
# xl info -sgxWhich will print out physical EPC size along with other physinfo. And
# xl list <did> -sgxQuoted 7 lines
Quoted 4 lines
Which will print out domain's virtual EPC size.
Comments?
Can a guest have multiple EPC? If so, the proposed parameter is not good enough.
Quoted 8 lines
According to SDM a machine may have multiple EPC, but it may have doesn't mean it must have. EPC is typically reserved by BIOS as Processor Reserved Memory (PRM), and in my understanding, client machine doesn't need to have multiple EPC. Currently, I don't see why we need to expose multiple EPC to guest. Even physical machine reports multiple EPC, exposing one EPC to guest is enough. Currently SGX should not be supported with virtual NUMA simultaneously for a single domain.
Quoted 3 lines
When you say "is enough", do you mean Intel doesn't recommend users to use more than one? I don't think from reading this doc precludes using more then one technically.
No I don't think Intel would make such recommendation. For real hardware yes it's possible there are multiple EPC sections, but for client or single socket server machine, typically there will be only one EPC. In case of VM, I don't see there's any benefit of exposing multiple EPCs to guest, except the vNUMA case. My thinking is although SDM doesn't preclude using more than one EPC but for VM there's no need to use more than one.
Quoted 31 lines
Quoted 25 lines
Quoted 4 lines
Can a guest with EPC enabled be migrated? The answer to this question can lead to multiple other questions.
See the last section of my design. I saw you've already seen it. :)
Quoted 3 lines
Another question, is EPC going to be backed by normal memory? This is related to memory accounting of the guest.
Although SDM says typically EPC is allocated by BIOS as PRM, but I think we can just treat EPC as PRM, so I believe yes, physically EPC is backed by normal memory. But EPC is reported as reserved memory in e820 table.
Quoted 3 lines
Is EPC going to be modeled as a device or another type of memory? This is related to how we manage it in the toolstack.
I think we'd better to treat EPC as another type of memory. I am not sure whether it should be modeled as device, as on real machine, EPC is also exposed in ACPI table via "INT0E0C" device under \_SB (however it is not modeled as PCIE device for sure).
Quoted 1 line
Quoted 5 lines
Quoted 1 line
Finally why do you not allow the users to specify the base address?
I don't see any reason why user needs to specify base address. If we do, then specify what address? On real machine, BIOS set the base address, and for VM, I think toolstack/Xen should do this.
Quoted 8 lines
We can expose an option for user to control that if they want to and at the same time provide the logic to calculate the base address internally. I'm not sure if that's going to be very useful, but I'm not convinced it is entirely useless either.
Thinking a bit more we can always extend the syntax and API to support that if need be, so I'm fine with not providing such mechanism at early stage.
Yeah I think we can extend if needed in the future. Thanks Wei.
Thanks,
-Kai
Quoted 1 line
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel