EPT Emulation
cover EPT Emulation
This patch set is part-6 of this RFC patches. It introduces EPT emulation for pKVM on Intel platform.
Add EPT emulation through shadowing vEPT in host VM.
Host VM launches its guest, and manage such guest's memory through a EPT page table maintained in host KVM. Meanwhile this EPT page table is untrusted to pKVM, so pKVM shall not directly use this EPT as guest's active EPT. To ensure isolating of guest memory for protected VM, pKVM hypervisor shadows such guest's EPT in host KVM, to build out active EPT page table after necessary check (the check is based on page state management which will be introduced later). It's actually an emulation for guest EPT page table, the guest EPT page table in host KVM is called "virtual EPT", while the active EPT page table in pKVM is called "shadow EPT".
Shadow EPT in pKVM is initialized with empty mapping. Guest EPT violation leads to the EPT shadowing. The EPT shadowing first walks virtual EPT in host VM to find out the virtual mapping, then setup the same mapping in shadow EPT - this will be updated in the future to be managed by page state.
And for invept emulation, just simply free all the mapping of shadow EPT now.
pKVM on ARM is using PV ops to directly manage stage-2 MMU page table in the hypervisor, while pKVM on Intel platform choose above EPT emulation solution - it increases the complexity by doing EPT shadowing but avoids changes in the KVM MMU code.
Chuanxiao Dong (12):
pkvm: x86: Pre-define the maximum number of supported VMs
pkvm: x86: init: Reserve memory for shadow EPT
pkvm: x86: Initialize the shadow EPT pool
pkvm: x86: Introduce shadow EPT
pkvm: x86: Introduce vEPT to record guest EPT information
pkvm: x86: Add API to get the max phys address bits
pkvm: x86: Initialize ept_zero_check
pkvm: x86: Add support for pKVM to handle the nested EPT violation
pkvm: x86: Introduce PKVM_ASSERT
pkvm: x86: add pkvm_pgtable_unmap_safe for a safe unmap
pkvm: x86: Add INVEPT instruction emulation
pkvm: x86: Switch to use shadow EPT for nested guests
Jason Chen CJ (1):
pkvm: x86: Introduce shadow EPT invalidation support
arch/x86/include/asm/kvm_pkvm.h | 34 ++++ arch/x86/kvm/vmx/pkvm/hyp/bug.h | 23 +++ arch/x86/kvm/vmx/pkvm/hyp/ept.c | 238 +++++++++++++++++++++- arch/x86/kvm/vmx/pkvm/hyp/ept.h | 24 +++ arch/x86/kvm/vmx/pkvm/hyp/init_finalise.c | 13 ++ arch/x86/kvm/vmx/pkvm/hyp/memory.c | 27 +++ arch/x86/kvm/vmx/pkvm/hyp/memory.h | 3 + arch/x86/kvm/vmx/pkvm/hyp/mmu.c | 4 +- arch/x86/kvm/vmx/pkvm/hyp/mmu.h | 3 +- arch/x86/kvm/vmx/pkvm/hyp/nested.c | 191 +++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/nested.h | 1 + arch/x86/kvm/vmx/pkvm/hyp/pgtable.c | 67 ++++-- arch/x86/kvm/vmx/pkvm/hyp/pgtable.h | 4 +- arch/x86/kvm/vmx/pkvm/hyp/pkvm.c | 36 +++- arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h | 32 +++ arch/x86/kvm/vmx/pkvm/hyp/vmexit.c | 3 + arch/x86/kvm/vmx/pkvm/hyp/vmx.h | 31 +++ arch/x86/kvm/vmx/pkvm/pkvm_constants.c | 3 +- arch/x86/kvm/vmx/pkvm/pkvm_host.c | 3 + 19 files changed, 711 insertions(+), 29 deletions(-) create mode 100644 arch/x86/kvm/vmx/pkvm/hyp/bug.h
Signature · 1 line
2.25.1
RFC PATCH 1/13 pkvm: x86: Pre-define the maximum number of supported VMs
From: Chuanxiao Dong <[email protected]>
Define the maximum number of the VMs pKVM can support. pKVM needs reserve related memory based on the VM number it support (e.g., shadow EPT page table pages). The reserved memory size for pKVM is calculated then exposed to pkvm_init through pkvm_constants definition PKVM_MAX_VM_NUM.
--- arch/x86/kvm/vmx/pkvm/hyp/pkvm.c | 2 +- arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h | 3 +++ arch/x86/kvm/vmx/pkvm/pkvm_constants.c | 1 + 3 files changed, 5 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/pkvm/hyp/pkvm.c +1 -1
@@ -10,7 +10,7 @@struct pkvm_hyp *pkvm_hyp; -#define MAX_SHADOW_VMS 255 +#define MAX_SHADOW_VMS (PKVM_MAX_NORMAL_VM_NUM + PKVM_MAX_PROTECTED_VM_NUM)#define HANDLE_OFFSET 1#define to_shadow_vm_handle (vcpu_handle) ((s64)(vcpu_handle) >> SHADOW_VM_HANDLE_SHIFT)
arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h +3 -0
@@ -7,6 +7,9 @@#include <asm/pkvm_spinlock.h> +#define PKVM_MAX_NORMAL_VM_NUM 8 +#define PKVM_MAX_PROTECTED_VM_NUM 2 +/* * A container for the vcpu state that hyp needs to maintain for protected VMs. */
arch/x86/kvm/vmx/pkvm/pkvm_constants.c +1 -0
@@ -15,5 +15,6 @@ int main(void)DEFINE (PKVM_VMEMMAP_ENTRY_SIZE ,sizeof (struct hyp_page ));DEFINE (PKVM_SHADOW_VM_SIZE ,sizeof (struct pkvm_shadow_vm ) +pkvm_shadow_vcpu_array_size ());DEFINE (PKVM_SHADOW_VCPU_STATE_SIZE ,sizeof (struct shadow_vcpu_state )); +DEFINE (PKVM_MAX_VM_NUM ,PKVM_MAX_NORMAL_VM_NUM +PKVM_MAX_PROTECTED_VM_NUM );return 0 ; }
Signature · 1 line
2.25.1
RFC PATCH 2/13 pkvm: x86: init: Reserve memory for shadow EPT
In reply to Jason Chen CJFrom: Chuanxiao Dong <[email protected]>
Shadow EPT pages are from the reserved memory of pKVM. The size for Shadow EPT pages is calculated based on the number of supported VMs (both normal VM and protected VM), with several assumptions:
- there is no shared memory between each VMs (normal or protected).
- all VMs total memory size is not larger than the platform memory size.
- the virtual MMIO range for each VM is not larger than 1G.
which makes the reserved memory size is calculable and predictable.
--- arch/x86/include/asm/kvm_pkvm.h | 34 +++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/pkvm_host.c | 1 + 2 files changed, 35 insertions(+)
arch/x86/include/asm/kvm_pkvm.h +34 -0
@@ -131,6 +131,40 @@ static inline int hyp_pre_reserve_check(void)return 0 ; } +/* +* Calculate the total pages for shadow EPT with assumptions: +* 1. there is no shared memory between each VMs (normal or protected). +* 2. all VMs total memory size is no larger than the platform memory size. +* 3. the virtual MMIO range for each VM is no larger than 1G. +*/ +static inline unsigned long pkvm_shadow_ept_pgtable_pages (int nr_vm) +{ +unsigned long pgtable_pages =__pkvm_pgtable_total_pages (); +unsigned long res; + + res = pgtable_pages; + +/* +* Above 'res' covered enough pages for a shadow EPT to create all +* possible levels mapping for the whole platform memory size. The pages +* reserved for lvl-1 mapping (usually 4K size) actually can be shared +* among all VM shadow EPTs as their occupied memory are partitioned +* according to the assumption. While for lvl-2,3,4 or possible lvl-5 +* mappings, above 'res' pages only covered for one shadow EPT, we need +* reserve more for other 'nr_vm -1' VMs. +* +* The lvl-2 to lvl-5 pages for one VM's shadow EPT can be approximately +* got by: +* __pkvm_pgtable_total_pages()/PTRS_PER_PTE +*/ + res +=DIV_ROUND_UP (pgtable_pages,PTRS_PER_PTE ) * (nr_vm -1 ); + +/* Allow 1 GiB for MMIO mappings for each VM */ + res +=__pkvm_pgtable_max_pages (SZ_1G >>PAGE_SHIFT ) * nr_vm; + +return res; +} +u64 hyp_total_reserve_pages (void );int pkvm_init_shadow_vm (struct kvm *kvm);
arch/x86/kvm/vmx/pkvm/pkvm_host.c +1 -0
@@ -41,6 +41,7 @@ u64 hyp_total_reserve_pages(void) total +=pkvm_vmemmap_pages (PKVM_VMEMMAP_ENTRY_SIZE ); total +=pkvm_mmu_pgtable_pages (); total +=host_ept_pgtable_pages (); + total +=pkvm_shadow_ept_pgtable_pages (PKVM_MAX_VM_NUM );return total; }
Signature · 1 line
2.25.1
RFC PATCH 3/13 pkvm: x86: Initialize the shadow EPT pool
In reply to Jason Chen CJFrom: Chuanxiao Dong <[email protected]>
Shadow EPT pool is the memory pool to allocate the memory for the shadow EPT paging structure pages. When destroy a shadow EPT, these pages will be put back to this pool again.
--- arch/x86/kvm/vmx/pkvm/hyp/ept.c | 9 +++++++++ arch/x86/kvm/vmx/pkvm/hyp/ept.h | 1 + arch/x86/kvm/vmx/pkvm/hyp/init_finalise.c | 13 +++++++++++++ 3 files changed, 23 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +9 -0
@@ -24,6 +24,8 @@ static struct hyp_pool host_ept_pool;static struct pkvm_pgtable host_ept;static pkvm_spinlock_t host_ept_lock = __PKVM_SPINLOCK_UNLOCKED; +static struct hyp_pool shadow_ept_pool; +static void flush_tlb_noop (void ) { };static void *host_ept_zalloc_page (void ) {
@@ -221,3 +223,10 @@ int handle_host_ept_violation(unsigned long gpa)pkvm_spin_unlock (&host_ept_lock);return ret; } + +int pkvm_shadow_ept_pool_init (void *ept_pool_base,unsigned long ept_pool_pages) +{ +unsigned long pfn =__pkvm_pa (ept_pool_base) >>PAGE_SHIFT ; + +return hyp_pool_init (&shadow_ept_pool, pfn, ept_pool_pages,0 ); +}
arch/x86/kvm/vmx/pkvm/hyp/ept.h +1 -0
@@ -17,5 +17,6 @@ int pkvm_host_ept_unmap(unsigned long vaddr_start, unsigned long phys_start,int pkvm_host_ept_init (struct pkvm_pgtable_cap *cap,void *ept_pool_base,unsigned long ept_pool_pages);int handle_host_ept_violation (unsigned long gpa); +int pkvm_shadow_ept_pool_init (void *ept_pool_base,unsigned long ept_pool_pages);#endif
arch/x86/kvm/vmx/pkvm/hyp/init_finalise.c +13 -0
@@ -23,6 +23,7 @@void *pkvm_mmu_pgt_base;void *pkvm_vmemmap_base;void *host_ept_pgt_base; +static void *shadow_ept_base;static int divide_memory_pool (phys_addr_t phys,unsigned long size) {
@@ -49,6 +50,12 @@ static int divide_memory_pool(phys_addr_t phys, unsigned long size)if (!host_ept_pgt_base)return -ENOMEM ; + nr_pages =pkvm_shadow_ept_pgtable_pages (PKVM_MAX_NORMAL_VM_NUM + +PKVM_MAX_PROTECTED_VM_NUM ); + shadow_ept_base =pkvm_early_alloc_contig (nr_pages); +if (!shadow_ept_base) +return -ENOMEM ; +return 0 ; }
@@ -291,6 +298,12 @@ int __pkvm_init_finalise(struct kvm_vcpu *vcpu, struct pkvm_section sections[],pkvm_init_nest (); + ret =pkvm_shadow_ept_pool_init (shadow_ept_base, +pkvm_shadow_ept_pgtable_pages (PKVM_MAX_NORMAL_VM_NUM + +PKVM_MAX_PROTECTED_VM_NUM )); +if (ret) + goto out; + pkvm_init = true; switch_pgt:
Signature · 1 line
2.25.1
RFC PATCH 4/13 pkvm: x86: Introduce shadow EPT
In reply to Jason Chen CJFrom: Chuanxiao Dong <[email protected]>
KVM in the host VM (KVM-high) can create its guest (nested VM).
Currently the KVM-high created guest EPT for such nested VM is directly used in its physical VMCS. This means that, the KVM-high is able to map any memory in this guest EPT, which will make this nested VM able to see any memory, including the private memory(pKVM's memory and future supported protected VM's memory) which needed to be protected.
Using shadow EPT can resolve this kind of security issue. In this solution, pKVM maintains a shadow EPT for each nested VM. Shadow EPT is populated during handling nested EPT violation with checking the mapping in guest EPT through page state mechanism first (introduced in later patches), so that pKVM can guarantee only allowed memory can be mapped for a nested VM.
This patch introduces APIs to initialize shadow EPT for nested VMs. How to build the page table entries from virtual EPT (in host VM) will be introduced later.
FIXME:
The current design is that one VM only has one shadow EPT. This brings a security issue for a VM to use vSMM mode, which is that the vCPU in non-vSMM mode will be able to access the memory belongs to vSMM mode because all vCPU are sharing one single shadow EPT.
--- arch/x86/kvm/vmx/pkvm/hyp/ept.c | 67 ++++++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/ept.h | 4 ++ arch/x86/kvm/vmx/pkvm/hyp/pkvm.c | 7 +++ arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h | 20 ++++++++ arch/x86/kvm/vmx/pkvm/hyp/vmx.h | 13 +++++ arch/x86/kvm/vmx/pkvm/pkvm_constants.c | 2 +- 6 files changed, 112 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +67 -0
@@ -18,6 +18,7 @@#include "pgtable.h" #include "ept.h" #include "memory.h" +#include "vmx.h" #include "debug.h" static struct hyp_pool host_ept_pool;
@@ -230,3 +231,69 @@ int pkvm_shadow_ept_pool_init(void *ept_pool_base, unsigned long ept_pool_pages)return hyp_pool_init (&shadow_ept_pool, pfn, ept_pool_pages,0 ); } + +static void *shadow_ept_zalloc_page (void ) +{ +return hyp_alloc_pages (&shadow_ept_pool,0 ); +} + +static void shadow_ept_get_page (void *vaddr) +{ +hyp_get_page (&shadow_ept_pool, vaddr); +} + +static void shadow_ept_put_page (void *vaddr) +{ +hyp_put_page (&shadow_ept_pool, vaddr); +} + +/*TODO: add tlb flush support for shadow ept */ +static struct pkvm_mm_ops shadow_ept_mm_ops = { + .phys_to_virt = pkvm_phys_to_virt, + .virt_to_phys = pkvm_virt_to_phys, + .zalloc_page = shadow_ept_zalloc_page, + .get_page = shadow_ept_get_page, + .put_page = shadow_ept_put_page, + .page_count = hyp_page_count, + .flush_tlb = flush_tlb_noop, +}; + +void pkvm_shadow_ept_deinit (struct shadow_ept_desc *desc) +{ +struct pkvm_pgtable *sept = &desc->sept; +struct pkvm_shadow_vm *vm =sept_desc_to_shadow_vm (desc); + +pkvm_spin_lock (&vm->lock); + +if (desc->shadow_eptp) { +pkvm_pgtable_destroy (sept); + +flush_ept (desc->shadow_eptp); + +memset (sept,0 ,sizeof (struct pkvm_pgtable )); + desc->shadow_eptp =0 ; + } + +pkvm_spin_unlock (&vm->lock); +} + +int pkvm_shadow_ept_init (struct shadow_ept_desc *desc) +{ +struct pkvm_pgtable_cap cap = { + .level = pkvm_hyp->ept_cap.level, + .allowed_pgsz = pkvm_hyp->ept_cap.allowed_pgsz, + .table_prot = pkvm_hyp->ept_cap.table_prot, + }; +int ret; + +memset (desc,0 ,sizeof (struct shadow_ept_desc )); + + ret =pkvm_pgtable_init (&desc->sept, &shadow_ept_mm_ops, &ept_ops, &cap, true); +if (ret) +return ret; + + desc->shadow_eptp =pkvm_construct_eptp (desc->sept.root_pa, cap.level); +flush_ept (desc->shadow_eptp); + +return 0 ; +}
arch/x86/kvm/vmx/pkvm/hyp/ept.h +4 -0
@@ -5,6 +5,8 @@#ifndef __PKVM_EPT_H#define __PKVM_EPT_H +#include "pkvm_hyp.h" +#define HOST_EPT_DEF_MEM_PROT (VMX_EPT_RWX_MASK | \ (MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT))#define HOST_EPT_DEF_MMIO_PROT (VMX_EPT_RWX_MASK | \
@@ -18,5 +20,7 @@ int pkvm_host_ept_init(struct pkvm_pgtable_cap *cap, void *ept_pool_base, unsigned long ept_pool_pages);int handle_host_ept_violation (unsigned long gpa);int pkvm_shadow_ept_pool_init (void *ept_pool_base,unsigned long ept_pool_pages); +int pkvm_shadow_ept_init (struct shadow_ept_desc *desc); +void pkvm_shadow_ept_deinit (struct shadow_ept_desc *desc);#endif
arch/x86/kvm/vmx/pkvm/hyp/pkvm.c +7 -0
@@ -7,6 +7,7 @@#include <pkvm.h> #include "pkvm_hyp.h" +#include "ept.h" struct pkvm_hyp *pkvm_hyp;
@@ -101,6 +102,10 @@ int __pkvm_init_shadow_vm(unsigned long kvm_va,pkvm_spin_lock_init (&vm->lock); vm->host_kvm_va = kvm_va; + +if (pkvm_shadow_ept_init (&vm->sept_desc)) +return -EINVAL ; +return allocate_shadow_vm_handle (vm); }
@@ -111,6 +116,8 @@ unsigned long __pkvm_teardown_shadow_vm(int shadow_vm_handle)if (!vm)return 0 ; +pkvm_shadow_ept_deinit (&vm->sept_desc); +memset (vm,0 ,sizeof (struct pkvm_shadow_vm ) +pkvm_shadow_vcpu_array_size ());return pkvm_virt_to_phys (vm);
arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h +20 -0
@@ -6,6 +6,17 @@#define __PKVM_HYP_H#include <asm/pkvm_spinlock.h> +#include "pgtable.h" + +/* +* Descriptor for shadow EPT +*/ +struct shadow_ept_desc { +/* shadow EPTP value configured by pkvm */ +u64 shadow_eptp; + +struct pkvm_pgtable sept; +};#define PKVM_MAX_NORMAL_VM_NUM 8#define PKVM_MAX_PROTECTED_VM_NUM 2
@@ -76,9 +87,18 @@ struct pkvm_shadow_vm {/* The host's kvm va. */ unsigned long host_kvm_va; +/* +* VM's shadow EPT. All vCPU shares one mapping. +* FIXME: a potential security issue if some vCPUs are +* in SMM but the others are not. +*/ +struct shadow_ept_desc sept_desc; +pkvm_spinlock_t lock; }__aligned(PAGE_SIZE) ; +#define sept_desc_to_shadow_vm (desc) container_of(desc, struct pkvm_shadow_vm, sept_desc) +int __pkvm_init_shadow_vm (unsigned long kvm_va,unsigned long shadow_pa,size_t shadow_size);unsigned long __pkvm_teardown_shadow_vm (int shadow_vm_handle);
arch/x86/kvm/vmx/pkvm/hyp/vmx.h +13 -0
@@ -28,6 +28,11 @@ static inline bool vmx_ept_has_mt_wb(void)return vmx_ept_capability_check (VMX_EPTP_WB_BIT ); } +static inline bool vmx_has_invept_context (void ) +{ +return vmx_ept_capability_check (VMX_EPT_EXTENT_CONTEXT_BIT ); +} +static inline u64 pkvm_construct_eptp (unsigned long root_hpa,int level) {u64 eptp =0 ;
@@ -71,6 +76,14 @@ static inline void vmcs_clear_track(struct vcpu_vmx *vmx, struct vmcs *vmcs) vmcs_clear(vmcs); } +static inline void flush_ept (u64 eptp) +{ +if (vmx_has_invept_context ()) +__invept (VMX_EPT_EXTENT_CONTEXT , eptp,0 ); +else +__invept (VMX_EPT_EXTENT_GLOBAL ,0 ,0 ); +} +void pkvm_init_host_state_area (struct pkvm_pcpu *pcpu,int cpu);#endif
arch/x86/kvm/vmx/pkvm/pkvm_constants.c +1 -1
@@ -7,7 +7,7 @@#include <linux/bug.h> #include <vdso/limits.h> #include <buddy_memory.h> -#include <vmx/vmx.h> +#include <pkvm.h> #include "hyp/pkvm_hyp.h" int main (void )
Signature · 1 line
2.25.1
RFC PATCH 5/13 pkvm: x86: Introduce vEPT to record guest EPT information
In reply to Jason Chen CJFrom: Chuanxiao Dong <[email protected]>
KVM-high manages a guest EPT for its nested VM, which shall be shadowed to shadow EPT in pKVM hypervisor.
Introduce a per vcpu pgtable(vEPT) to save the guest EPT related information from KVM-high when vmwrite(EPTP).
Each time when a vcpu vmwrites to EPTP, pKVM will setup the vEPT according to the new value or just do nothing if it is the same with the last one.
--- arch/x86/kvm/vmx/pkvm/hyp/ept.c | 33 ++++++++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/ept.h | 10 +++++++++ arch/x86/kvm/vmx/pkvm/hyp/memory.c | 6 +++++ arch/x86/kvm/vmx/pkvm/hyp/memory.h | 1 + arch/x86/kvm/vmx/pkvm/hyp/nested.c | 16 ++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h | 3 +++ 6 files changed, 69 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +33 -0
@@ -297,3 +297,36 @@ int pkvm_shadow_ept_init(struct shadow_ept_desc *desc)return 0 ; } + +/* +* virtual_ept_mm_ops is used as the ops for the ept constructed by +* KVM high in host. +* The physical address in this ept is the host VM GPA, which is +* the same with HPA. +*/ +struct pkvm_mm_ops virtual_ept_mm_ops = { + .phys_to_virt = host_gpa2hva, +}; + +void pkvm_guest_ept_deinit (struct shadow_vcpu_state *shadow_vcpu) +{ +struct pkvm_pgtable *vept = &shadow_vcpu->vept; + +memset (vept,0 ,sizeof (struct pkvm_pgtable )); +} + +void pkvm_guest_ept_init (struct shadow_vcpu_state *shadow_vcpu,u64 guest_eptp) +{ +/* +* TODO: we just assume guest will use page level the HW supported, +* it actually need align with KVM high +*/ +struct pkvm_pgtable_cap cap = { + .level = pkvm_hyp->ept_cap.level, + .allowed_pgsz = pkvm_hyp->ept_cap.allowed_pgsz, + .table_prot = pkvm_hyp->ept_cap.table_prot, + }; + +pkvm_pgtable_init (&shadow_vcpu->vept, &virtual_ept_mm_ops, &ept_ops, &cap, false); + shadow_vcpu->vept.root_pa =host_gpa2hpa (guest_eptp &SPTE_BASE_ADDR_MASK ); +}
arch/x86/kvm/vmx/pkvm/hyp/ept.h +10 -0
@@ -22,5 +22,15 @@ int handle_host_ept_violation(unsigned long gpa);int pkvm_shadow_ept_pool_init (void *ept_pool_base,unsigned long ept_pool_pages);int pkvm_shadow_ept_init (struct shadow_ept_desc *desc);void pkvm_shadow_ept_deinit (struct shadow_ept_desc *desc); +void pkvm_guest_ept_init (struct shadow_vcpu_state *shadow_vcpu,u64 guest_eptp); +void pkvm_guest_ept_deinit (struct shadow_vcpu_state *shadow_vcpu); +static inline bool is_valid_eptp (u64 eptp) +{ +if (!eptp || (eptp ==INVALID_GPA )) +return false; + +/* TODO: other bits check */ +return true; +}#endif
arch/x86/kvm/vmx/pkvm/hyp/memory.c +6 -0
@@ -73,6 +73,12 @@ void *host_gpa2hva(unsigned long gpa)return pkvm_phys_to_virt (gpa); } +unsigned long host_gpa2hpa (unsigned long gpa) +{ +/* Host VM is using identity mapping so GPA == HPA */ +return gpa; +} +extern struct pkvm_pgtable_ops mmu_ops;static struct pkvm_mm_ops mm_ops = { .phys_to_virt = host_gpa2hva,
arch/x86/kvm/vmx/pkvm/hyp/memory.h +1 -0
@@ -22,6 +22,7 @@ bool mem_range_included(struct mem_range *child, struct mem_range *parent);#include <linux/kvm_host.h> void *host_gpa2hva (unsigned long gpa); +unsigned long host_gpa2hpa (unsigned long gpa);int gva2gpa (struct kvm_vcpu *vcpu,gva_t gva,gpa_t *gpa,u32 access,struct x86_exception *exception); int read_gva(struct kvm_vcpu *vcpu, gva_t gva, void *addr,
arch/x86/kvm/vmx/pkvm/hyp/nested.c +16 -0
@@ -9,6 +9,7 @@#include "nested.h" #include "cpu.h" #include "vmx.h" +#include "ept.h" #include "debug.h" /*
@@ -704,6 +705,18 @@ static void nested_vmx_run(struct kvm_vcpu *vcpu, bool launch) } } +static void setup_guest_ept (struct shadow_vcpu_state *shadow_vcpu,u64 guest_eptp) +{ +struct vmcs12 *vmcs12 = (struct vmcs12 *)shadow_vcpu->cached_vmcs12; + +if (!is_valid_eptp (guest_eptp)) +pkvm_guest_ept_deinit (shadow_vcpu); +else if (vmcs12->ept_pointer != guest_eptp) { +pkvm_guest_ept_deinit (shadow_vcpu); +pkvm_guest_ept_init (shadow_vcpu, guest_eptp); + } +} +int handle_vmxon (struct kvm_vcpu *vcpu) {struct vcpu_vmx *vmx =to_vmx (vcpu);
@@ -908,6 +921,9 @@ int handle_vmwrite(struct kvm_vcpu *vcpu)if (field >=GUEST_ES_AR_BYTES && field <=GUEST_TR_AR_BYTES ) value &=0x1f0ff ; +if (field ==EPT_POINTER ) +setup_guest_ept (cur_shadow_vcpu, value); +vmcs12_write_any (vmcs12, field, offset, value);if (is_emulated_fields (field)) {
arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h +3 -0
@@ -41,6 +41,9 @@ struct shadow_vcpu_state {struct vcpu_vmx vmx; +/* represents for the virtual EPT configured by kvm-high */ +struct pkvm_pgtable vept; +/* assume vmcs02 is one page */ u8 vmcs02[PAGE_SIZE ]__aligned(PAGE_SIZE) ;u8 cached_vmcs12[VMCS12_SIZE ]__aligned(PAGE_SIZE) ;
Signature · 1 line
2.25.1
RFC PATCH 6/13 pkvm: x86: Add API to get the max phys address bits
In reply to Jason Chen CJFrom: Chuanxiao Dong <[email protected]>
Add API get_max_physaddr_bits(), this is necessary used to calculate the reserved bits for EPT entry.
--- arch/x86/kvm/vmx/pkvm/hyp/memory.c | 21 +++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/memory.h | 2 ++ 2 files changed, 23 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/memory.c +21 -0
@@ -10,9 +10,11 @@#include "memory.h" #include "pgtable.h" #include "pkvm_hyp.h" +#include "cpu.h" unsigned long __page_base_offset;unsigned long __symbol_base_offset; +static u8 max_physaddr_bits;unsigned int hyp_memblock_nr;struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS ];
@@ -285,3 +287,22 @@ int write_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, void *addr, unsigned int bytes) {return copy_gpa (vcpu, gpa, addr, bytes, false); } + +u64 get_max_physaddr_bits (void ) +{ +u32 eax, ebx, ecx, edx; + +if (max_physaddr_bits) +return max_physaddr_bits; + + eax =0x80000000 ; + ecx =0 ; +native_cpuid (&eax, &ebx, &ecx, &edx); +if (eax >=0x80000008 ) { + eax =0x80000008 ; +native_cpuid (&eax, &ebx, &ecx, &edx); + max_physaddr_bits = (u8 )eax &0xff ; + } + +return max_physaddr_bits; +}
arch/x86/kvm/vmx/pkvm/hyp/memory.h +2 -0
@@ -32,4 +32,6 @@ int write_gva(struct kvm_vcpu *vcpu, gva_t gva, void *addr,int read_gpa (struct kvm_vcpu *vcpu,gpa_t gpa,void *addr,unsigned int bytes);int write_gpa (struct kvm_vcpu *vcpu,gpa_t gpa,void *addr,unsigned int bytes); +u64 get_max_physaddr_bits (void ); +#endif
Signature · 1 line
2.25.1
RFC PATCH 7/13 pkvm: x86: Initialize ept_zero_check
In reply to Jason Chen CJFrom: Chuanxiao Dong <[email protected]>
ept_zero_check is used to check if a pte entry has any reserved bits set, or has any unsupported bit combinations. This is helpful to check if this entry can cause EPT misconfig or not during shadowing in the later patch.
--- arch/x86/kvm/vmx/pkvm/hyp/ept.c | 47 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/vmx.h | 7 +++++ 2 files changed, 54 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +47 -0
@@ -26,6 +26,7 @@ static struct pkvm_pgtable host_ept;static pkvm_spinlock_t host_ept_lock = __PKVM_SPINLOCK_UNLOCKED;static struct hyp_pool shadow_ept_pool; +static struct rsvd_bits_validate ept_zero_check;static void flush_tlb_noop (void ) { };static void *host_ept_zalloc_page (void )
@@ -151,16 +152,62 @@ int pkvm_host_ept_unmap(unsigned long vaddr_start, unsigned long phys_start,return pkvm_pgtable_unmap (&host_ept, vaddr_start, phys_start, size); } +static void reset_rsvds_bits_mask_ept (struct rsvd_bits_validate *rsvd_check, +u64 pa_bits_rsvd,bool execonly, +int huge_page_level) +{ +u64 high_bits_rsvd = pa_bits_rsvd &rsvd_bits (0 ,51 ); +u64 large_1g_rsvd =0 , large_2m_rsvd =0 ; +u64 bad_mt_xwr; + +if (huge_page_level <PG_LEVEL_1G ) + large_1g_rsvd =rsvd_bits (7 ,7 ); +if (huge_page_level <PG_LEVEL_2M ) + large_2m_rsvd =rsvd_bits (7 ,7 ); + + rsvd_check->rsvd_bits_mask[0 ][4 ] = high_bits_rsvd |rsvd_bits (3 ,7 ); + rsvd_check->rsvd_bits_mask[0 ][3 ] = high_bits_rsvd |rsvd_bits (3 ,7 ); + rsvd_check->rsvd_bits_mask[0 ][2 ] = high_bits_rsvd |rsvd_bits (3 ,6 ) | large_1g_rsvd; + rsvd_check->rsvd_bits_mask[0 ][1 ] = high_bits_rsvd |rsvd_bits (3 ,6 ) | large_2m_rsvd; + rsvd_check->rsvd_bits_mask[0 ][0 ] = high_bits_rsvd; + +/* large page */ + rsvd_check->rsvd_bits_mask[1 ][4 ] = rsvd_check->rsvd_bits_mask[0 ][4 ]; + rsvd_check->rsvd_bits_mask[1 ][3 ] = rsvd_check->rsvd_bits_mask[0 ][3 ]; + rsvd_check->rsvd_bits_mask[1 ][2 ] = high_bits_rsvd |rsvd_bits (12 ,29 ) | large_1g_rsvd; + rsvd_check->rsvd_bits_mask[1 ][1 ] = high_bits_rsvd |rsvd_bits (12 ,20 ) | large_2m_rsvd; + rsvd_check->rsvd_bits_mask[1 ][0 ] = rsvd_check->rsvd_bits_mask[0 ][0 ]; + + bad_mt_xwr =0xFFull << (2 *8 );/* bits 3..5 must not be 2 */ + bad_mt_xwr |=0xFFull << (3 *8 );/* bits 3..5 must not be 3 */ + bad_mt_xwr |=0xFFull << (7 *8 );/* bits 3..5 must not be 7 */ + bad_mt_xwr |=REPEAT_BYTE (1ull <<2 );/* bits 0..2 must not be 010 */ + bad_mt_xwr |=REPEAT_BYTE (1ull <<6 );/* bits 0..2 must not be 110 */ +if (!execonly) { +/* bits 0..2 must not be 100 unless VMX capabilities allow it */ + bad_mt_xwr |=REPEAT_BYTE (1ull <<4 ); + } + rsvd_check->bad_mt_xwr = bad_mt_xwr; +} +int pkvm_host_ept_init (struct pkvm_pgtable_cap *cap,void *ept_pool_base,unsigned long ept_pool_pages) {unsigned long pfn =__pkvm_pa (ept_pool_base) >>PAGE_SHIFT ;int ret; +u8 pa_bits; ret =hyp_pool_init (&host_ept_pool, pfn, ept_pool_pages,0 );if (ret)return ret; + pa_bits =get_max_physaddr_bits (); +if (!pa_bits) +return -EINVAL ; +reset_rsvds_bits_mask_ept (&ept_zero_check,rsvd_bits (pa_bits,63 ), +vmx_has_ept_execute_only (), +fls (cap->allowed_pgsz) -1 ); + pkvm_hyp->host_vm.ept = &host_ept;return pkvm_pgtable_init (&host_ept, &host_ept_mm_ops, &ept_ops, cap, true); }
arch/x86/kvm/vmx/pkvm/hyp/vmx.h +7 -0
@@ -6,6 +6,8 @@#ifndef __PKVM_VMX_H#define __PKVM_VMX_H +#include "pkvm_hyp.h" +static inline bool vmx_ept_capability_check (u32 bit) {struct vmx_capability *vmx_cap = &pkvm_hyp->vmx_cap;
@@ -33,6 +35,11 @@ static inline bool vmx_has_invept_context(void)return vmx_ept_capability_check (VMX_EPT_EXTENT_CONTEXT_BIT ); } +static inline bool vmx_has_ept_execute_only (void ) +{ +return vmx_ept_capability_check (VMX_EPT_EXECUTE_ONLY_BIT ); +} +static inline u64 pkvm_construct_eptp (unsigned long root_hpa,int level) {u64 eptp =0 ;
Signature · 1 line
2.25.1
RFC PATCH 8/13 pkvm: x86: Add support for pKVM to handle the nested EPT violation
In reply to Jason Chen CJFrom: Chuanxiao Dong <[email protected]>
Shadow EPT is used as EPTP for nested VMs and it's initialized with empty entries. So when EPT violation happened, pKVM shall shadowing EPT page table entry from guest vEPT under well-designed rules.
pKVM walks the guest vEPT to find out if the L2 GPA already has a proper mapping in vEPT. If already has one, pKVM map it in the shadow EPT - now just directly do the same mapping as vEPT, in the future, pKVM shall follow some rules to do the mapping, like only after confirming the page's ownership belong to this VM.
For other cases non-proper mapping in vEPT, they actually seek for the handling from host VM (or more accurate from KVM-high):
- for the case vEPT is not present, it's actually a #PF and requesting page allocation from the host, so pKVM directly forward the violation back to the KVM-high for proper handling.
- and for the case that vEPT contains a misconfig mapping, such misconfig is set by KVM-high so it expected to handle it. pKVM modify VM_EXIT_REASON field to EPT_MISCONFIG if VMX support read-only field modification, then deliver back to KVM-high. Otherwise deliver EPT_VIOLATION to KVM-high directly as KVM handle EPT_MISCONFIG in its EPT_VIOLATION handler as well.
- another case is that if the EPT violation vmexit has pending event to be injected, such event also seeks for host handling to do further event injection, pKVM shall return back to the KVM-high with corresponding pending event vmexit info.
--- arch/x86/kvm/vmx/pkvm/hyp/ept.c | 60 ++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/ept.h | 8 ++++ arch/x86/kvm/vmx/pkvm/hyp/nested.c | 40 ++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/vmx.h | 6 +++ arch/x86/kvm/vmx/pkvm/pkvm_host.c | 2 + 5 files changed, 116 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +60 -0
@@ -377,3 +377,63 @@ void pkvm_guest_ept_init(struct shadow_vcpu_state *shadow_vcpu, u64 guest_eptp)pkvm_pgtable_init (&shadow_vcpu->vept, &virtual_ept_mm_ops, &ept_ops, &cap, false); shadow_vcpu->vept.root_pa =host_gpa2hpa (guest_eptp &SPTE_BASE_ADDR_MASK ); } + +static bool is_access_violation (u64 ept_entry,u64 exit_qual) +{ +bool access_violation = false; + +if (/* Caused by data read */ + (((exit_qual &0x1UL ) !=0UL ) && ((ept_entry &VMX_EPT_READABLE_MASK ) ==0 )) || +/* Caused by data write */ + (((exit_qual &0x2UL ) !=0UL ) && ((ept_entry &VMX_EPT_WRITABLE_MASK ) ==0 )) || +/* Caused by instruction fetch */ + (((exit_qual &0x4UL ) !=0UL ) && ((ept_entry &VMX_EPT_EXECUTABLE_MASK ) ==0 ))) { + access_violation = true; + } + +return access_violation; +} + +enum sept_handle_ret +pkvm_handle_shadow_ept_violation (struct shadow_vcpu_state *shadow_vcpu,u64 l2_gpa,u64 exit_quali) +{ +struct pkvm_shadow_vm *vm = shadow_vcpu->vm; +struct shadow_ept_desc *desc = &vm->sept_desc; +struct pkvm_pgtable *sept = &desc->sept; +struct pkvm_pgtable_ops *pgt_ops = sept->pgt_ops; +struct pkvm_pgtable *vept = &shadow_vcpu->vept; +enum sept_handle_ret ret =PKVM_NOT_HANDLED ; +unsigned long phys; +int level; +u64 gprot, rsvd_chk_gprot; + +pkvm_spin_lock (&vm->lock); + +pkvm_pgtable_lookup (vept, l2_gpa, &phys, &gprot, &level); +if (phys ==INVALID_ADDR ) +/* Geust EPT not valid, back to kvm-high */ + goto out; + +if (is_access_violation (gprot, exit_quali)) +/* Guest EPT error, refuse to handle in shadow ept */ + goto out; + + rsvd_chk_gprot = gprot; +/* is_rsvd_spte() need based on PAGE_SIZE bit */ +if (level !=PG_LEVEL_4K ) + pgt_ops->pgt_entry_mkhuge (&rsvd_chk_gprot); + +if (is_rsvd_spte (&ept_zero_check, rsvd_chk_gprot, level)) { + ret =PKVM_INJECT_EPT_MISC ; + }else { +unsigned long level_size = pgt_ops->pgt_level_to_size (level); +unsigned long gpa =ALIGN_DOWN (l2_gpa, level_size); +unsigned long hpa =ALIGN_DOWN (host_gpa2hpa (phys), level_size); + +if (!pkvm_pgtable_map (sept, gpa, hpa, level_size,0 , gprot)) + ret =PKVM_HANDLED ; + } +out: +pkvm_spin_unlock (&vm->lock); +return ret; +}
arch/x86/kvm/vmx/pkvm/hyp/ept.h +8 -0
@@ -12,6 +12,12 @@#define HOST_EPT_DEF_MMIO_PROT (VMX_EPT_RWX_MASK | \ (MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT)) +enum sept_handle_ret { +PKVM_NOT_HANDLED , +PKVM_HANDLED , +PKVM_INJECT_EPT_MISC , +}; +int pkvm_host_ept_map (unsigned long vaddr_start,unsigned long phys_start,unsigned long size,int pgsz_mask,u64 prot); int pkvm_host_ept_unmap(unsigned long vaddr_start, unsigned long phys_start,
@@ -24,6 +30,8 @@ int pkvm_shadow_ept_init(struct shadow_ept_desc *desc);void pkvm_shadow_ept_deinit (struct shadow_ept_desc *desc);void pkvm_guest_ept_init (struct shadow_vcpu_state *shadow_vcpu,u64 guest_eptp);void pkvm_guest_ept_deinit (struct shadow_vcpu_state *shadow_vcpu); +enum sept_handle_ret +pkvm_handle_shadow_ept_violation (struct shadow_vcpu_state *shadow_vcpu,u64 l2_gpa,u64 exit_quali);static inline bool is_valid_eptp (u64 eptp) {
arch/x86/kvm/vmx/pkvm/hyp/nested.c +40 -0
@@ -1007,6 +1007,39 @@ int handle_vmlaunch(struct kvm_vcpu *vcpu)return 0 ; } +static bool nested_handle_ept_violation (struct shadow_vcpu_state *shadow_vcpu, +u64 l2_gpa,u64 exit_quali) +{ +enum sept_handle_ret ret =pkvm_handle_shadow_ept_violation (shadow_vcpu, + l2_gpa, exit_quali); +bool handled = false; + +switch (ret) { +case PKVM_INJECT_EPT_MISC : { +/* +* Inject EPT_MISCONFIG vmexit reason if can directly modify +* the read-only fields. Otherwise still deliver EPT_VIOLATION +* for simplification. +*/ +if (vmx_has_vmwrite_any_field ()) +vmcs_write32 (VM_EXIT_REASON ,EXIT_REASON_EPT_MISCONFIG ); +break ; + } +case PKVM_HANDLED : + handled = true; +break ; +default : +break ; + } + +if (handled && (vmcs_read32 (IDT_VECTORING_INFO_FIELD ) & +VECTORING_INFO_VALID_MASK )) +/* pending interrupt, back to kvm-high to inject */ + handled = false; + +return handled; +} +int nested_vmexit (struct kvm_vcpu *vcpu) {struct vcpu_vmx *vmx =to_vmx (vcpu);
@@ -1015,6 +1048,13 @@ int nested_vmexit(struct kvm_vcpu *vcpu)struct vmcs *vmcs02 = (struct vmcs *)cur_shadow_vcpu->vmcs02;struct vmcs12 *vmcs12 = (struct vmcs12 *)cur_shadow_vcpu->cached_vmcs12; +if ((vmx->exit_reason.full ==EXIT_REASON_EPT_VIOLATION ) && +nested_handle_ept_violation (cur_shadow_vcpu, +vmcs_read64 (GUEST_PHYSICAL_ADDRESS ), + vmx->exit_qualification)) +/* EPT violation can be handled by pkvm, no need back to kvm-high */ +return 0 ; +/* clear guest mode if need switch back to host */ vcpu->arch.hflags &= ~HF_GUEST_MASK ;
arch/x86/kvm/vmx/pkvm/hyp/vmx.h +6 -0
@@ -40,6 +40,12 @@ static inline bool vmx_has_ept_execute_only(void)return vmx_ept_capability_check (VMX_EPT_EXECUTE_ONLY_BIT ); } +static inline bool vmx_has_vmwrite_any_field (void ) +{ +return !!(pkvm_hyp->vmcs_config.nested.misc_low & +MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS ); +} +static inline u64 pkvm_construct_eptp (unsigned long root_hpa,int level) {u64 eptp =0 ;
arch/x86/kvm/vmx/pkvm/pkvm_host.c +2 -0
@@ -380,6 +380,8 @@ static __init void pkvm_host_setup_nested_vmx_cap(struct pkvm_hyp *pkvm)rdmsr (MSR_IA32_VMX_ENTRY_CTLS , msrs->entry_ctls_low, msrs->entry_ctls_high); + +rdmsr (MSR_IA32_VMX_MISC , msrs->misc_low, msrs->misc_high); }static __init int pkvm_host_check_and_setup_vmx_cap (struct pkvm_hyp *pkvm)
Signature · 1 line
2.25.1
RFC PATCH 9/13 pkvm: x86: Introduce PKVM_ASSERT
In reply to Jason Chen CJFrom: Chuanxiao Dong <[email protected]>
Add new file bug.h to support PKVM_ASSERT.
If CONFIG_PKVM_INTEL_DEBUG enabled, print out assertion information then call into Linux BUG(), otherwise just do a dead loop.
--- arch/x86/kvm/vmx/pkvm/hyp/bug.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/bug.h new file +23 -0
@@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2022 Intel Corporation +*/ +#ifndef __PKVM_BUG_H +#define __PKVM_BUG_H + +#ifdef CONFIG_PKVM_INTEL_DEBUG +#include <linux/printk.h> + +#define PKVM_ASSERT (c) \ +do { \ + if (!(c)) { \ + pr_err("assertion failed %s: %d: %s\n", \ + __FILE__, __LINE__, #c); \ + BUG(); \ + } \ +} while (0) +#else +#define PKVM_ASSERT (c) do { } while (!(c)) +#endif + +#endif
Signature · 1 line
2.25.1
RFC PATCH 10/13 pkvm: x86: add pkvm_pgtable_unmap_safe for a safe unmap
In reply to Jason Chen CJFrom: Chuanxiao Dong <[email protected]>
The current pkvm_pgtable_unmap API takes a phys parameter which will check if the unmap entry really maps this phys memory. This is not necessary for all the unmap cases, and actually phys is unknown to some cases.
However, for some cases, still prefer to perform such check to avoid unmapping an incorrect entry because such failure is not easy to be detected but cause a serial security issue. The unmap from page state API and from the host EPT are the cases better to have such check. So introduce a safe version unmap API.
The safe version unmap API will be used by page state machine and host EPT to do unmap. For the MMU page table, it can still use the normal version of the unmap.
--- arch/x86/kvm/vmx/pkvm/hyp/ept.c | 4 +- arch/x86/kvm/vmx/pkvm/hyp/mmu.c | 4 +- arch/x86/kvm/vmx/pkvm/hyp/mmu.h | 3 +- arch/x86/kvm/vmx/pkvm/hyp/pgtable.c | 67 ++++++++++++++++++++--------- arch/x86/kvm/vmx/pkvm/hyp/pgtable.h | 4 +- 5 files changed, 55 insertions(+), 27 deletions(-)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +2 -2
@@ -147,9 +147,9 @@ int pkvm_host_ept_map(unsigned long vaddr_start, unsigned long phys_start, }int pkvm_host_ept_unmap (unsigned long vaddr_start,unsigned long phys_start, -unsigned long size) +unsigned long size) { -return pkvm_pgtable_unmap (&host_ept, vaddr_start, phys_start, size); +return pkvm_pgtable_unmap_safe (&host_ept, vaddr_start, phys_start, size); } static void reset_rsvds_bits_mask_ept(struct rsvd_bits_validate *rsvd_check,
arch/x86/kvm/vmx/pkvm/hyp/mmu.c +2 -2
@@ -192,11 +192,11 @@ int pkvm_mmu_map(unsigned long vaddr_start, unsigned long phys_start,return ret; } -int pkvm_mmu_unmap (unsigned long vaddr_start,unsigned long phys_start,unsigned long size) +int pkvm_mmu_unmap (unsigned long vaddr_start,unsigned long size) {int ret; - ret =pkvm_pgtable_unmap (&hyp_mmu, vaddr_start, phys_start, size); + ret =pkvm_pgtable_unmap (&hyp_mmu, vaddr_start, size);return ret; }
arch/x86/kvm/vmx/pkvm/hyp/mmu.h +1 -2
@@ -8,8 +8,7 @@int pkvm_mmu_map (unsigned long vaddr_start,unsigned long phys_start,unsigned long size,int pgsz_mask,u64 prot); -int pkvm_mmu_unmap (unsigned long vaddr_start,unsigned long phys_start, -unsigned long size); +int pkvm_mmu_unmap (unsigned long vaddr_start,unsigned long size);int pkvm_early_mmu_init (struct pkvm_pgtable_cap *cap,void *mmu_pool_base,unsigned long mmu_pool_pages);
arch/x86/kvm/vmx/pkvm/hyp/pgtable.c +47 -20
109 lines not on this page · raw message
arch/x86/kvm/vmx/pkvm/hyp/pgtable.h +3 -1
@@ -74,7 +74,9 @@ int pkvm_pgtable_map(struct pkvm_pgtable *pgt, unsigned long vaddr_start, unsigned long phys_start, unsigned long size, int pgsz_mask, u64 entry_prot);int pkvm_pgtable_unmap (struct pkvm_pgtable *pgt,unsigned long vaddr_start, -unsigned long phys_start,unsigned long size); +unsigned long size); +int pkvm_pgtable_unmap_safe (struct pkvm_pgtable *pgt,unsigned long vaddr_start, +unsigned long phys_start,unsigned long size);void pkvm_pgtable_lookup (struct pkvm_pgtable *pgt,unsigned long vaddr,unsigned long *pphys,u64 *pprot,int *plevel);void pkvm_pgtable_destroy (struct pkvm_pgtable *pgt);
Signature · 1 line
2.25.1
RFC PATCH 11/13 pkvm: x86: Introduce shadow EPT invalidation support
In reply to Jason Chen CJShadow EPT invalidation is required to emulate INVEPT.
And as a VM only has one shadow EPT, when vEPTP got changed, the shadow EPT shall be invalidated as well.
The invalidation is implemented by just simply destroying all the existing mappings for corresponding shadow EPT without freeing the root.
--- arch/x86/kvm/vmx/pkvm/hyp/ept.c | 18 ++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/ept.h | 1 + arch/x86/kvm/vmx/pkvm/hyp/nested.c | 12 ++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h | 3 +++ 4 files changed, 34 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +18 -0
@@ -305,6 +305,24 @@ static struct pkvm_mm_ops shadow_ept_mm_ops = { .flush_tlb = flush_tlb_noop, }; +void pkvm_invalidate_shadow_ept (struct shadow_ept_desc *desc) +{ +struct pkvm_shadow_vm *vm =sept_desc_to_shadow_vm (desc); +struct pkvm_pgtable *sept = &desc->sept; +unsigned long size = sept->pgt_ops->pgt_level_to_size (sept->level +1 ); + +pkvm_spin_lock (&vm->lock); + +if (!is_valid_eptp (desc->shadow_eptp)) + goto out; + +pkvm_pgtable_unmap (sept,0 , size); + +flush_ept (desc->shadow_eptp); +out: +pkvm_spin_unlock (&vm->lock); +} +void pkvm_shadow_ept_deinit (struct shadow_ept_desc *desc) {struct pkvm_pgtable *sept = &desc->sept;
arch/x86/kvm/vmx/pkvm/hyp/ept.h +1 -0
@@ -32,6 +32,7 @@ void pkvm_guest_ept_init(struct shadow_vcpu_state *shadow_vcpu, u64 guest_eptp);void pkvm_guest_ept_deinit (struct shadow_vcpu_state *shadow_vcpu);enum sept_handle_ret pkvm_handle_shadow_ept_violation (struct shadow_vcpu_state *shadow_vcpu,u64 l2_gpa,u64 exit_quali); +void pkvm_invalidate_shadow_ept (struct shadow_ept_desc *desc);static inline bool is_valid_eptp (u64 eptp) {
arch/x86/kvm/vmx/pkvm/hyp/nested.c +12 -0
@@ -708,6 +708,8 @@ static void nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)static void setup_guest_ept (struct shadow_vcpu_state *shadow_vcpu,u64 guest_eptp) {struct vmcs12 *vmcs12 = (struct vmcs12 *)shadow_vcpu->cached_vmcs12; +struct pkvm_shadow_vm *vm = shadow_vcpu->vm; +bool invalidate = false;if (!is_valid_eptp (guest_eptp))pkvm_guest_ept_deinit (shadow_vcpu);
@@ -715,6 +717,16 @@ static void setup_guest_ept(struct shadow_vcpu_state *shadow_vcpu, u64 guest_eptpkvm_guest_ept_deinit (shadow_vcpu);pkvm_guest_ept_init (shadow_vcpu, guest_eptp); } + +pkvm_spin_lock (&vm->lock); +if (vm->sept_desc.last_guest_eptp != guest_eptp) { + vm->sept_desc.last_guest_eptp = guest_eptp; + invalidate = true; + } +pkvm_spin_unlock (&vm->lock); + +if (invalidate) +pkvm_invalidate_shadow_ept (&vm->sept_desc); }int handle_vmxon (struct kvm_vcpu *vcpu)
arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h +3 -0
@@ -15,6 +15,9 @@ struct shadow_ept_desc {/* shadow EPTP value configured by pkvm */ u64 shadow_eptp; +/* Save the last guest EPTP value configured by kvm high */ +u64 last_guest_eptp; +struct pkvm_pgtable sept; };
Signature · 1 line
2.25.1
RFC PATCH 12/13 pkvm: x86: Add INVEPT instruction emulation
In reply to Jason Chen CJFrom: Chuanxiao Dong <[email protected]>
For INVEPT with single context, look up the vCPU vmcs12 to find out the corresponding shadow EPT to do the invalidation.
For INVEPT with global context, invalidate all the valid shadow EPTs.
TODO: current emulation may cause a lot of unnecessary shadow EPT violation due to inappropriate EPT invalidation as all page table entries are freed. Optimization shall be done in the future to do PV INVEPT with specific range.
--- arch/x86/kvm/vmx/pkvm/hyp/nested.c | 106 +++++++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/nested.h | 1 + arch/x86/kvm/vmx/pkvm/hyp/pkvm.c | 27 +++++++ arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h | 3 + arch/x86/kvm/vmx/pkvm/hyp/vmexit.c | 3 + arch/x86/kvm/vmx/pkvm/hyp/vmx.h | 5 ++ 6 files changed, 145 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/nested.c +106 -0
@@ -729,6 +729,56 @@ static void setup_guest_ept(struct shadow_vcpu_state *shadow_vcpu, u64 guest_eptpkvm_invalidate_shadow_ept (&vm->sept_desc); } +static int __do_handle_invept (struct pkvm_shadow_vm *vm,void *data) +{ +if (data) { +/* +* Single context invalidate with specific guest eptp +*/ +u64 guest_eptp = *(u64 *)data; +struct shadow_vcpu_state *vcpu; +struct vmcs12 *vmcs12; +s64 shadow_vcpu_handle; +int i; + +pkvm_spin_lock (&vm->lock); + +for (i =0 ; i < vm->created_vcpus; i++) { + shadow_vcpu_handle =to_shadow_vcpu_handle (vm->shadow_vm_handle, i); + vcpu =get_shadow_vcpu (shadow_vcpu_handle); +if (!vcpu) +continue ; + vmcs12 = (struct vmcs12 *)vcpu->cached_vmcs12; +if (vmcs12->ept_pointer == guest_eptp) { +put_shadow_vcpu (shadow_vcpu_handle); +break ; + } +put_shadow_vcpu (shadow_vcpu_handle); + } + +pkvm_spin_unlock (&vm->lock); + +/* No vCPU is using this guest eptp so no need invalidate */ +if (i == vm->created_vcpus) + goto done; + } + +/* +* Gloable context invalidation directly went here. +*/ +pkvm_invalidate_shadow_ept (&vm->sept_desc); +done: +return data ?1 :0 ; +} + +static void do_handle_invept (u64 *guest_eptp) +{ +if (guest_eptp && !is_valid_eptp (*guest_eptp)) +return ; + +for_each_valid_shadow_vm (__do_handle_invept, guest_eptp); +} +int handle_vmxon (struct kvm_vcpu *vcpu) {struct vcpu_vmx *vmx =to_vmx (vcpu);
@@ -1019,6 +1069,62 @@ int handle_vmlaunch(struct kvm_vcpu *vcpu)return 0 ; } +int handle_invept (struct kvm_vcpu *vcpu) +{ +struct vmx_capability *vmx_cap = &pkvm_hyp->vmx_cap; +struct vcpu_vmx *vmx =to_vmx (vcpu); +u32 vmx_instruction_info, types; +unsigned long type; +int gpr_index; + +if (!vmx_has_invept ()) +/* TODO: inject #UD */ +return -EINVAL ; + +if (!check_vmx_permission (vcpu)) +return 0 ; + + vmx_instruction_info =vmcs_read32 (VMX_INSTRUCTION_INFO ); + gpr_index =vmx_get_instr_info_reg2 (vmx_instruction_info); + type = vcpu->arch.regs[gpr_index]; + types = (vmx_cap->ept >>VMX_EPT_EXTENT_SHIFT ) &6 ; + +if (type >=32 || !(types & (1 << type))) { +nested_vmx_result (VMfailValid,VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID ); +return 0 ; + } + +switch (type) { +case VMX_EPT_EXTENT_CONTEXT : { +struct x86_exception e; +gva_t gva; +struct { +u64 eptp, gpa; + } operand; + +if (get_vmx_mem_address (vcpu, vmx->exit_qualification, + vmx_instruction_info, &gva)) +/* TODO: handle the decode failure */ +return -EINVAL ; + +if (read_gva (vcpu, gva, &operand,sizeof (operand), &e) <0 ) +/*TODO: handle memory failure exception */ +return -EINVAL ; + +do_handle_invept (&operand.eptp); +break ; + } +case VMX_EPT_EXTENT_GLOBAL : +do_handle_invept (NULL ); +break ; +default : +break ; + } + +nested_vmx_result (VMsucceed,0 ); +return 0 ; +} +static bool nested_handle_ept_violation (struct shadow_vcpu_state *shadow_vcpu,u64 l2_gpa,u64 exit_quali) {
arch/x86/kvm/vmx/pkvm/hyp/nested.h +1 -0
@@ -13,6 +13,7 @@ int handle_vmwrite(struct kvm_vcpu *vcpu);int handle_vmread (struct kvm_vcpu *vcpu);int handle_vmresume (struct kvm_vcpu *vcpu);int handle_vmlaunch (struct kvm_vcpu *vcpu); +int handle_invept (struct kvm_vcpu *vcpu);int nested_vmexit (struct kvm_vcpu *vcpu);void pkvm_init_nest (void );
arch/x86/kvm/vmx/pkvm/hyp/pkvm.c +27 -0
@@ -355,3 +355,30 @@ unsigned long __pkvm_teardown_shadow_vcpu(s64 shadow_vcpu_handle)memset (shadow_vcpu,0 ,sizeof (struct shadow_vcpu_state ));return pkvm_virt_to_phys (shadow_vcpu); } + +int for_each_valid_shadow_vm (int (*fn)(struct pkvm_shadow_vm *vm,void *data), +void *data) +{ +struct pkvm_shadow_vm *vm; +int i, ret; + +pkvm_spin_lock (&shadow_vms_lock); +for (i =HANDLE_OFFSET ; i <MAX_SHADOW_VMS ; i++) { +if (!test_bit (i, shadow_vms_bitmap)) +continue ; + + vm =get_shadow_vm (i); +if (!vm) +continue ; + + ret =fn (vm, data); + +put_shadow_vm (i); + +if (ret) +break ; + } +pkvm_spin_unlock (&shadow_vms_lock); + +return ret <0 ? ret :0 ; +}
arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h +3 -0
@@ -116,6 +116,9 @@ struct shadow_vcpu_state *get_shadow_vcpu(s64 shadow_vcpu_handle);void put_shadow_vcpu (s64 shadow_vcpu_handle);s64 find_shadow_vcpu_handle_by_vmcs (unsigned long vmcs12_pa); +int for_each_valid_shadow_vm (int (*fn)(struct pkvm_shadow_vm *vm,void *data), +void *data); +extern struct pkvm_hyp *pkvm_hyp;#endif
arch/x86/kvm/vmx/pkvm/hyp/vmexit.c +3 -0
@@ -241,6 +241,9 @@ int pkvm_main(struct kvm_vcpu *vcpu)handle_irq_window (vcpu);break ;case EXIT_REASON_INVEPT : +handle_invept (vcpu); + skip_instruction = true; +break ;case EXIT_REASON_INVVPID :ept_sync_global (); skip_instruction = true;
arch/x86/kvm/vmx/pkvm/hyp/vmx.h +5 -0
@@ -15,6 +15,11 @@ static inline bool vmx_ept_capability_check(u32 bit)return vmx_cap->ept & bit; } +static inline bool vmx_has_invept (void ) +{ +return vmx_ept_capability_check (VMX_EPT_INVEPT_BIT ); +} +static inline bool vmx_ept_has_4levels (void ) {return vmx_ept_capability_check (VMX_EPT_PAGE_WALK_4_BIT );
Signature · 1 line
2.25.1
RFC PATCH 13/13 pkvm: x86: Switch to use shadow EPT for nested guests
In reply to Jason Chen CJFrom: Chuanxiao Dong <[email protected]>
Shadow EPT currently is per VM which is configured when creating VM. The shadow EPTP won't be changed at runtime. So just configure the EPTP for vmcs02 when the first this vmcs02 is loaded and skip the later EPTP sync from vmcs12 to vmcs02.
--- arch/x86/kvm/vmx/pkvm/hyp/nested.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/nested.c +17 -0
@@ -572,6 +572,14 @@ static void sync_vmcs12_dirty_fields_to_vmcs02(struct vcpu_vmx *vmx, struct vmcsif (vmx->nested.dirty_vmcs12) {for (i =0 ; i < max_emulated_fields; i++) { field = emulated_fields[i]; +if (field.encoding ==EPT_POINTER ) +/* +* EPTP is configured as shadow EPTP when the first +* time the vmcs02 is loaded. As shadow EPTP is not +* changed at the runtime, also cannot use the virtual +* EPT from KVM high, no need to sync to vmcs02 again. +*/ +continue ; val =vmcs12_read_any (vmcs12, field.encoding, field.offset); phys_val =emulate_field_for_vmcs02 (vmx, field.encoding, val);__vmcs_writel (field.encoding, phys_val);
@@ -860,6 +868,15 @@ int handle_vmptrld(struct kvm_vcpu *vcpu)vmcs_load_track (vmx, vmcs02);pkvm_init_host_state_area (pkvm_hvcpu->pcpu, vcpu->cpu);vmcs_writel (HOST_RIP , (unsigned long )__pkvm_vmx_vmexit); +/* +* EPTP is mantained by pKVM and configured with +* shadow EPTP from its corresponding shadow VM. +* As shadow EPTP is not changed at runtime, set +* it to EPTP when the first time this vmcs02 is +* loading. +*/ +vmcs_write64 (EPT_POINTER , + shadow_vcpu->vm->sept_desc.shadow_eptp); shadow_vcpu->last_cpu = vcpu->cpu; shadow_vcpu->vmcs02_inited = true; }else {
Signature · 1 line
2.25.1