Skip to contentSkip to conversation

EPT Emulation

RFC PATCH New Jason Chen CJ 1 participant 14 messages Mar 12, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover EPT Emulation

This patch set is part-6 of this RFC patches. It introduces EPT emulation for pKVM on Intel platform.

Add EPT emulation through shadowing vEPT in host VM.

Host VM launches its guest, and manage such guest's memory through a EPT page table maintained in host KVM. Meanwhile this EPT page table is untrusted to pKVM, so pKVM shall not directly use this EPT as guest's active EPT. To ensure isolating of guest memory for protected VM, pKVM hypervisor shadows such guest's EPT in host KVM, to build out active EPT page table after necessary check (the check is based on page state management which will be introduced later). It's actually an emulation for guest EPT page table, the guest EPT page table in host KVM is called "virtual EPT", while the active EPT page table in pKVM is called "shadow EPT".

Shadow EPT in pKVM is initialized with empty mapping. Guest EPT violation leads to the EPT shadowing. The EPT shadowing first walks virtual EPT in host VM to find out the virtual mapping, then setup the same mapping in shadow EPT - this will be updated in the future to be managed by page state.

And for invept emulation, just simply free all the mapping of shadow EPT now.

pKVM on ARM is using PV ops to directly manage stage-2 MMU page table in the hypervisor, while pKVM on Intel platform choose above EPT emulation solution - it increases the complexity by doing EPT shadowing but avoids changes in the KVM MMU code.

Chuanxiao Dong (12):
pkvm: x86: Pre-define the maximum number of supported VMs
pkvm: x86: init: Reserve memory for shadow EPT
pkvm: x86: Initialize the shadow EPT pool
pkvm: x86: Introduce shadow EPT
pkvm: x86: Introduce vEPT to record guest EPT information
pkvm: x86: Add API to get the max phys address bits
pkvm: x86: Initialize ept_zero_check
pkvm: x86: Add support for pKVM to handle the nested EPT violation
pkvm: x86: Introduce PKVM_ASSERT
pkvm: x86: add pkvm_pgtable_unmap_safe for a safe unmap
pkvm: x86: Add INVEPT instruction emulation
pkvm: x86: Switch to use shadow EPT for nested guests

Jason Chen CJ (1):
pkvm: x86: Introduce shadow EPT invalidation support

 arch/x86/include/asm/kvm_pkvm.h           |  34 ++++
 arch/x86/kvm/vmx/pkvm/hyp/bug.h           |  23 +++
 arch/x86/kvm/vmx/pkvm/hyp/ept.c           | 238 +++++++++++++++++++++-
 arch/x86/kvm/vmx/pkvm/hyp/ept.h           |  24 +++
 arch/x86/kvm/vmx/pkvm/hyp/init_finalise.c |  13 ++
 arch/x86/kvm/vmx/pkvm/hyp/memory.c        |  27 +++
 arch/x86/kvm/vmx/pkvm/hyp/memory.h        |   3 +
 arch/x86/kvm/vmx/pkvm/hyp/mmu.c           |   4 +-
 arch/x86/kvm/vmx/pkvm/hyp/mmu.h           |   3 +-
 arch/x86/kvm/vmx/pkvm/hyp/nested.c        | 191 +++++++++++++++++
 arch/x86/kvm/vmx/pkvm/hyp/nested.h        |   1 +
 arch/x86/kvm/vmx/pkvm/hyp/pgtable.c       |  67 ++++--
 arch/x86/kvm/vmx/pkvm/hyp/pgtable.h       |   4 +-
 arch/x86/kvm/vmx/pkvm/hyp/pkvm.c          |  36 +++-
 arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h      |  32 +++
 arch/x86/kvm/vmx/pkvm/hyp/vmexit.c        |   3 +
 arch/x86/kvm/vmx/pkvm/hyp/vmx.h           |  31 +++
 arch/x86/kvm/vmx/pkvm/pkvm_constants.c    |   3 +-
 arch/x86/kvm/vmx/pkvm/pkvm_host.c         |   3 +
 19 files changed, 711 insertions(+), 29 deletions(-)
 create mode 100644 arch/x86/kvm/vmx/pkvm/hyp/bug.h
Signature · 1 line
2.25.1
#1 lore raw

RFC PATCH 1/13 pkvm: x86: Pre-define the maximum number of supported VMs

From: Chuanxiao Dong <[email protected]>

Define the maximum number of the VMs pKVM can support. pKVM needs reserve related memory based on the VM number it support (e.g., shadow EPT page table pages). The reserved memory size for pKVM is calculated then exposed to pkvm_init through pkvm_constants definition PKVM_MAX_VM_NUM.

---
 arch/x86/kvm/vmx/pkvm/hyp/pkvm.c       | 2 +-
 arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h   | 3 +++
 arch/x86/kvm/vmx/pkvm/pkvm_constants.c | 1 +
 3 files changed, 5 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/pkvm/hyp/pkvm.c +1 -1
index 9efedba2b3c9..25904252d2dd 100644
@@ -10,7 +10,7 @@
 
 struct pkvm_hyp *pkvm_hyp;
 
-#define MAX_SHADOW_VMS	255
+#define MAX_SHADOW_VMS	(PKVM_MAX_NORMAL_VM_NUM + PKVM_MAX_PROTECTED_VM_NUM)
 #define HANDLE_OFFSET 1
 
 #define to_shadow_vm_handle(vcpu_handle)	((s64)(vcpu_handle) >> SHADOW_VM_HANDLE_SHIFT)
arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h +3 -0
index 82a59b5d7fd5..b7c3f8c478b4 100644
@@ -7,6 +7,9 @@
 
 #include <asm/pkvm_spinlock.h>
 
+#define PKVM_MAX_NORMAL_VM_NUM		8
+#define PKVM_MAX_PROTECTED_VM_NUM	2
+
 /*
  * A container for the vcpu state that hyp needs to maintain for protected VMs.
  */
arch/x86/kvm/vmx/pkvm/pkvm_constants.c +1 -0
index c6dc35b52664..6222a8fff6af 100644
@@ -15,5 +15,6 @@ int main(void)
 	DEFINE(PKVM_VMEMMAP_ENTRY_SIZE, sizeof(struct hyp_page));
 	DEFINE(PKVM_SHADOW_VM_SIZE, sizeof(struct pkvm_shadow_vm) + pkvm_shadow_vcpu_array_size());
 	DEFINE(PKVM_SHADOW_VCPU_STATE_SIZE, sizeof(struct shadow_vcpu_state));
+	DEFINE(PKVM_MAX_VM_NUM, PKVM_MAX_NORMAL_VM_NUM + PKVM_MAX_PROTECTED_VM_NUM);
 	return 0;
 }
Signature · 1 line
2.25.1
#2 lore raw

RFC PATCH 2/13 pkvm: x86: init: Reserve memory for shadow EPT

In reply to Jason Chen CJ

From: Chuanxiao Dong <[email protected]>

Shadow EPT pages are from the reserved memory of pKVM. The size for Shadow EPT pages is calculated based on the number of supported VMs (both normal VM and protected VM), with several assumptions:

- there is no shared memory between each VMs (normal or protected).
- all VMs total memory size is not larger than the platform memory size.
- the virtual MMIO range for each VM is not larger than 1G.

which makes the reserved memory size is calculable and predictable.

---
 arch/x86/include/asm/kvm_pkvm.h   | 34 +++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/pkvm/pkvm_host.c |  1 +
 2 files changed, 35 insertions(+)
arch/x86/include/asm/kvm_pkvm.h +34 -0
index 4e9531d88417..0712ef34b95b 100644
@@ -131,6 +131,40 @@ static inline int hyp_pre_reserve_check(void)
 	return 0;
 }
 
+/*
+ * Calculate the total pages for shadow EPT with assumptions:
+ * 1. there is no shared memory between each VMs (normal or protected).
+ * 2. all VMs total memory size is no larger than the platform memory size.
+ * 3. the virtual MMIO range for each VM is no larger than 1G.
+ */
+static inline unsigned long pkvm_shadow_ept_pgtable_pages(int nr_vm)
+{
+	unsigned long pgtable_pages = __pkvm_pgtable_total_pages();
+	unsigned long res;
+
+	res = pgtable_pages;
+
+	/*
+	 * Above 'res' covered enough pages for a shadow EPT to create all
+	 * possible levels mapping for the whole platform memory size. The pages
+	 * reserved for lvl-1 mapping (usually 4K size) actually can be shared
+	 * among all VM shadow EPTs as their occupied memory are partitioned
+	 * according to the assumption. While for lvl-2,3,4 or possible lvl-5
+	 * mappings, above 'res' pages only covered for one shadow EPT, we need
+	 * reserve more for other 'nr_vm -1' VMs.
+	 *
+	 * The lvl-2 to lvl-5 pages for one VM's shadow EPT can be approximately
+	 * got by:
+	 *     __pkvm_pgtable_total_pages()/PTRS_PER_PTE
+	 */
+	res += DIV_ROUND_UP(pgtable_pages, PTRS_PER_PTE) * (nr_vm - 1);
+
+	/* Allow 1 GiB for MMIO mappings for each VM */
+	 res += __pkvm_pgtable_max_pages(SZ_1G >> PAGE_SHIFT) * nr_vm;
+
+	return res;
+}
+
 u64 hyp_total_reserve_pages(void);
 
 int pkvm_init_shadow_vm(struct kvm *kvm);
arch/x86/kvm/vmx/pkvm/pkvm_host.c +1 -0
index 90d7cddde9ef..e5eab94f3e5e 100644
@@ -41,6 +41,7 @@ u64 hyp_total_reserve_pages(void)
 	total += pkvm_vmemmap_pages(PKVM_VMEMMAP_ENTRY_SIZE);
 	total += pkvm_mmu_pgtable_pages();
 	total += host_ept_pgtable_pages();
+	total += pkvm_shadow_ept_pgtable_pages(PKVM_MAX_VM_NUM);
 
 	return total;
 }
Signature · 1 line
2.25.1
#3 lore raw

RFC PATCH 3/13 pkvm: x86: Initialize the shadow EPT pool

In reply to Jason Chen CJ

From: Chuanxiao Dong <[email protected]>

Shadow EPT pool is the memory pool to allocate the memory for the shadow EPT paging structure pages. When destroy a shadow EPT, these pages will be put back to this pool again.

---
 arch/x86/kvm/vmx/pkvm/hyp/ept.c           |  9 +++++++++
 arch/x86/kvm/vmx/pkvm/hyp/ept.h           |  1 +
 arch/x86/kvm/vmx/pkvm/hyp/init_finalise.c | 13 +++++++++++++
 3 files changed, 23 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +9 -0
index b0a542b47e83..14bc8f4429db 100644
@@ -24,6 +24,8 @@ static struct hyp_pool host_ept_pool;
 static struct pkvm_pgtable host_ept;
 static pkvm_spinlock_t host_ept_lock = __PKVM_SPINLOCK_UNLOCKED;
 
+static struct hyp_pool shadow_ept_pool;
+
 static void flush_tlb_noop(void) { };
 static void *host_ept_zalloc_page(void)
 {
@@ -221,3 +223,10 @@ int handle_host_ept_violation(unsigned long gpa)
 	pkvm_spin_unlock(&host_ept_lock);
 	return ret;
 }
+
+int pkvm_shadow_ept_pool_init(void *ept_pool_base, unsigned long ept_pool_pages)
+{
+	unsigned long pfn = __pkvm_pa(ept_pool_base) >> PAGE_SHIFT;
+
+	return hyp_pool_init(&shadow_ept_pool, pfn, ept_pool_pages, 0);
+}
arch/x86/kvm/vmx/pkvm/hyp/ept.h +1 -0
index d517bf8ec169..c4ad5c269d5c 100644
@@ -17,5 +17,6 @@ int pkvm_host_ept_unmap(unsigned long vaddr_start, unsigned long phys_start,
 int pkvm_host_ept_init(struct pkvm_pgtable_cap *cap, void *ept_pool_base,
 		unsigned long ept_pool_pages);
 int handle_host_ept_violation(unsigned long gpa);
+int pkvm_shadow_ept_pool_init(void *ept_pool_base, unsigned long ept_pool_pages);
 
 #endif
arch/x86/kvm/vmx/pkvm/hyp/init_finalise.c +13 -0
index c16b53b7bcd0..8d52a20f6497 100644
@@ -23,6 +23,7 @@
 void *pkvm_mmu_pgt_base;
 void *pkvm_vmemmap_base;
 void *host_ept_pgt_base;
+static void *shadow_ept_base;
 
 static int divide_memory_pool(phys_addr_t phys, unsigned long size)
 {
@@ -49,6 +50,12 @@ static int divide_memory_pool(phys_addr_t phys, unsigned long size)
 	if (!host_ept_pgt_base)
 		return -ENOMEM;
 
+	nr_pages = pkvm_shadow_ept_pgtable_pages(PKVM_MAX_NORMAL_VM_NUM +
+						 PKVM_MAX_PROTECTED_VM_NUM);
+	shadow_ept_base = pkvm_early_alloc_contig(nr_pages);
+	if (!shadow_ept_base)
+		return -ENOMEM;
+
 	return 0;
 }
 
@@ -291,6 +298,12 @@ int __pkvm_init_finalise(struct kvm_vcpu *vcpu, struct pkvm_section sections[],
 
 	pkvm_init_nest();
 
+	ret = pkvm_shadow_ept_pool_init(shadow_ept_base,
+					pkvm_shadow_ept_pgtable_pages(PKVM_MAX_NORMAL_VM_NUM +
+								      PKVM_MAX_PROTECTED_VM_NUM));
+	if (ret)
+		goto out;
+
 	pkvm_init = true;
 
 switch_pgt:
Signature · 1 line
2.25.1
#4 lore raw

RFC PATCH 4/13 pkvm: x86: Introduce shadow EPT

In reply to Jason Chen CJ

From: Chuanxiao Dong <[email protected]>

KVM in the host VM (KVM-high) can create its guest (nested VM).

Currently the KVM-high created guest EPT for such nested VM is directly used in its physical VMCS. This means that, the KVM-high is able to map any memory in this guest EPT, which will make this nested VM able to see any memory, including the private memory(pKVM's memory and future supported protected VM's memory) which needed to be protected.

Using shadow EPT can resolve this kind of security issue. In this solution, pKVM maintains a shadow EPT for each nested VM. Shadow EPT is populated during handling nested EPT violation with checking the mapping in guest EPT through page state mechanism first (introduced in later patches), so that pKVM can guarantee only allowed memory can be mapped for a nested VM.

This patch introduces APIs to initialize shadow EPT for nested VMs. How to build the page table entries from virtual EPT (in host VM) will be introduced later.

FIXME:

The current design is that one VM only has one shadow EPT. This brings a security issue for a VM to use vSMM mode, which is that the vCPU in non-vSMM mode will be able to access the memory belongs to vSMM mode because all vCPU are sharing one single shadow EPT.

---
 arch/x86/kvm/vmx/pkvm/hyp/ept.c        | 67 ++++++++++++++++++++++++++
 arch/x86/kvm/vmx/pkvm/hyp/ept.h        |  4 ++
 arch/x86/kvm/vmx/pkvm/hyp/pkvm.c       |  7 +++
 arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h   | 20 ++++++++
 arch/x86/kvm/vmx/pkvm/hyp/vmx.h        | 13 +++++
 arch/x86/kvm/vmx/pkvm/pkvm_constants.c |  2 +-
 6 files changed, 112 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +67 -0
index 14bc8f4429db..0edea266b8bc 100644
@@ -18,6 +18,7 @@
 #include "pgtable.h"
 #include "ept.h"
 #include "memory.h"
+#include "vmx.h"
 #include "debug.h"
 
 static struct hyp_pool host_ept_pool;
@@ -230,3 +231,69 @@ int pkvm_shadow_ept_pool_init(void *ept_pool_base, unsigned long ept_pool_pages)
 
 	return hyp_pool_init(&shadow_ept_pool, pfn, ept_pool_pages, 0);
 }
+
+static void *shadow_ept_zalloc_page(void)
+{
+	return hyp_alloc_pages(&shadow_ept_pool, 0);
+}
+
+static void shadow_ept_get_page(void *vaddr)
+{
+	hyp_get_page(&shadow_ept_pool, vaddr);
+}
+
+static void shadow_ept_put_page(void *vaddr)
+{
+	hyp_put_page(&shadow_ept_pool, vaddr);
+}
+
+/*TODO: add tlb flush support for shadow ept */
+static struct pkvm_mm_ops shadow_ept_mm_ops = {
+	.phys_to_virt = pkvm_phys_to_virt,
+	.virt_to_phys = pkvm_virt_to_phys,
+	.zalloc_page = shadow_ept_zalloc_page,
+	.get_page = shadow_ept_get_page,
+	.put_page = shadow_ept_put_page,
+	.page_count = hyp_page_count,
+	.flush_tlb = flush_tlb_noop,
+};
+
+void pkvm_shadow_ept_deinit(struct shadow_ept_desc *desc)
+{
+	struct pkvm_pgtable *sept = &desc->sept;
+	struct pkvm_shadow_vm *vm = sept_desc_to_shadow_vm(desc);
+
+	pkvm_spin_lock(&vm->lock);
+
+	if (desc->shadow_eptp) {
+		pkvm_pgtable_destroy(sept);
+
+		flush_ept(desc->shadow_eptp);
+
+		memset(sept, 0, sizeof(struct pkvm_pgtable));
+		desc->shadow_eptp = 0;
+	}
+
+	pkvm_spin_unlock(&vm->lock);
+}
+
+int pkvm_shadow_ept_init(struct shadow_ept_desc *desc)
+{
+	struct pkvm_pgtable_cap cap = {
+		.level = pkvm_hyp->ept_cap.level,
+		.allowed_pgsz = pkvm_hyp->ept_cap.allowed_pgsz,
+		.table_prot = pkvm_hyp->ept_cap.table_prot,
+	};
+	int ret;
+
+	memset(desc, 0, sizeof(struct shadow_ept_desc));
+
+	ret = pkvm_pgtable_init(&desc->sept, &shadow_ept_mm_ops, &ept_ops, &cap, true);
+	if (ret)
+		return ret;
+
+	desc->shadow_eptp = pkvm_construct_eptp(desc->sept.root_pa, cap.level);
+	flush_ept(desc->shadow_eptp);
+
+	return 0;
+}
arch/x86/kvm/vmx/pkvm/hyp/ept.h +4 -0
index c4ad5c269d5c..7badcb3dd621 100644
@@ -5,6 +5,8 @@
 #ifndef __PKVM_EPT_H
 #define __PKVM_EPT_H
 
+#include "pkvm_hyp.h"
+
 #define HOST_EPT_DEF_MEM_PROT   (VMX_EPT_RWX_MASK |				\
 				(MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT))
 #define HOST_EPT_DEF_MMIO_PROT	(VMX_EPT_RWX_MASK |				\
@@ -18,5 +20,7 @@ int pkvm_host_ept_init(struct pkvm_pgtable_cap *cap, void *ept_pool_base,
 		unsigned long ept_pool_pages);
 int handle_host_ept_violation(unsigned long gpa);
 int pkvm_shadow_ept_pool_init(void *ept_pool_base, unsigned long ept_pool_pages);
+int pkvm_shadow_ept_init(struct shadow_ept_desc *desc);
+void pkvm_shadow_ept_deinit(struct shadow_ept_desc *desc);
 
 #endif
arch/x86/kvm/vmx/pkvm/hyp/pkvm.c +7 -0
index 25904252d2dd..321df4dd2998 100644
@@ -7,6 +7,7 @@
 #include <pkvm.h>
 
 #include "pkvm_hyp.h"
+#include "ept.h"
 
 struct pkvm_hyp *pkvm_hyp;
 
@@ -101,6 +102,10 @@ int __pkvm_init_shadow_vm(unsigned long kvm_va,
 	pkvm_spin_lock_init(&vm->lock);
 
 	vm->host_kvm_va = kvm_va;
+
+	if (pkvm_shadow_ept_init(&vm->sept_desc))
+		return -EINVAL;
+
 	return allocate_shadow_vm_handle(vm);
 }
 
@@ -111,6 +116,8 @@ unsigned long __pkvm_teardown_shadow_vm(int shadow_vm_handle)
 	if (!vm)
 		return 0;
 
+	pkvm_shadow_ept_deinit(&vm->sept_desc);
+
 	memset(vm, 0, sizeof(struct pkvm_shadow_vm) + pkvm_shadow_vcpu_array_size());
 
 	return pkvm_virt_to_phys(vm);
arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h +20 -0
index b7c3f8c478b4..a1f3644a4a34 100644
@@ -6,6 +6,17 @@
 #define __PKVM_HYP_H
 
 #include <asm/pkvm_spinlock.h>
+#include "pgtable.h"
+
+/*
+ * Descriptor for shadow EPT
+ */
+struct shadow_ept_desc {
+	/* shadow EPTP value configured by pkvm */
+	u64 shadow_eptp;
+
+	struct pkvm_pgtable sept;
+};
 
 #define PKVM_MAX_NORMAL_VM_NUM		8
 #define PKVM_MAX_PROTECTED_VM_NUM	2
@@ -76,9 +87,18 @@ struct pkvm_shadow_vm {
 	/* The host's kvm va. */
 	unsigned long host_kvm_va;
 
+	/*
+	 * VM's shadow EPT. All vCPU shares one mapping.
+	 * FIXME: a potential security issue if some vCPUs are
+	 * in SMM but the others are not.
+	 */
+	struct shadow_ept_desc sept_desc;
+
 	pkvm_spinlock_t lock;
 } __aligned(PAGE_SIZE);
 
+#define sept_desc_to_shadow_vm(desc) container_of(desc, struct pkvm_shadow_vm, sept_desc)
+
 int __pkvm_init_shadow_vm(unsigned long kvm_va, unsigned long shadow_pa,
 			  size_t shadow_size);
 unsigned long __pkvm_teardown_shadow_vm(int shadow_vm_handle);
arch/x86/kvm/vmx/pkvm/hyp/vmx.h +13 -0
index 54c17e256107..13405166bccf 100644
@@ -28,6 +28,11 @@ static inline bool vmx_ept_has_mt_wb(void)
 	return vmx_ept_capability_check(VMX_EPTP_WB_BIT);
 }
 
+static inline bool vmx_has_invept_context(void)
+{
+	return vmx_ept_capability_check(VMX_EPT_EXTENT_CONTEXT_BIT);
+}
+
 static inline u64 pkvm_construct_eptp(unsigned long root_hpa, int level)
 {
 	u64 eptp = 0;
@@ -71,6 +76,14 @@ static inline void vmcs_clear_track(struct vcpu_vmx *vmx, struct vmcs *vmcs)
 	vmcs_clear(vmcs);
 }
 
+static inline void flush_ept(u64 eptp)
+{
+	if (vmx_has_invept_context())
+		__invept(VMX_EPT_EXTENT_CONTEXT, eptp, 0);
+	else
+		__invept(VMX_EPT_EXTENT_GLOBAL, 0, 0);
+}
+
 void pkvm_init_host_state_area(struct pkvm_pcpu *pcpu, int cpu);
 
 #endif
arch/x86/kvm/vmx/pkvm/pkvm_constants.c +1 -1
index 6222a8fff6af..665d983040e8 100644
@@ -7,7 +7,7 @@
 #include <linux/bug.h>
 #include <vdso/limits.h>
 #include <buddy_memory.h>
-#include <vmx/vmx.h>
+#include <pkvm.h>
 #include "hyp/pkvm_hyp.h"
 
 int main(void)
Signature · 1 line
2.25.1
#5 lore raw

RFC PATCH 5/13 pkvm: x86: Introduce vEPT to record guest EPT information

In reply to Jason Chen CJ

From: Chuanxiao Dong <[email protected]>

KVM-high manages a guest EPT for its nested VM, which shall be shadowed to shadow EPT in pKVM hypervisor.

Introduce a per vcpu pgtable(vEPT) to save the guest EPT related information from KVM-high when vmwrite(EPTP).

Each time when a vcpu vmwrites to EPTP, pKVM will setup the vEPT according to the new value or just do nothing if it is the same with the last one.

---
 arch/x86/kvm/vmx/pkvm/hyp/ept.c      | 33 ++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/pkvm/hyp/ept.h      | 10 +++++++++
 arch/x86/kvm/vmx/pkvm/hyp/memory.c   |  6 +++++
 arch/x86/kvm/vmx/pkvm/hyp/memory.h   |  1 +
 arch/x86/kvm/vmx/pkvm/hyp/nested.c   | 16 ++++++++++++++
 arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h |  3 +++
 6 files changed, 69 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +33 -0
index 0edea266b8bc..641b8252071e 100644
@@ -297,3 +297,36 @@ int pkvm_shadow_ept_init(struct shadow_ept_desc *desc)
 
 	return 0;
 }
+
+/*
+ * virtual_ept_mm_ops is used as the ops for the ept constructed by
+ * KVM high in host.
+ * The physical address in this ept is the host VM GPA, which is
+ * the same with HPA.
+ */
+struct pkvm_mm_ops virtual_ept_mm_ops = {
+	.phys_to_virt = host_gpa2hva,
+};
+
+void pkvm_guest_ept_deinit(struct shadow_vcpu_state *shadow_vcpu)
+{
+	struct pkvm_pgtable *vept = &shadow_vcpu->vept;
+
+	memset(vept, 0, sizeof(struct pkvm_pgtable));
+}
+
+void pkvm_guest_ept_init(struct shadow_vcpu_state *shadow_vcpu, u64 guest_eptp)
+{
+	/*
+	 * TODO: we just assume guest will use page level the HW supported,
+	 * it actually need align with KVM high
+	 */
+	struct pkvm_pgtable_cap cap = {
+		.level = pkvm_hyp->ept_cap.level,
+		.allowed_pgsz = pkvm_hyp->ept_cap.allowed_pgsz,
+		.table_prot = pkvm_hyp->ept_cap.table_prot,
+	};
+
+	pkvm_pgtable_init(&shadow_vcpu->vept, &virtual_ept_mm_ops, &ept_ops, &cap, false);
+	shadow_vcpu->vept.root_pa = host_gpa2hpa(guest_eptp & SPTE_BASE_ADDR_MASK);
+}
arch/x86/kvm/vmx/pkvm/hyp/ept.h +10 -0
index 7badcb3dd621..420c9c5816e9 100644
@@ -22,5 +22,15 @@ int handle_host_ept_violation(unsigned long gpa);
 int pkvm_shadow_ept_pool_init(void *ept_pool_base, unsigned long ept_pool_pages);
 int pkvm_shadow_ept_init(struct shadow_ept_desc *desc);
 void pkvm_shadow_ept_deinit(struct shadow_ept_desc *desc);
+void pkvm_guest_ept_init(struct shadow_vcpu_state *shadow_vcpu, u64 guest_eptp);
+void pkvm_guest_ept_deinit(struct shadow_vcpu_state *shadow_vcpu);
 
+static inline bool is_valid_eptp(u64 eptp)
+{
+	if (!eptp || (eptp == INVALID_GPA))
+		return false;
+
+	/* TODO: other bits check */
+	return true;
+}
 #endif
arch/x86/kvm/vmx/pkvm/hyp/memory.c +6 -0
index 6a400aef1bd8..43fc39d44c3d 100644
@@ -73,6 +73,12 @@ void *host_gpa2hva(unsigned long gpa)
 	return pkvm_phys_to_virt(gpa);
 }
 
+unsigned long host_gpa2hpa(unsigned long gpa)
+{
+	/* Host VM is using identity mapping so GPA == HPA */
+	return gpa;
+}
+
 extern struct pkvm_pgtable_ops mmu_ops;
 static struct pkvm_mm_ops mm_ops = {
 	.phys_to_virt = host_gpa2hva,
arch/x86/kvm/vmx/pkvm/hyp/memory.h +1 -0
index 4a75d8dff1b3..a95ae5f71841 100644
@@ -22,6 +22,7 @@ bool mem_range_included(struct mem_range *child, struct mem_range *parent);
 
 #include <linux/kvm_host.h>
 void *host_gpa2hva(unsigned long gpa);
+unsigned long host_gpa2hpa(unsigned long gpa);
 int gva2gpa(struct kvm_vcpu *vcpu, gva_t gva, gpa_t *gpa,
 		u32 access, struct x86_exception *exception);
 int read_gva(struct kvm_vcpu *vcpu, gva_t gva, void *addr,
arch/x86/kvm/vmx/pkvm/hyp/nested.c +16 -0
index 429bfe7bb309..68eddb459cfa 100644
@@ -9,6 +9,7 @@
 #include "nested.h"
 #include "cpu.h"
 #include "vmx.h"
+#include "ept.h"
 #include "debug.h"
 
 /*
@@ -704,6 +705,18 @@ static void nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)
 	}
 }
 
+static void setup_guest_ept(struct shadow_vcpu_state *shadow_vcpu, u64 guest_eptp)
+{
+	struct vmcs12 *vmcs12 = (struct vmcs12 *)shadow_vcpu->cached_vmcs12;
+
+	if (!is_valid_eptp(guest_eptp))
+		pkvm_guest_ept_deinit(shadow_vcpu);
+	else if (vmcs12->ept_pointer != guest_eptp) {
+		pkvm_guest_ept_deinit(shadow_vcpu);
+		pkvm_guest_ept_init(shadow_vcpu, guest_eptp);
+	}
+}
+
 int handle_vmxon(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -908,6 +921,9 @@ int handle_vmwrite(struct kvm_vcpu *vcpu)
 			if (field >= GUEST_ES_AR_BYTES && field <= GUEST_TR_AR_BYTES)
 				value &= 0x1f0ff;
 
+			if (field == EPT_POINTER)
+				setup_guest_ept(cur_shadow_vcpu, value);
+
 			vmcs12_write_any(vmcs12, field, offset, value);
 
 			if (is_emulated_fields(field)) {
arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h +3 -0
index a1f3644a4a34..f891660d9085 100644
@@ -41,6 +41,9 @@ struct shadow_vcpu_state {
 
 	struct vcpu_vmx vmx;
 
+	/* represents for the virtual EPT configured by kvm-high */
+	struct pkvm_pgtable vept;
+
 	/* assume vmcs02 is one page */
 	u8 vmcs02[PAGE_SIZE] __aligned(PAGE_SIZE);
 	u8 cached_vmcs12[VMCS12_SIZE] __aligned(PAGE_SIZE);
Signature · 1 line
2.25.1
#6 lore raw

RFC PATCH 6/13 pkvm: x86: Add API to get the max phys address bits

In reply to Jason Chen CJ

From: Chuanxiao Dong <[email protected]>

Add API get_max_physaddr_bits(), this is necessary used to calculate the reserved bits for EPT entry.

---
 arch/x86/kvm/vmx/pkvm/hyp/memory.c | 21 +++++++++++++++++++++
 arch/x86/kvm/vmx/pkvm/hyp/memory.h |  2 ++
 2 files changed, 23 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/memory.c +21 -0
index 43fc39d44c3d..9a9616ac4cc3 100644
@@ -10,9 +10,11 @@
 #include "memory.h"
 #include "pgtable.h"
 #include "pkvm_hyp.h"
+#include "cpu.h"
 
 unsigned long __page_base_offset;
 unsigned long __symbol_base_offset;
+static u8 max_physaddr_bits;
 
 unsigned int hyp_memblock_nr;
 struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS];
@@ -285,3 +287,22 @@ int write_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, void *addr, unsigned int bytes)
 {
 	return copy_gpa(vcpu, gpa, addr, bytes, false);
 }
+
+u64 get_max_physaddr_bits(void)
+{
+	u32 eax, ebx, ecx, edx;
+
+	if (max_physaddr_bits)
+		return max_physaddr_bits;
+
+	eax = 0x80000000;
+	ecx = 0;
+	native_cpuid(&eax, &ebx, &ecx, &edx);
+	if (eax >= 0x80000008) {
+		eax = 0x80000008;
+		native_cpuid(&eax, &ebx, &ecx, &edx);
+		max_physaddr_bits = (u8)eax & 0xff;
+	}
+
+	return max_physaddr_bits;
+}
arch/x86/kvm/vmx/pkvm/hyp/memory.h +2 -0
index a95ae5f71841..b1062f3703e3 100644
@@ -32,4 +32,6 @@ int write_gva(struct kvm_vcpu *vcpu, gva_t gva, void *addr,
 int read_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, void *addr, unsigned int bytes);
 int write_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, void *addr, unsigned int bytes);
 
+u64 get_max_physaddr_bits(void);
+
 #endif
Signature · 1 line
2.25.1
#7 lore raw

RFC PATCH 7/13 pkvm: x86: Initialize ept_zero_check

In reply to Jason Chen CJ

From: Chuanxiao Dong <[email protected]>

ept_zero_check is used to check if a pte entry has any reserved bits set, or has any unsupported bit combinations. This is helpful to check if this entry can cause EPT misconfig or not during shadowing in the later patch.

---
 arch/x86/kvm/vmx/pkvm/hyp/ept.c | 47 +++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/pkvm/hyp/vmx.h |  7 +++++
 2 files changed, 54 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +47 -0
index 641b8252071e..823e255de155 100644
@@ -26,6 +26,7 @@ static struct pkvm_pgtable host_ept;
 static pkvm_spinlock_t host_ept_lock = __PKVM_SPINLOCK_UNLOCKED;
 
 static struct hyp_pool shadow_ept_pool;
+static struct rsvd_bits_validate ept_zero_check;
 
 static void flush_tlb_noop(void) { };
 static void *host_ept_zalloc_page(void)
@@ -151,16 +152,62 @@ int pkvm_host_ept_unmap(unsigned long vaddr_start, unsigned long phys_start,
 	return pkvm_pgtable_unmap(&host_ept, vaddr_start, phys_start, size);
 }
 
+static void reset_rsvds_bits_mask_ept(struct rsvd_bits_validate *rsvd_check,
+				      u64 pa_bits_rsvd, bool execonly,
+				      int huge_page_level)
+{
+	u64 high_bits_rsvd = pa_bits_rsvd & rsvd_bits(0, 51);
+	u64 large_1g_rsvd = 0, large_2m_rsvd = 0;
+	u64 bad_mt_xwr;
+
+	if (huge_page_level < PG_LEVEL_1G)
+		large_1g_rsvd = rsvd_bits(7, 7);
+	if (huge_page_level < PG_LEVEL_2M)
+		large_2m_rsvd = rsvd_bits(7, 7);
+
+	rsvd_check->rsvd_bits_mask[0][4] = high_bits_rsvd | rsvd_bits(3, 7);
+	rsvd_check->rsvd_bits_mask[0][3] = high_bits_rsvd | rsvd_bits(3, 7);
+	rsvd_check->rsvd_bits_mask[0][2] = high_bits_rsvd | rsvd_bits(3, 6) | large_1g_rsvd;
+	rsvd_check->rsvd_bits_mask[0][1] = high_bits_rsvd | rsvd_bits(3, 6) | large_2m_rsvd;
+	rsvd_check->rsvd_bits_mask[0][0] = high_bits_rsvd;
+
+	/* large page */
+	rsvd_check->rsvd_bits_mask[1][4] = rsvd_check->rsvd_bits_mask[0][4];
+	rsvd_check->rsvd_bits_mask[1][3] = rsvd_check->rsvd_bits_mask[0][3];
+	rsvd_check->rsvd_bits_mask[1][2] = high_bits_rsvd | rsvd_bits(12, 29) | large_1g_rsvd;
+	rsvd_check->rsvd_bits_mask[1][1] = high_bits_rsvd | rsvd_bits(12, 20) | large_2m_rsvd;
+	rsvd_check->rsvd_bits_mask[1][0] = rsvd_check->rsvd_bits_mask[0][0];
+
+	bad_mt_xwr = 0xFFull << (2 * 8);	/* bits 3..5 must not be 2 */
+	bad_mt_xwr |= 0xFFull << (3 * 8);	/* bits 3..5 must not be 3 */
+	bad_mt_xwr |= 0xFFull << (7 * 8);	/* bits 3..5 must not be 7 */
+	bad_mt_xwr |= REPEAT_BYTE(1ull << 2);	/* bits 0..2 must not be 010 */
+	bad_mt_xwr |= REPEAT_BYTE(1ull << 6);	/* bits 0..2 must not be 110 */
+	if (!execonly) {
+		/* bits 0..2 must not be 100 unless VMX capabilities allow it */
+		bad_mt_xwr |= REPEAT_BYTE(1ull << 4);
+	}
+	rsvd_check->bad_mt_xwr = bad_mt_xwr;
+}
+
 int pkvm_host_ept_init(struct pkvm_pgtable_cap *cap,
 		void *ept_pool_base, unsigned long ept_pool_pages)
 {
 	unsigned long pfn = __pkvm_pa(ept_pool_base) >> PAGE_SHIFT;
 	int ret;
+	u8 pa_bits;
 
 	ret = hyp_pool_init(&host_ept_pool, pfn, ept_pool_pages, 0);
 	if (ret)
 		return ret;
 
+	pa_bits = get_max_physaddr_bits();
+	if (!pa_bits)
+		return -EINVAL;
+	reset_rsvds_bits_mask_ept(&ept_zero_check, rsvd_bits(pa_bits, 63),
+				  vmx_has_ept_execute_only(),
+				  fls(cap->allowed_pgsz) - 1);
+
 	pkvm_hyp->host_vm.ept = &host_ept;
 	return pkvm_pgtable_init(&host_ept, &host_ept_mm_ops, &ept_ops, cap, true);
 }
arch/x86/kvm/vmx/pkvm/hyp/vmx.h +7 -0
index 13405166bccf..3282f228964d 100644
@@ -6,6 +6,8 @@
 #ifndef __PKVM_VMX_H
 #define __PKVM_VMX_H
 
+#include "pkvm_hyp.h"
+
 static inline bool vmx_ept_capability_check(u32 bit)
 {
 	struct vmx_capability *vmx_cap = &pkvm_hyp->vmx_cap;
@@ -33,6 +35,11 @@ static inline bool vmx_has_invept_context(void)
 	return vmx_ept_capability_check(VMX_EPT_EXTENT_CONTEXT_BIT);
 }
 
+static inline bool vmx_has_ept_execute_only(void)
+{
+	return vmx_ept_capability_check(VMX_EPT_EXECUTE_ONLY_BIT);
+}
+
 static inline u64 pkvm_construct_eptp(unsigned long root_hpa, int level)
 {
 	u64 eptp = 0;
Signature · 1 line
2.25.1
#8 lore raw

RFC PATCH 8/13 pkvm: x86: Add support for pKVM to handle the nested EPT violation

In reply to Jason Chen CJ

From: Chuanxiao Dong <[email protected]>

Shadow EPT is used as EPTP for nested VMs and it's initialized with empty entries. So when EPT violation happened, pKVM shall shadowing EPT page table entry from guest vEPT under well-designed rules.

pKVM walks the guest vEPT to find out if the L2 GPA already has a proper mapping in vEPT. If already has one, pKVM map it in the shadow EPT - now just directly do the same mapping as vEPT, in the future, pKVM shall follow some rules to do the mapping, like only after confirming the page's ownership belong to this VM.

For other cases non-proper mapping in vEPT, they actually seek for the handling from host VM (or more accurate from KVM-high):

- for the case vEPT is not present, it's actually a #PF and requesting page allocation from the host, so pKVM directly forward the violation back to the KVM-high for proper handling.

- and for the case that vEPT contains a misconfig mapping, such misconfig is set by KVM-high so it expected to handle it. pKVM modify VM_EXIT_REASON field to EPT_MISCONFIG if VMX support read-only field modification, then deliver back to KVM-high. Otherwise deliver EPT_VIOLATION to KVM-high directly as KVM handle EPT_MISCONFIG in its EPT_VIOLATION handler as well.

- another case is that if the EPT violation vmexit has pending event to be injected, such event also seeks for host handling to do further event injection, pKVM shall return back to the KVM-high with corresponding pending event vmexit info.

---
 arch/x86/kvm/vmx/pkvm/hyp/ept.c    | 60 ++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/pkvm/hyp/ept.h    |  8 ++++
 arch/x86/kvm/vmx/pkvm/hyp/nested.c | 40 ++++++++++++++++++++
 arch/x86/kvm/vmx/pkvm/hyp/vmx.h    |  6 +++
 arch/x86/kvm/vmx/pkvm/pkvm_host.c  |  2 +
 5 files changed, 116 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +60 -0
index 823e255de155..65f3a39210db 100644
@@ -377,3 +377,63 @@ void pkvm_guest_ept_init(struct shadow_vcpu_state *shadow_vcpu, u64 guest_eptp)
 	pkvm_pgtable_init(&shadow_vcpu->vept, &virtual_ept_mm_ops, &ept_ops, &cap, false);
 	shadow_vcpu->vept.root_pa = host_gpa2hpa(guest_eptp & SPTE_BASE_ADDR_MASK);
 }
+
+static bool is_access_violation(u64 ept_entry, u64 exit_qual)
+{
+	bool access_violation = false;
+
+	if (/* Caused by data read */
+	    (((exit_qual & 0x1UL) != 0UL) && ((ept_entry & VMX_EPT_READABLE_MASK) == 0)) ||
+	    /* Caused by data write */
+	    (((exit_qual & 0x2UL) != 0UL) && ((ept_entry & VMX_EPT_WRITABLE_MASK) == 0)) ||
+	    /* Caused by instruction fetch */
+	    (((exit_qual & 0x4UL) != 0UL) && ((ept_entry & VMX_EPT_EXECUTABLE_MASK) == 0))) {
+		access_violation = true;
+	}
+
+	return access_violation;
+}
+
+enum sept_handle_ret
+pkvm_handle_shadow_ept_violation(struct shadow_vcpu_state *shadow_vcpu, u64 l2_gpa, u64 exit_quali)
+{
+	struct pkvm_shadow_vm *vm = shadow_vcpu->vm;
+	struct shadow_ept_desc *desc = &vm->sept_desc;
+	struct pkvm_pgtable *sept = &desc->sept;
+	struct pkvm_pgtable_ops *pgt_ops = sept->pgt_ops;
+	struct pkvm_pgtable *vept = &shadow_vcpu->vept;
+	enum sept_handle_ret ret = PKVM_NOT_HANDLED;
+	unsigned long phys;
+	int level;
+	u64 gprot, rsvd_chk_gprot;
+
+	pkvm_spin_lock(&vm->lock);
+
+	pkvm_pgtable_lookup(vept, l2_gpa, &phys, &gprot, &level);
+	if (phys == INVALID_ADDR)
+		/* Geust EPT not valid, back to kvm-high */
+		goto out;
+
+	if (is_access_violation(gprot, exit_quali))
+		/* Guest EPT error, refuse to handle in shadow ept */
+		goto out;
+
+	rsvd_chk_gprot = gprot;
+	/* is_rsvd_spte() need based on PAGE_SIZE bit */
+	if (level != PG_LEVEL_4K)
+		pgt_ops->pgt_entry_mkhuge(&rsvd_chk_gprot);
+
+	if (is_rsvd_spte(&ept_zero_check, rsvd_chk_gprot, level)) {
+		ret = PKVM_INJECT_EPT_MISC;
+	} else {
+		unsigned long level_size = pgt_ops->pgt_level_to_size(level);
+		unsigned long gpa = ALIGN_DOWN(l2_gpa, level_size);
+		unsigned long hpa = ALIGN_DOWN(host_gpa2hpa(phys), level_size);
+
+		if (!pkvm_pgtable_map(sept, gpa, hpa, level_size, 0, gprot))
+			ret = PKVM_HANDLED;
+	}
+out:
+	pkvm_spin_unlock(&vm->lock);
+	return ret;
+}
arch/x86/kvm/vmx/pkvm/hyp/ept.h +8 -0
index 420c9c5816e9..92a4f18535ea 100644
@@ -12,6 +12,12 @@
 #define HOST_EPT_DEF_MMIO_PROT	(VMX_EPT_RWX_MASK |				\
 				(MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT))
 
+enum sept_handle_ret {
+	PKVM_NOT_HANDLED,
+	PKVM_HANDLED,
+	PKVM_INJECT_EPT_MISC,
+};
+
 int pkvm_host_ept_map(unsigned long vaddr_start, unsigned long phys_start,
 		unsigned long size, int pgsz_mask, u64 prot);
 int pkvm_host_ept_unmap(unsigned long vaddr_start, unsigned long phys_start,
@@ -24,6 +30,8 @@ int pkvm_shadow_ept_init(struct shadow_ept_desc *desc);
 void pkvm_shadow_ept_deinit(struct shadow_ept_desc *desc);
 void pkvm_guest_ept_init(struct shadow_vcpu_state *shadow_vcpu, u64 guest_eptp);
 void pkvm_guest_ept_deinit(struct shadow_vcpu_state *shadow_vcpu);
+enum sept_handle_ret
+pkvm_handle_shadow_ept_violation(struct shadow_vcpu_state *shadow_vcpu, u64 l2_gpa, u64 exit_quali);
 
 static inline bool is_valid_eptp(u64 eptp)
 {
arch/x86/kvm/vmx/pkvm/hyp/nested.c +40 -0
index 68eddb459cfa..22c161100145 100644
@@ -1007,6 +1007,39 @@ int handle_vmlaunch(struct kvm_vcpu *vcpu)
 	return 0;
 }
 
+static bool nested_handle_ept_violation(struct shadow_vcpu_state *shadow_vcpu,
+					u64 l2_gpa, u64 exit_quali)
+{
+	enum sept_handle_ret ret = pkvm_handle_shadow_ept_violation(shadow_vcpu,
+								    l2_gpa, exit_quali);
+	bool handled = false;
+
+	switch (ret) {
+	case PKVM_INJECT_EPT_MISC: {
+		/*
+		 * Inject EPT_MISCONFIG vmexit reason if can directly modify
+		 * the read-only fields. Otherwise still deliver EPT_VIOLATION
+		 * for simplification.
+		 */
+		if (vmx_has_vmwrite_any_field())
+			vmcs_write32(VM_EXIT_REASON, EXIT_REASON_EPT_MISCONFIG);
+		break;
+	}
+	case PKVM_HANDLED:
+		handled = true;
+		break;
+	default:
+		break;
+	}
+
+	if (handled && (vmcs_read32(IDT_VECTORING_INFO_FIELD) &
+			VECTORING_INFO_VALID_MASK))
+		/* pending interrupt, back to kvm-high to inject */
+		handled = false;
+
+	return handled;
+}
+
 int nested_vmexit(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -1015,6 +1048,13 @@ int nested_vmexit(struct kvm_vcpu *vcpu)
 	struct vmcs *vmcs02 = (struct vmcs *)cur_shadow_vcpu->vmcs02;
 	struct vmcs12 *vmcs12 = (struct vmcs12 *)cur_shadow_vcpu->cached_vmcs12;
 
+	if ((vmx->exit_reason.full == EXIT_REASON_EPT_VIOLATION) &&
+		nested_handle_ept_violation(cur_shadow_vcpu,
+					    vmcs_read64(GUEST_PHYSICAL_ADDRESS),
+					    vmx->exit_qualification))
+		/* EPT violation can be handled by pkvm, no need back to kvm-high */
+		return 0;
+
 	/* clear guest mode if need switch back to host */
 	vcpu->arch.hflags &= ~HF_GUEST_MASK;
 
arch/x86/kvm/vmx/pkvm/hyp/vmx.h +6 -0
index 3282f228964d..463780776666 100644
@@ -40,6 +40,12 @@ static inline bool vmx_has_ept_execute_only(void)
 	return vmx_ept_capability_check(VMX_EPT_EXECUTE_ONLY_BIT);
 }
 
+static inline bool vmx_has_vmwrite_any_field(void)
+{
+	return !!(pkvm_hyp->vmcs_config.nested.misc_low &
+			MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS);
+}
+
 static inline u64 pkvm_construct_eptp(unsigned long root_hpa, int level)
 {
 	u64 eptp = 0;
arch/x86/kvm/vmx/pkvm/pkvm_host.c +2 -0
index e5eab94f3e5e..498e304cfb94 100644
@@ -380,6 +380,8 @@ static __init void pkvm_host_setup_nested_vmx_cap(struct pkvm_hyp *pkvm)
 	rdmsr(MSR_IA32_VMX_ENTRY_CTLS,
 		msrs->entry_ctls_low,
 		msrs->entry_ctls_high);
+
+	rdmsr(MSR_IA32_VMX_MISC, msrs->misc_low, msrs->misc_high);
 }
 
 static __init int pkvm_host_check_and_setup_vmx_cap(struct pkvm_hyp *pkvm)
Signature · 1 line
2.25.1
#9 lore raw

RFC PATCH 9/13 pkvm: x86: Introduce PKVM_ASSERT

In reply to Jason Chen CJ

From: Chuanxiao Dong <[email protected]>

Add new file bug.h to support PKVM_ASSERT.

If CONFIG_PKVM_INTEL_DEBUG enabled, print out assertion information then call into Linux BUG(), otherwise just do a dead loop.

---
 arch/x86/kvm/vmx/pkvm/hyp/bug.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/bug.h new file +23 -0
new file mode 100644
index 000000000000..ca333708cc98
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2022 Intel Corporation
+ */
+#ifndef __PKVM_BUG_H
+#define __PKVM_BUG_H
+
+#ifdef CONFIG_PKVM_INTEL_DEBUG
+#include <linux/printk.h>
+
+#define PKVM_ASSERT(c)                                          \
+do {                                                            \
+	if (!(c)) {                                             \
+		pr_err("assertion failed %s: %d: %s\n",         \
+			__FILE__, __LINE__, #c);                \
+		BUG();                                          \
+	}                                                       \
+} while (0)
+#else
+#define PKVM_ASSERT(c) do { } while (!(c))
+#endif
+
+#endif
Signature · 1 line
2.25.1
#10 lore raw

RFC PATCH 10/13 pkvm: x86: add pkvm_pgtable_unmap_safe for a safe unmap

In reply to Jason Chen CJ

From: Chuanxiao Dong <[email protected]>

The current pkvm_pgtable_unmap API takes a phys parameter which will check if the unmap entry really maps this phys memory. This is not necessary for all the unmap cases, and actually phys is unknown to some cases.

However, for some cases, still prefer to perform such check to avoid unmapping an incorrect entry because such failure is not easy to be detected but cause a serial security issue. The unmap from page state API and from the host EPT are the cases better to have such check. So introduce a safe version unmap API.

The safe version unmap API will be used by page state machine and host EPT to do unmap. For the MMU page table, it can still use the normal version of the unmap.

---
 arch/x86/kvm/vmx/pkvm/hyp/ept.c     |  4 +-
 arch/x86/kvm/vmx/pkvm/hyp/mmu.c     |  4 +-
 arch/x86/kvm/vmx/pkvm/hyp/mmu.h     |  3 +-
 arch/x86/kvm/vmx/pkvm/hyp/pgtable.c | 67 ++++++++++++++++++++---------
 arch/x86/kvm/vmx/pkvm/hyp/pgtable.h |  4 +-
 5 files changed, 55 insertions(+), 27 deletions(-)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +2 -2
index 65f3a39210db..a0793e4d02ef 100644
@@ -147,9 +147,9 @@ int pkvm_host_ept_map(unsigned long vaddr_start, unsigned long phys_start,
 }
 
 int pkvm_host_ept_unmap(unsigned long vaddr_start, unsigned long phys_start,
-		unsigned long size)
+			unsigned long size)
 {
-	return pkvm_pgtable_unmap(&host_ept, vaddr_start, phys_start, size);
+	return pkvm_pgtable_unmap_safe(&host_ept, vaddr_start, phys_start, size);
 }
 
 static void reset_rsvds_bits_mask_ept(struct rsvd_bits_validate *rsvd_check,
arch/x86/kvm/vmx/pkvm/hyp/mmu.c +2 -2
index b32ca706fa4b..f2fe8aa45b46 100644
@@ -192,11 +192,11 @@ int pkvm_mmu_map(unsigned long vaddr_start, unsigned long phys_start,
 	return ret;
 }
 
-int pkvm_mmu_unmap(unsigned long vaddr_start, unsigned long phys_start, unsigned long size)
+int pkvm_mmu_unmap(unsigned long vaddr_start, unsigned long size)
 {
 	int ret;
 
-	ret = pkvm_pgtable_unmap(&hyp_mmu, vaddr_start, phys_start, size);
+	ret = pkvm_pgtable_unmap(&hyp_mmu, vaddr_start, size);
 
 	return ret;
 }
arch/x86/kvm/vmx/pkvm/hyp/mmu.h +1 -2
index 218e3d4ef92c..776e6ceebd4c 100644
@@ -8,8 +8,7 @@
 int pkvm_mmu_map(unsigned long vaddr_start, unsigned long phys_start,
 		unsigned long size, int pgsz_mask, u64 prot);
 
-int pkvm_mmu_unmap(unsigned long vaddr_start, unsigned long phys_start,
-		unsigned long size);
+int pkvm_mmu_unmap(unsigned long vaddr_start, unsigned long size);
 
 int pkvm_early_mmu_init(struct pkvm_pgtable_cap *cap,
 		void *mmu_pool_base, unsigned long mmu_pool_pages);
arch/x86/kvm/vmx/pkvm/hyp/pgtable.c +47 -20

109 lines not on this page · raw message

arch/x86/kvm/vmx/pkvm/hyp/pgtable.h +3 -1
index 00d3742b7f48..61ee00ee07af 100644
@@ -74,7 +74,9 @@ int pkvm_pgtable_map(struct pkvm_pgtable *pgt, unsigned long vaddr_start,
 		unsigned long phys_start, unsigned long size,
 		int pgsz_mask, u64 entry_prot);
 int pkvm_pgtable_unmap(struct pkvm_pgtable *pgt, unsigned long vaddr_start,
-		unsigned long phys_start, unsigned long size);
+		       unsigned long size);
+int pkvm_pgtable_unmap_safe(struct pkvm_pgtable *pgt, unsigned long vaddr_start,
+			    unsigned long phys_start, unsigned long size);
 void pkvm_pgtable_lookup(struct pkvm_pgtable *pgt, unsigned long vaddr,
 		unsigned long *pphys, u64 *pprot, int *plevel);
 void pkvm_pgtable_destroy(struct pkvm_pgtable *pgt);
Signature · 1 line
2.25.1
#11 lore raw

RFC PATCH 11/13 pkvm: x86: Introduce shadow EPT invalidation support

In reply to Jason Chen CJ

Shadow EPT invalidation is required to emulate INVEPT.

And as a VM only has one shadow EPT, when vEPTP got changed, the shadow EPT shall be invalidated as well.

The invalidation is implemented by just simply destroying all the existing mappings for corresponding shadow EPT without freeing the root.

---
 arch/x86/kvm/vmx/pkvm/hyp/ept.c      | 18 ++++++++++++++++++
 arch/x86/kvm/vmx/pkvm/hyp/ept.h      |  1 +
 arch/x86/kvm/vmx/pkvm/hyp/nested.c   | 12 ++++++++++++
 arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h |  3 +++
 4 files changed, 34 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +18 -0
index a0793e4d02ef..de68f8c9eeb0 100644
@@ -305,6 +305,24 @@ static struct pkvm_mm_ops shadow_ept_mm_ops = {
 	.flush_tlb = flush_tlb_noop,
 };
 
+void pkvm_invalidate_shadow_ept(struct shadow_ept_desc *desc)
+{
+	struct pkvm_shadow_vm *vm = sept_desc_to_shadow_vm(desc);
+	struct pkvm_pgtable *sept = &desc->sept;
+	unsigned long size = sept->pgt_ops->pgt_level_to_size(sept->level + 1);
+
+	pkvm_spin_lock(&vm->lock);
+
+	if (!is_valid_eptp(desc->shadow_eptp))
+		goto out;
+
+	pkvm_pgtable_unmap(sept, 0, size);
+
+	flush_ept(desc->shadow_eptp);
+out:
+	pkvm_spin_unlock(&vm->lock);
+}
+
 void pkvm_shadow_ept_deinit(struct shadow_ept_desc *desc)
 {
 	struct pkvm_pgtable *sept = &desc->sept;
arch/x86/kvm/vmx/pkvm/hyp/ept.h +1 -0
index 92a4f18535ea..f63538368746 100644
@@ -32,6 +32,7 @@ void pkvm_guest_ept_init(struct shadow_vcpu_state *shadow_vcpu, u64 guest_eptp);
 void pkvm_guest_ept_deinit(struct shadow_vcpu_state *shadow_vcpu);
 enum sept_handle_ret
 pkvm_handle_shadow_ept_violation(struct shadow_vcpu_state *shadow_vcpu, u64 l2_gpa, u64 exit_quali);
+void pkvm_invalidate_shadow_ept(struct shadow_ept_desc *desc);
 
 static inline bool is_valid_eptp(u64 eptp)
 {
arch/x86/kvm/vmx/pkvm/hyp/nested.c +12 -0
index 22c161100145..8b9202ecafff 100644
@@ -708,6 +708,8 @@ static void nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)
 static void setup_guest_ept(struct shadow_vcpu_state *shadow_vcpu, u64 guest_eptp)
 {
 	struct vmcs12 *vmcs12 = (struct vmcs12 *)shadow_vcpu->cached_vmcs12;
+	struct pkvm_shadow_vm *vm = shadow_vcpu->vm;
+	bool invalidate = false;
 
 	if (!is_valid_eptp(guest_eptp))
 		pkvm_guest_ept_deinit(shadow_vcpu);
@@ -715,6 +717,16 @@ static void setup_guest_ept(struct shadow_vcpu_state *shadow_vcpu, u64 guest_ept
 		pkvm_guest_ept_deinit(shadow_vcpu);
 		pkvm_guest_ept_init(shadow_vcpu, guest_eptp);
 	}
+
+	pkvm_spin_lock(&vm->lock);
+	if (vm->sept_desc.last_guest_eptp != guest_eptp) {
+		vm->sept_desc.last_guest_eptp = guest_eptp;
+		invalidate = true;
+	}
+	pkvm_spin_unlock(&vm->lock);
+
+	if (invalidate)
+		pkvm_invalidate_shadow_ept(&vm->sept_desc);
 }
 
 int handle_vmxon(struct kvm_vcpu *vcpu)
arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h +3 -0
index f891660d9085..cc7ec8505a98 100644
@@ -15,6 +15,9 @@ struct shadow_ept_desc {
 	/* shadow EPTP value configured by pkvm */
 	u64 shadow_eptp;
 
+	/* Save the last guest EPTP value configured by kvm high */
+	u64 last_guest_eptp;
+
 	struct pkvm_pgtable sept;
 };
 
Signature · 1 line
2.25.1
#12 lore raw

RFC PATCH 12/13 pkvm: x86: Add INVEPT instruction emulation

In reply to Jason Chen CJ

From: Chuanxiao Dong <[email protected]>

For INVEPT with single context, look up the vCPU vmcs12 to find out the corresponding shadow EPT to do the invalidation.

For INVEPT with global context, invalidate all the valid shadow EPTs.

TODO: current emulation may cause a lot of unnecessary shadow EPT violation due to inappropriate EPT invalidation as all page table entries are freed. Optimization shall be done in the future to do PV INVEPT with specific range.

---
 arch/x86/kvm/vmx/pkvm/hyp/nested.c   | 106 +++++++++++++++++++++++++++
 arch/x86/kvm/vmx/pkvm/hyp/nested.h   |   1 +
 arch/x86/kvm/vmx/pkvm/hyp/pkvm.c     |  27 +++++++
 arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h |   3 +
 arch/x86/kvm/vmx/pkvm/hyp/vmexit.c   |   3 +
 arch/x86/kvm/vmx/pkvm/hyp/vmx.h      |   5 ++
 6 files changed, 145 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/nested.c +106 -0
index 8b9202ecafff..3a338f7f5a69 100644
@@ -729,6 +729,56 @@ static void setup_guest_ept(struct shadow_vcpu_state *shadow_vcpu, u64 guest_ept
 		pkvm_invalidate_shadow_ept(&vm->sept_desc);
 }
 
+static int __do_handle_invept(struct pkvm_shadow_vm *vm, void *data)
+{
+	if (data) {
+		/*
+		 * Single context invalidate with specific guest eptp
+		 */
+		u64 guest_eptp = *(u64 *)data;
+		struct shadow_vcpu_state *vcpu;
+		struct vmcs12 *vmcs12;
+		s64 shadow_vcpu_handle;
+		int i;
+
+		pkvm_spin_lock(&vm->lock);
+
+		for (i = 0; i < vm->created_vcpus; i++) {
+			shadow_vcpu_handle = to_shadow_vcpu_handle(vm->shadow_vm_handle, i);
+			vcpu = get_shadow_vcpu(shadow_vcpu_handle);
+			if (!vcpu)
+				continue;
+			vmcs12 = (struct vmcs12 *)vcpu->cached_vmcs12;
+			if (vmcs12->ept_pointer == guest_eptp) {
+				put_shadow_vcpu(shadow_vcpu_handle);
+				break;
+			}
+			put_shadow_vcpu(shadow_vcpu_handle);
+		}
+
+		pkvm_spin_unlock(&vm->lock);
+
+		/* No vCPU is using this guest eptp so no need invalidate */
+		if (i == vm->created_vcpus)
+			goto done;
+	}
+
+	/*
+	 * Gloable context invalidation directly went here.
+	 */
+	pkvm_invalidate_shadow_ept(&vm->sept_desc);
+done:
+	return data ? 1 : 0;
+}
+
+static void do_handle_invept(u64 *guest_eptp)
+{
+	if (guest_eptp && !is_valid_eptp(*guest_eptp))
+		return;
+
+	for_each_valid_shadow_vm(__do_handle_invept, guest_eptp);
+}
+
 int handle_vmxon(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -1019,6 +1069,62 @@ int handle_vmlaunch(struct kvm_vcpu *vcpu)
 	return 0;
 }
 
+int handle_invept(struct kvm_vcpu *vcpu)
+{
+	struct vmx_capability *vmx_cap = &pkvm_hyp->vmx_cap;
+	struct vcpu_vmx *vmx = to_vmx(vcpu);
+	u32 vmx_instruction_info, types;
+	unsigned long type;
+	int gpr_index;
+
+	if (!vmx_has_invept())
+		/* TODO: inject #UD */
+		return -EINVAL;
+
+	if (!check_vmx_permission(vcpu))
+		return 0;
+
+	vmx_instruction_info = vmcs_read32(VMX_INSTRUCTION_INFO);
+	gpr_index = vmx_get_instr_info_reg2(vmx_instruction_info);
+	type = vcpu->arch.regs[gpr_index];
+	types = (vmx_cap->ept >> VMX_EPT_EXTENT_SHIFT) & 6;
+
+	if (type >= 32 || !(types & (1 << type))) {
+		nested_vmx_result(VMfailValid, VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID);
+		return 0;
+	}
+
+	switch (type) {
+	case VMX_EPT_EXTENT_CONTEXT: {
+		struct x86_exception e;
+		gva_t gva;
+		struct {
+			u64 eptp, gpa;
+		} operand;
+
+		if (get_vmx_mem_address(vcpu, vmx->exit_qualification,
+					vmx_instruction_info, &gva))
+			/* TODO: handle the decode failure */
+			return -EINVAL;
+
+		if (read_gva(vcpu, gva, &operand, sizeof(operand), &e) < 0)
+			/*TODO: handle memory failure exception */
+			return -EINVAL;
+
+		do_handle_invept(&operand.eptp);
+		break;
+	}
+	case VMX_EPT_EXTENT_GLOBAL:
+		do_handle_invept(NULL);
+		break;
+	default:
+		break;
+	}
+
+	nested_vmx_result(VMsucceed, 0);
+	return 0;
+}
+
 static bool nested_handle_ept_violation(struct shadow_vcpu_state *shadow_vcpu,
 					u64 l2_gpa, u64 exit_quali)
 {
arch/x86/kvm/vmx/pkvm/hyp/nested.h +1 -0
index 24cf731e96dd..01a0e5761e5a 100644
@@ -13,6 +13,7 @@ int handle_vmwrite(struct kvm_vcpu *vcpu);
 int handle_vmread(struct kvm_vcpu *vcpu);
 int handle_vmresume(struct kvm_vcpu *vcpu);
 int handle_vmlaunch(struct kvm_vcpu *vcpu);
+int handle_invept(struct kvm_vcpu *vcpu);
 int nested_vmexit(struct kvm_vcpu *vcpu);
 void pkvm_init_nest(void);
 
arch/x86/kvm/vmx/pkvm/hyp/pkvm.c +27 -0
index 321df4dd2998..e18688b1a235 100644
@@ -355,3 +355,30 @@ unsigned long __pkvm_teardown_shadow_vcpu(s64 shadow_vcpu_handle)
 	memset(shadow_vcpu, 0, sizeof(struct shadow_vcpu_state));
 	return pkvm_virt_to_phys(shadow_vcpu);
 }
+
+int for_each_valid_shadow_vm(int (*fn)(struct pkvm_shadow_vm *vm, void *data),
+			     void *data)
+{
+	struct pkvm_shadow_vm *vm;
+	int i, ret;
+
+	pkvm_spin_lock(&shadow_vms_lock);
+	for (i = HANDLE_OFFSET; i < MAX_SHADOW_VMS; i++) {
+		if (!test_bit(i, shadow_vms_bitmap))
+			continue;
+
+		vm = get_shadow_vm(i);
+		if (!vm)
+			continue;
+
+		ret = fn(vm, data);
+
+		put_shadow_vm(i);
+
+		if (ret)
+			break;
+	}
+	pkvm_spin_unlock(&shadow_vms_lock);
+
+	return ret < 0 ? ret : 0;
+}
arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h +3 -0
index cc7ec8505a98..bf5719eefa0e 100644
@@ -116,6 +116,9 @@ struct shadow_vcpu_state *get_shadow_vcpu(s64 shadow_vcpu_handle);
 void put_shadow_vcpu(s64 shadow_vcpu_handle);
 s64 find_shadow_vcpu_handle_by_vmcs(unsigned long vmcs12_pa);
 
+int for_each_valid_shadow_vm(int (*fn)(struct pkvm_shadow_vm *vm, void *data),
+			     void *data);
+
 extern struct pkvm_hyp *pkvm_hyp;
 
 #endif
arch/x86/kvm/vmx/pkvm/hyp/vmexit.c +3 -0
index 307514f44ec9..e768118fd1f3 100644
@@ -241,6 +241,9 @@ int pkvm_main(struct kvm_vcpu *vcpu)
 				handle_irq_window(vcpu);
 				break;
 			case EXIT_REASON_INVEPT:
+				handle_invept(vcpu);
+				skip_instruction = true;
+				break;
 			case EXIT_REASON_INVVPID:
 				ept_sync_global();
 				skip_instruction = true;
arch/x86/kvm/vmx/pkvm/hyp/vmx.h +5 -0
index 463780776666..6b9ad304fe1d 100644
@@ -15,6 +15,11 @@ static inline bool vmx_ept_capability_check(u32 bit)
 	return vmx_cap->ept & bit;
 }
 
+static inline bool vmx_has_invept(void)
+{
+	return vmx_ept_capability_check(VMX_EPT_INVEPT_BIT);
+}
+
 static inline bool vmx_ept_has_4levels(void)
 {
 	return vmx_ept_capability_check(VMX_EPT_PAGE_WALK_4_BIT);
Signature · 1 line
2.25.1
#13 lore raw

RFC PATCH 13/13 pkvm: x86: Switch to use shadow EPT for nested guests

In reply to Jason Chen CJ

From: Chuanxiao Dong <[email protected]>

Shadow EPT currently is per VM which is configured when creating VM. The shadow EPTP won't be changed at runtime. So just configure the EPTP for vmcs02 when the first this vmcs02 is loaded and skip the later EPTP sync from vmcs12 to vmcs02.

---
 arch/x86/kvm/vmx/pkvm/hyp/nested.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/nested.c +17 -0
index 3a338f7f5a69..f43e559cb594 100644
@@ -572,6 +572,14 @@ static void sync_vmcs12_dirty_fields_to_vmcs02(struct vcpu_vmx *vmx, struct vmcs
 	if (vmx->nested.dirty_vmcs12) {
 		for (i = 0; i < max_emulated_fields; i++) {
 			field = emulated_fields[i];
+			if (field.encoding == EPT_POINTER)
+				/*
+				 * EPTP is configured as shadow EPTP when the first
+				 * time the vmcs02 is loaded. As shadow EPTP is not
+				 * changed at the runtime, also cannot use the virtual
+				 * EPT from KVM high, no need to sync to vmcs02 again.
+				 */
+				continue;
 			val = vmcs12_read_any(vmcs12, field.encoding, field.offset);
 			phys_val = emulate_field_for_vmcs02(vmx, field.encoding, val);
 			__vmcs_writel(field.encoding, phys_val);
@@ -860,6 +868,15 @@ int handle_vmptrld(struct kvm_vcpu *vcpu)
 						vmcs_load_track(vmx, vmcs02);
 						pkvm_init_host_state_area(pkvm_hvcpu->pcpu, vcpu->cpu);
 						vmcs_writel(HOST_RIP, (unsigned long)__pkvm_vmx_vmexit);
+						/*
+						 * EPTP is mantained by pKVM and configured with
+						 * shadow EPTP from its corresponding shadow VM.
+						 * As shadow EPTP is not changed at runtime, set
+						 * it to EPTP when the first time this vmcs02 is
+						 * loading.
+						 */
+						vmcs_write64(EPT_POINTER,
+							     shadow_vcpu->vm->sept_desc.shadow_eptp);
 						shadow_vcpu->last_cpu = vcpu->cpu;
 						shadow_vcpu->vmcs02_inited = true;
 					} else {
Signature · 1 line
2.25.1