Memory protection based on page state
cover Memory protection based on page state
This patch set is part-7 of this RFC patches. It introduces memory protection based on page state management for pKVM on Intel platform, and enable running normal VM based on it.
Take use of ignored bits in EPT page table entry [1] to record page state and owner id of a page:
63 ... 58 | 57 56 | ... | 31 ... 12 | 11 ... 0
---------------------------------------------------------
| ... | page state | ... | [owner_id] | ...
Page state - bits[57,56]:
- PKVM_NOPAGE(00b):
the page has no mapping in page table.
under this page state, host EPT is using the pte ignored bits[31~12] to record owner_id.
- PKVM_PAGE_OWNED(01b):
the page is owned exclusively by the page-table owner.
- PKVM_PAGE_SHARED_OWNED(10b):
the page is owned by the page-table owner, but is shared with another.
- PKVM_PAGE_SHARED_BORROWED(11b):
the page is shared with, but not owned by the page-table owner.
Owner_id - bits[31~12] (only valid in host EPT when PKVM_NOPAGE):
- 0: PKVM_ID_HYP
- 1: PKVM_ID_HOST
- others: PKVM_ID_GUEST
Below state machine defines how page states are transformed among different entities (host EPT, and guest shadow EPT - which include normal VM & protected VM):
+------------------+
+------------------+ | (Init state) |
| host : NOPAGE | <---------------- | host* : OWNED |
| guestA*: OWNED | ----------------> | guest: NOPAGE |
+------------------+ / +------------------+
| ^ / | ^
| | / | |
| | / | |
| | / | |
| | / | |
v | / v |
+----------------------------+ +----------------------------+
| host : SHARED_BORROWED | | host : SHARED_OWNED |
| guestA: SHARED_OWNED | | guestB*: SHARED_BORROWED |
+----------------------------+ +----------------------------+ [*] host: EPT of host VM
[*] guestA: shadow EPT of a protected VM
[*] guestB: shadow EPT of a normal VM
Initially, all pages except pKVM owned ones are owned by host VM, so these pages are marked with PKVM_PAGE_OWNED in host EPT. Meantime, before guest first EPT_VIOLATION, there is no page mapped in guest shadow EPT, so all page states in its shadow EPT are PKVM_NOPAGE.
When guest EPT_VIOLATION happen, pKVM needs to do EPT shadowing to build shadow EPT page mapping based on virtual EPT. During it, the corresponding page's state shall follow above state machine to do page donation or page sharing.
- page donation
For a protected VM (guestA), during EPT shadowing, the page assigned to guestA shall be donated from host VM. Which means the page's ownership is moved from host to guestA. So in host EPT, the mapping of corresponding page table entry (host_gpa to hpa(== host_gpa)) is removed and its page state is marked as PKVM_NOPAGE (meantime the guestA is recorded as owner_id). Meanwhile in guestA shadow EPT, the mapping of corresponding page table entry (gpa to hpa) is setup and its page state is marked as PKVM_PAGE_OWNED.
Once a page is donated to a guest, it cannot be donated or shared to other guests before undonate back to host.
Sometimes, host also need donate pages to the pKVM hypervisor (e.g., when creating a VM, its shadow VM data strtucture is allocated in host then donated to the pKVM hypervisor).
This patch set coveres page donation from host to the pKVM hypervisor, but does not include page dontation from host to a protected VM - it's essential to run a protected VM.
- page sharing
For a normal VM (guestB), during EPT shadowing, the page assigned to guestB shall be shared from host VM. Which means both host VM and guestB can access this page. So in host EPT, the mapping of corresponding page table entry is kept and its page state is marked as PKVM_PAGE_SHARED_OWNED. Meanwhile in guestB shadow EPT, the mapping of corresponding page table entry is setup and its page state is marked as PKVM_PAGE_SHARED_BORROWED.
Once a page is shared to a guest, it cannot be donated or shared to other guests before unshare back to host.
For a protected VM (guestA), a page can be shared back to host VM after donated to this guest (e.g., to support virtio). For this case, in host EPT, the mapping of corresponding page table entry is setup again and its page state is marked as PKVM_PAGE_SHARED_BORROWED. Meanwhile in guestA shadow EPT, the mapping of corresponding page table entry is kept but its page state is changed to PKVM_PAGE_SHARED_OWNED.
Once a page is shared back to host after donated, guestA is allowed to unshare it. And this page can also be returned back to host directly.
[Note: above page sharing from a protected VM is not covered in the RFC]
Based on above, this patch set support page state APIs:
- __pkvm_host_donate_hyp/__pkvm_hyp_donate_host
help to donate/undonate shadow VM/VCPU structure from host to pKVM
- __pkvm_host_share_guest/__pkvm_host_unshare_guest
help to manage page state of a normal VM's memory, which in the future disallow protected VMs to allocate pages under such shared page state.
[1]: SDM: The Extended Page Table Mechanism (EPT) chapter
Jason Chen CJ (2):
pkvm: x86: Add pgtable override helper functions for map/unmap/free
leaf
pkvm: x86: Use page state API in shadow EPT for normal VM
Shaoqin Huang (10):
pkvm: x86: Introduce pkvm_pgtable_annotate
pkvm: x86: Use host EPT to track page ownership
pkvm: x86: Use SW bits to track page state
pkvm: x86: Add the record of the page state into page table entry
pkvm: x86: Expose host EPT lock
pkvm: x86: Implement do_donate() helper for donating memory
pkvm: x86: Implement __pkvm_hyp_donate_host()
pkvm: x86: Donate shadow vm & vcpu pages to hypervisor
pkvm: x86: Implement do_share() helper for sharing memory
pkvm: x86: Implement do_unshare() helper for unsharing memory
arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/vmx/pkvm/hyp/Makefile | 2 +- arch/x86/kvm/vmx/pkvm/hyp/ept.c | 84 ++- arch/x86/kvm/vmx/pkvm/hyp/ept.h | 3 + arch/x86/kvm/vmx/pkvm/hyp/init_finalise.c | 6 +- arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c | 593 ++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h | 118 +++++ arch/x86/kvm/vmx/pkvm/hyp/mmu.c | 5 +- arch/x86/kvm/vmx/pkvm/hyp/pgtable.c | 222 +++++--- arch/x86/kvm/vmx/pkvm/hyp/pgtable.h | 49 +- arch/x86/kvm/vmx/pkvm/hyp/pkvm.c | 61 ++- arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h | 6 + 12 files changed, 1061 insertions(+), 89 deletions(-) create mode 100644 arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c create mode 100644 arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h
Signature · 1 line
2.25.1
RFC PATCH 1/12 pkvm: x86: Introduce pkvm_pgtable_annotate
From: Shaoqin Huang <[email protected]>
To protect guest memory, pKVM needs a mechanism to record page's ownership. This will help memory ownership's transition among different entities - host, guests and pKVM hypervisor.
Host EPT page-table is a good place to record the page's ownership, as by default, host VM owns almost all the memory resource, so it shall have almost all memory mapping in its EPT page-table. And host VM will manage these memory resource by allocating to different guests, so host VM knows each page's exact owner.
pKVM uses the ignored bits([12,31]) of invalid mappings in the host EPT page entry to store the unique identifier of the page owner. Choose to use these 20 bits is trying to avoid to conflict with low 12 pte prot bits.
Introduce pkvm_pgtable_annotate to help set ownership id in the pgtable PTE, it re-uses most of the map() logic, but ends up creating invalid mapping with 'annotation' instead. This impacts how pKVM do refcount as it now need to count invalid mappings when they are used for ownership tracking.
--- arch/x86/kvm/vmx/pkvm/hyp/pgtable.c | 78 +++++++++++++++++++++++------ arch/x86/kvm/vmx/pkvm/hyp/pgtable.h | 2 + 2 files changed, 65 insertions(+), 15 deletions(-)
arch/x86/kvm/vmx/pkvm/hyp/pgtable.c +63 -15
@@ -20,6 +20,7 @@ struct pgt_walk_data {struct pkvm_pgtable_map_data {unsigned long phys; +u64 annotation;u64 prot;int pgsz_mask; };
@@ -35,6 +36,21 @@ struct pkvm_pgtable_lookup_data {int level; }; +static bool pkvm_phys_is_valid (u64 phys) +{ +return phys !=INVALID_ADDR ; +} + +static bool pgtable_pte_is_counted (u64 pte) +{ +/* +* Due to we use the invalid pte to record the page ownership, +* the refcount tracks both valid and invalid pte if the pte is +* not 0. +*/ +return !!pte; +} + static bool leaf_mapping_valid(struct pkvm_pgtable_ops *pgt_ops, unsigned long vaddr, unsigned long vaddr_end,
@@ -64,7 +80,7 @@ static bool leaf_mapping_allowed(struct pkvm_pgtable_ops *pgt_ops, {unsigned long page_size = pgt_ops->pgt_level_to_size (level); -if (!IS_ALIGNED (phys, page_size)) +if (pkvm_phys_is_valid (phys) && !IS_ALIGNED (phys, page_size))return false;return leaf_mapping_valid (pgt_ops, vaddr, vaddr_end, pgsz_mask, level);
@@ -97,7 +113,7 @@ static int pgtable_map_try_leaf(struct pkvm_pgtable *pgt, unsigned long vaddr, {struct pkvm_pgtable_ops *pgt_ops = pgt->pgt_ops;struct pkvm_mm_ops *mm_ops = pgt->mm_ops; -u64 new; +u64 old = *(u64 *)ptep, new;if (!leaf_mapping_allowed (pgt_ops, vaddr, vaddr_end, data->phys, data->pgsz_mask, level)) {
@@ -105,20 +121,28 @@ static int pgtable_map_try_leaf(struct pkvm_pgtable *pgt, unsigned long vaddr,return (level ==PG_LEVEL_4K ? -EINVAL : -E2BIG ); } - new = data->phys | data->prot; -if (level !=PG_LEVEL_4K ) - pgt_ops->pgt_entry_mkhuge (&new); +if (pkvm_phys_is_valid (data->phys)) { + new = data->phys | data->prot; +if (level !=PG_LEVEL_4K ) + pgt_ops->pgt_entry_mkhuge (&new); + }else { + new = data->annotation; + } -if (pgt_ops->pgt_entry_present (ptep)) { - pgt_ops->pgt_set_entry (ptep,0 ); - flush_data->flushtlb |= true; +if (pgtable_pte_is_counted (old)) { +if (pgt_ops->pgt_entry_present (ptep)) { + pgt_ops->pgt_set_entry (ptep,0 ); + flush_data->flushtlb |= true; + } mm_ops->put_page (ptep); } - mm_ops->get_page (ptep); - pgt_ops->pgt_set_entry (ptep, new); +if (pgtable_pte_is_counted (new)) + mm_ops->get_page (ptep); - data->phys +=page_level_size (level); + pgt_ops->pgt_set_entry (ptep, new); +if (pkvm_phys_is_valid (data->phys)) + data->phys +=page_level_size (level);return 0 ; }
@@ -489,12 +513,13 @@ int pkvm_pgtable_init(struct pkvm_pgtable *pgt,return 0 ; } -int pkvm_pgtable_map (struct pkvm_pgtable *pgt,unsigned long vaddr_start, -unsigned long phys_start,unsigned long size, -int pgsz_mask,u64 prot) +static int __pkvm_pgtable_map (struct pkvm_pgtable *pgt,unsigned long vaddr_start, +unsigned long phys,unsigned long size, +int pgsz_mask,u64 prot,u64 annotation) {struct pkvm_pgtable_map_data data = { - .phys =ALIGN_DOWN (phys_start,PAGE_SIZE ), + .phys = phys, + .annotation = annotation, .prot = prot, .pgsz_mask = pgsz_mask ? pgt->allowed_pgsz & pgsz_mask : pgt->allowed_pgsz,
@@ -508,6 +533,14 @@ int pkvm_pgtable_map(struct pkvm_pgtable *pgt, unsigned long vaddr_start,return pgtable_walk (pgt, vaddr_start, size, &walker); } +int pkvm_pgtable_map (struct pkvm_pgtable *pgt,unsigned long vaddr_start, +unsigned long phys_start,unsigned long size, +int pgsz_mask,u64 prot) +{ +return __pkvm_pgtable_map (pgt, vaddr_start,ALIGN_DOWN (phys_start,PAGE_SIZE ), + size, pgsz_mask, prot,0 ); +} +int pkvm_pgtable_unmap (struct pkvm_pgtable *pgt,unsigned long vaddr_start,unsigned long size) {
@@ -585,3 +618,18 @@ void pkvm_pgtable_destroy(struct pkvm_pgtable *pgt) virt_root = pgt->mm_ops->phys_to_virt (pgt->root_pa); pgt->mm_ops->put_page (virt_root); } + +/* +* pkvm_pgtable_annotate() - Unmap and annotate pages to track ownership. +* @annotation: The value stored in the invalid pte. +* @annotation[2:0] must be 0. +*/ +int pkvm_pgtable_annotate (struct pkvm_pgtable *pgt,unsigned long addr, +unsigned long size,u64 annotation) +{ +if (pgt->pgt_ops->pgt_entry_present (&annotation)) +return -EINVAL ; + +return __pkvm_pgtable_map (pgt, addr,INVALID_ADDR , size, +1 <<PG_LEVEL_4K ,0 , annotation); +}
arch/x86/kvm/vmx/pkvm/hyp/pgtable.h +2 -0
@@ -80,4 +80,6 @@ int pkvm_pgtable_unmap_safe(struct pkvm_pgtable *pgt, unsigned long vaddr_start,void pkvm_pgtable_lookup (struct pkvm_pgtable *pgt,unsigned long vaddr,unsigned long *pphys,u64 *pprot,int *plevel);void pkvm_pgtable_destroy (struct pkvm_pgtable *pgt); +int pkvm_pgtable_annotate (struct pkvm_pgtable *pgt,unsigned long addr, +unsigned long size,u64 annotation);#endif
Signature · 1 line
2.25.1
RFC PATCH 2/12 pkvm: x86: Use host EPT to track page ownership
In reply to Jason Chen CJFrom: Shaoqin Huang <[email protected]>
Provide host_ept_set_owner_locked() which allows to modify the ownership of pages in the host EPT through pkvm_pgtable_annotate(). This API is used when caller hold host EPT lock.
The page owner id is 20 bits, take use of host EPT PTE bits[12~31], and different owner are defined as the following:
- pKVM hypervisor: 0
- host: 1
- guest: 2 ~ 1 << 20 - 1
The guest's owner id is actually reused from shadow_vm_handle, so also limit MAX_SHADOW_VMS based on page owner id mask.
TODO: now pKVM set owner in host EPT based on 4K pages, it means host EPT may finally be broken into small pages. In the future, can consider to support set owner based on huge pages.
--- arch/x86/kvm/vmx/pkvm/hyp/Makefile | 2 +- arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c | 24 ++++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h | 17 +++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/pkvm.c | 11 +++++++++-- 4 files changed, 51 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/pkvm/hyp/Makefile +1 -1
@@ -12,7 +12,7 @@ ccflags-y += -D__PKVM_HYP__ virt-dir := ../../../../../../$(KVM_PKVM) pkvm-hyp-y := vmx_asm.o vmexit.o memory.o early_alloc.o pgtable.o mmu.o pkvm.o \ - init_finalise.o ept.o idt.o irq.o nested.o vmx.o vmsr.o + init_finalise.o ept.o idt.o irq.o nested.o vmx.o vmsr.o mem_protect.o ifndef CONFIG_PKVM_INTEL_DEBUG lib-dir := lib
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2022 Intel Corporation +*/ +#include <linux/bitfield.h> +#include <pkvm.h> +#include "pkvm_hyp.h" +#include "mem_protect.h" +#include "pgtable.h" + +static u64 pkvm_init_invalid_leaf_owner (pkvm_id owner_id) +{ +return FIELD_PREP (PKVM_INVALID_PTE_OWNER_MASK , owner_id); +} + +static int host_ept_set_owner_locked (phys_addr_t addr,u64 size,pkvm_id owner_id) +{ +u64 annotation =pkvm_init_invalid_leaf_owner (owner_id); +int ret; + + ret =pkvm_pgtable_annotate (pkvm_hyp->host_vm.ept, addr, size, annotation); + +return ret; +}
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2022 Intel Corporation +*/ +#ifndef __PKVM_MEM_PROTECT_H__ +#define __PKVM_MEM_PROTECT_H__ + +/* use 20 bits[12~31] - not conflict w/ low 12 bits pte prot */ +#define PKVM_INVALID_PTE_OWNER_MASK GENMASK(31, 12) + +typedef u32 pkvm_id ; + +#define OWNER_ID_HYP 0UL +#define OWNER_ID_HOST 1UL +#define OWNER_ID_INV (~(u32)0UL) + +#endif
arch/x86/kvm/vmx/pkvm/hyp/pkvm.c +9 -2
@@ -8,6 +8,7 @@#include "pkvm_hyp.h" #include "ept.h" +#include "mem_protect.h" struct pkvm_hyp *pkvm_hyp;
@@ -37,8 +38,14 @@ static int allocate_shadow_vm_handle(struct pkvm_shadow_vm *vm)struct shadow_vm_ref *vm_ref;int handle; -/* The shadow_vm_handle is an int so cannot exceed the INT_MAX */ -BUILD_BUG_ON (MAX_SHADOW_VMS >INT_MAX ); +/* +* The shadow_vm_handle is an int so cannot exceed the INT_MAX. +* Meanwhile shadow_vm_handle will also be used as owner_id in +* the page state machine so it also cannot exceed the max +* owner_id. +*/ +BUILD_BUG_ON (MAX_SHADOW_VMS > +min (INT_MAX , ((1 <<hweight_long (PKVM_INVALID_PTE_OWNER_MASK )) -1 )));pkvm_spin_lock (&shadow_vms_lock);
Signature · 1 line
2.25.1
RFC PATCH 3/12 pkvm: x86: Use SW bits to track page state
In reply to Jason Chen CJFrom: Shaoqin Huang <[email protected]>
pKVM already had a mechnism to record page ownership in host EPT, it still needs a place to record page state based on the ownership; the page state is used to identify the exact state of corresponding page:
- PKVM_NOPAGE: the page has no mapping in page table (also invalid pte). And under this page state, host EPT is using the pte ignored bits to record owner_id.
- PKVM_PAGE_OWNED: the page is owned exclusively by the page-table owner.
- PKVM_PAGE_SHARED_OWNED: the page is owned by the page-table owner, but is shared with another entity.
- PKVM_PAGE_SHARED_BORROWED: the page is shared with, but not owned by the page-table owner.
The bit[56,57] of a EPT page table are ignored in the PTE, so use these two bits as SW bits to record the page state.
--- arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c | 13 ++++++++- arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h | 39 +++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c +12 -1
@@ -10,7 +10,9 @@static u64 pkvm_init_invalid_leaf_owner (pkvm_id owner_id) { -return FIELD_PREP (PKVM_INVALID_PTE_OWNER_MASK , owner_id); +/* the page owned by others also means NOPAGE in page state */ +return FIELD_PREP (PKVM_INVALID_PTE_OWNER_MASK , owner_id) | +FIELD_PREP (PKVM_PAGE_STATE_PROT_MASK ,PKVM_NOPAGE ); }static int host_ept_set_owner_locked (phys_addr_t addr,u64 size,pkvm_id owner_id)
@@ -18,6 +20,15 @@ static int host_ept_set_owner_locked(phys_addr_t addr, u64 size, pkvm_id owner_iu64 annotation =pkvm_init_invalid_leaf_owner (owner_id);int ret; + +/* +* The memory [addr, addr + size) will be unmapped from host ept. At the +* same time, the annotation with a NOPAGE flag will be put in the +* invalid pte that has been unmapped. And the information shows that +* the page has been used by some guest and its id can be read from +* annotation. Also when later these pages are back to host, the annotation +* will be helpful to check the right page transition. +*/ ret =pkvm_pgtable_annotate (pkvm_hyp->host_vm.ept, addr, size, annotation);return ret;
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h +39 -0
@@ -5,9 +5,48 @@#ifndef __PKVM_MEM_PROTECT_H__#define __PKVM_MEM_PROTECT_H__ +/* +* enum pkvm_pgtable_prot - The ignored bits in page-table. +* pkvm will use these ignored bits as software bits to +* identify the page status. +*/ +enum pkvm_pgtable_prot { +PKVM_PGTABLE_PROT_SW0 =BIT (56 ), +PKVM_PGTABLE_PROT_SW1 =BIT (57 ), +}; + +/* +* Using the ignored bits in page-table as SW bits. +* SW bits 0-1 are used to track the memory ownership state of each page: +* 00: The page has no mapping in page table (also invalid pte). And under +* this page state, host ept is using the pte ignored bits to record owner_id. +* 01: The page is owned exclusively by the page-table owner. +* 10: The page is owned by the page-table owner, but is shared +* with another entity. +* 11: The page is shared with, but not owned by the page-table owner. +*/ +enum pkvm_page_state { +PKVM_NOPAGE =0ULL , +PKVM_PAGE_OWNED =PKVM_PGTABLE_PROT_SW0 , +PKVM_PAGE_SHARED_OWNED =PKVM_PGTABLE_PROT_SW1 , +PKVM_PAGE_SHARED_BORROWED =PKVM_PGTABLE_PROT_SW0 | +PKVM_PGTABLE_PROT_SW1 , +}; + +#define PKVM_PAGE_STATE_PROT_MASK (PKVM_PGTABLE_PROT_SW0 | PKVM_PGTABLE_PROT_SW1)/* use 20 bits[12~31] - not conflict w/ low 12 bits pte prot */ #define PKVM_INVALID_PTE_OWNER_MASK GENMASK(31, 12) +static inline u64 pkvm_mkstate (u64 prot,enum pkvm_page_state state) +{ +return (prot & ~PKVM_PAGE_STATE_PROT_MASK ) | state; +} + +static inline enum pkvm_page_state pkvm_getstate (u64 pte) +{ +return pte &PKVM_PAGE_STATE_PROT_MASK ; +} +typedef u32 pkvm_id ;#define OWNER_ID_HYP 0UL
Signature · 1 line
2.25.1
RFC PATCH 4/12 pkvm: x86: Add the record of the page state into page table entry
In reply to Jason Chen CJFrom: Shaoqin Huang <[email protected]>
Add init page state of PKVM_PAGE_OWNED into PTE of host EPT page table when first creating host EPT mapping and handling EPT violation for MMIO ranges.
Besides, unmapping pages in host EPT (eg. pkvm code and data memory regions) has an implicit effect. After unmapping, the PTE of these pages are cleared to 0, that means they are under page state of PKVM_NOPAGE and owned by pKVM hypervisor (owner_id = 0).
Also refine pgtable map API for page state only changes and ensure page state will not be lost during pgtable split.
--- arch/x86/kvm/vmx/pkvm/hyp/ept.c | 3 ++- arch/x86/kvm/vmx/pkvm/hyp/init_finalise.c | 6 +++--- arch/x86/kvm/vmx/pkvm/hyp/pgtable.c | 19 +++++++++++++++---- 3 files changed, 20 insertions(+), 8 deletions(-)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +2 -1
@@ -19,6 +19,7 @@#include "ept.h" #include "memory.h" #include "vmx.h" +#include "mem_protect.h" #include "debug.h" static struct hyp_pool host_ept_pool;
@@ -217,7 +218,7 @@ int handle_host_ept_violation(unsigned long gpa)unsigned long hpa;struct mem_range range, cur;bool is_memory =find_mem_range (gpa, &range); -u64 prot =HOST_EPT_DEF_MMIO_PROT ; +u64 prot =pkvm_mkstate (HOST_EPT_DEF_MMIO_PROT ,PKVM_PAGE_OWNED );int level;int ret;
arch/x86/kvm/vmx/pkvm/hyp/init_finalise.c +3 -3
@@ -19,6 +19,7 @@#include "vmx.h" #include "nested.h" #include "debug.h" +#include "mem_protect.h" void *pkvm_mmu_pgt_base;void *pkvm_vmemmap_base;
@@ -182,8 +183,7 @@ static int create_host_ept_mapping(void)/* * Create EPT mapping for memory with WB + RWX property */ - entry_prot =HOST_EPT_DEF_MEM_PROT ; - + entry_prot =pkvm_mkstate (HOST_EPT_DEF_MEM_PROT ,PKVM_PAGE_OWNED );for (i =0 ; i < hyp_memblock_nr; i++) { reg = &hyp_memory[i]; ret = pkvm_host_ept_map((unsigned long)reg->base,
@@ -198,7 +198,7 @@ static int create_host_ept_mapping(void)* The holes in memblocks are treated as MMIO with the * mapping UC + RWX. */ - entry_prot =HOST_EPT_DEF_MMIO_PROT ; + entry_prot =pkvm_mkstate (HOST_EPT_DEF_MMIO_PROT ,PKVM_PAGE_OWNED );for (i =0 ; i < hyp_memblock_nr; i++, phys = reg->base + reg->size) { reg = &hyp_memory[i]; ret = pkvm_host_ept_map(phys, phys, (unsigned long)reg->base - phys,
arch/x86/kvm/vmx/pkvm/hyp/pgtable.c +15 -4
@@ -7,6 +7,7 @@#include "pgtable.h" #include "memory.h" +#include "mem_protect.h" #include "debug.h" #include "bug.h"
@@ -130,6 +131,10 @@ static int pgtable_map_try_leaf(struct pkvm_pgtable *pgt, unsigned long vaddr, }if (pgtable_pte_is_counted (old)) { +/* if just modify the page state, do set_pte directly */ +if (!((old ^ new) & ~PKVM_PAGE_STATE_PROT_MASK )) + goto set_pte; +if (pgt_ops->pgt_entry_present (ptep)) { pgt_ops->pgt_set_entry (ptep,0 ); flush_data->flushtlb |= true;
@@ -140,6 +145,7 @@ static int pgtable_map_try_leaf(struct pkvm_pgtable *pgt, unsigned long vaddr,if (pgtable_pte_is_counted (new)) mm_ops->get_page (ptep); +set_pte: pgt_ops->pgt_set_entry (ptep, new);if (pkvm_phys_is_valid (data->phys)) data->phys +=page_level_size (level);
@@ -175,6 +181,10 @@ static int pgtable_map_walk_leaf(struct pkvm_pgtable *pgt,return -ENOMEM ;if (pgt_ops->pgt_entry_huge (ptep)) { +u64 prot = pgt_ops->pgt_entry_to_prot (ptep); + + prot =pkvm_mkstate (prot,pkvm_getstate (*(u64 *)ptep)); +/* * Split the large mapping and reuse the * large mapping's prot. The translation
@@ -184,8 +194,7 @@ static int pgtable_map_walk_leaf(struct pkvm_pgtable *pgt, mm_ops->put_page (ptep);pgtable_split (pgt_ops, mm_ops,ALIGN_DOWN (vaddr, size), pgt_ops->pgt_entry_to_phys (ptep), - size, page, level -1 , - pgt_ops->pgt_entry_to_prot (ptep)); + size, page, level -1 , prot); } mm_ops->get_page (ptep);
@@ -283,10 +292,13 @@ static int pgtable_unmap_cb(struct pkvm_pgtable *pgt, unsigned long vaddr,/* * if it is huge pte, split and goto next level. */ +u64 prot = pgt_ops->pgt_entry_to_prot (ptep);void *page = mm_ops->zalloc_page ();if (!page)return -ENOMEM ; + + prot =pkvm_mkstate (prot,pkvm_getstate (*(u64 *)ptep));/* * Split the large mapping and reuse the * large mapping's prot. The translation
@@ -295,8 +307,7 @@ static int pgtable_unmap_cb(struct pkvm_pgtable *pgt, unsigned long vaddr,*/ pgtable_split (pgt_ops, mm_ops,ALIGN_DOWN (vaddr, size), pgt_ops->pgt_entry_to_phys (ptep), - size, page, level -1 , - pgt_ops->pgt_entry_to_prot (ptep)); + size, page, level -1 , prot); pgt_ops->pgt_set_entry (ptep, pgt->table_prot | mm_ops->virt_to_phys (page));return 0 ; }
Signature · 1 line
2.25.1
RFC PATCH 5/12 pkvm: x86: Expose host EPT lock
In reply to Jason Chen CJFrom: Shaoqin Huang <[email protected]>
pKVM already has host EPT lock which is used to protect host EPT violation. such lock will be used for other use cases in page state management in the future.
--- arch/x86/kvm/vmx/pkvm/hyp/ept.c | 16 +++++++++++++--- arch/x86/kvm/vmx/pkvm/hyp/ept.h | 2 ++ 2 files changed, 15 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +13 -3
@@ -24,7 +24,7 @@static struct hyp_pool host_ept_pool;static struct pkvm_pgtable host_ept; -static pkvm_spinlock_t host_ept_lock = __PKVM_SPINLOCK_UNLOCKED; +static pkvm_spinlock_t _host_ept_lock = __PKVM_SPINLOCK_UNLOCKED;static struct hyp_pool shadow_ept_pool;static struct rsvd_bits_validate ept_zero_check;
@@ -153,6 +153,16 @@ int pkvm_host_ept_unmap(unsigned long vaddr_start, unsigned long phys_start,return pkvm_pgtable_unmap_safe (&host_ept, vaddr_start, phys_start, size); } +void host_ept_lock (void ) +{ +pkvm_spin_lock (&_host_ept_lock); +} + +void host_ept_unlock (void ) +{ +pkvm_spin_unlock (&_host_ept_lock); +} +static void reset_rsvds_bits_mask_ept (struct rsvd_bits_validate *rsvd_check,u64 pa_bits_rsvd,bool execonly,int huge_page_level)
@@ -227,7 +237,7 @@ int handle_host_ept_violation(unsigned long gpa)return -EPERM ; } -pkvm_spin_lock (&host_ept_lock); +pkvm_spin_lock (&_host_ept_lock);pkvm_pgtable_lookup (&host_ept, gpa, &hpa,NULL , &level);if (hpa !=INVALID_ADDR ) {
@@ -269,7 +279,7 @@ int handle_host_ept_violation(unsigned long gpa) __func__, gpa); } out: -pkvm_spin_unlock (&host_ept_lock); +pkvm_spin_unlock (&_host_ept_lock);return ret; }
arch/x86/kvm/vmx/pkvm/hyp/ept.h +2 -0
@@ -18,6 +18,8 @@ enum sept_handle_ret { PKVM_INJECT_EPT_MISC, }; +void host_ept_lock (void ); +void host_ept_unlock (void );int pkvm_host_ept_map (unsigned long vaddr_start,unsigned long phys_start,unsigned long size,int pgsz_mask,u64 prot); int pkvm_host_ept_unmap(unsigned long vaddr_start, unsigned long phys_start,
Signature · 1 line
2.25.1
RFC PATCH 6/12 pkvm: x86: Implement do_donate() helper for donating memory
In reply to Jason Chen CJFrom: Shaoqin Huang <[email protected]>
Transferring ownership information of a memory region from one component to another can be achieved using a "donate" operation, which results in the previous owner losing access to the underlying pages entirely, and current owner will control the pages.
Introduce a do_donate() helper for safely donating a memory region from one component to another. Currently, only host_to_hyp donating is implemented, but the code is easily extended to handle other combinations and the permission checks for each component are reusable.
--- arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c | 200 ++++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h | 15 ++ 2 files changed, 215 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c +200 -0
@@ -4,9 +4,39 @@*/ #include <linux/bitfield.h> #include <pkvm.h> +#include <gfp.h> #include "pkvm_hyp.h" #include "mem_protect.h" #include "pgtable.h" +#include "ept.h" + +struct check_walk_data { +enum pkvm_page_state desired; +}; + +enum pkvm_component_id { +PKVM_ID_HYP , +PKVM_ID_HOST , +}; + +struct pkvm_mem_trans_desc { +enum pkvm_component_id id; +union { +struct { +u64 addr; + } host; + +struct { +u64 addr; + } hyp; + }; +}; + +struct pkvm_mem_transition { +u64 size; +struct pkvm_mem_trans_desc initiator; +struct pkvm_mem_trans_desc completer; +};static u64 pkvm_init_invalid_leaf_owner (pkvm_id owner_id) {
@@ -33,3 +63,173 @@ static int host_ept_set_owner_locked(phys_addr_t addr, u64 size, pkvm_id owner_ireturn ret; } + +static int +__check_page_state_walker (struct pkvm_pgtable *pgt,unsigned long vaddr, +unsigned long vaddr_end,int level,void *ptep, +unsigned long flags,struct pgt_flush_data *flush_data, +void *const arg) +{ +struct check_walk_data *data = arg; + +return pkvm_getstate (*(u64 *)ptep) == data->desired ?0 : -EPERM ; +} + +static int check_page_state_range (struct pkvm_pgtable *pgt,u64 addr,u64 size, +enum pkvm_page_state state) +{ +struct check_walk_data data = { + .desired = state, + }; +struct pkvm_pgtable_walker walker = { + .cb = __check_page_state_walker, + .flags =PKVM_PGTABLE_WALK_LEAF , + .arg = &data, + }; + +return pgtable_walk (pgt, addr, size, &walker); +} + +static int __host_check_page_state_range (u64 addr,u64 size, +enum pkvm_page_state state) +{ +return check_page_state_range (pkvm_hyp->host_vm.ept, addr, size, state); +} + +static pkvm_id __pkvm_owner_id (const struct pkvm_mem_trans_desc *desc) +{ +switch (desc->id) { +case PKVM_ID_HYP : +return OWNER_ID_HYP ; +default : +WARN_ON (1 ); +return OWNER_ID_INV ; + } +} + +static pkvm_id completer_owner_id (const struct pkvm_mem_transition *tx) +{ +return __pkvm_owner_id (&tx->completer); +} + +static int host_request_donation (const struct pkvm_mem_transition *tx) +{ +u64 addr = tx->initiator.host.addr; +u64 size = tx->size; + +return __host_check_page_state_range (addr, size,PKVM_PAGE_OWNED ); +} + +static int check_donation (const struct pkvm_mem_transition *tx) +{ +int ret; + +switch (tx->initiator.id) { +case PKVM_ID_HOST : + ret =host_request_donation (tx); +break ; +default : + ret = -EINVAL ; + } + +if (ret) +return ret; + +switch (tx->completer.id) { +case PKVM_ID_HYP : + ret =0 ; +break ; +default : + ret = -EINVAL ; + } + +return ret; +} + +static int host_initiate_donation (const struct pkvm_mem_transition *tx) +{ +pkvm_id owner_id =completer_owner_id (tx); +u64 addr = tx->initiator.host.addr; +u64 size = tx->size; + +if (owner_id ==OWNER_ID_INV ) +return -EINVAL ; +else +return host_ept_set_owner_locked (addr, size, owner_id); +} + +static int __do_donate (const struct pkvm_mem_transition *tx) +{ +int ret; + +switch (tx->initiator.id) { +case PKVM_ID_HOST : + ret =host_initiate_donation (tx); +break ; +default : + ret = -EINVAL ; + } + +if (ret) +return ret; + +switch (tx->completer.id) { +case PKVM_ID_HYP : + ret =0 ; +break ; +default : + ret = -EINVAL ; + } + +return ret; +} + +/* +* do_donate - the page owner transfer ownership to another component. +* +* Initiator: OWNED => NO_PAGE +* Completer: NO_APGE => OWNED +* +* The special component is pkvm_hyp. Since pkvm_hyp can access all the +* memory, nothing needs to be done if the page owner is transferred to +* hyp or hyp transfers the ownership to other entities. +*/ +static int do_donate (const struct pkvm_mem_transition *donation) +{ +int ret; + + ret =check_donation (donation); +if (ret) +return ret; + +return WARN_ON (__do_donate (donation)); +} + +int __pkvm_host_donate_hyp (u64 hpa,u64 size) +{ +int ret; +u64 hyp_addr = (u64 )__pkvm_va (hpa); +struct pkvm_mem_transition donation = { + .size = size, + .initiator = { + .id =PKVM_ID_HOST , + .host = { + .addr = hpa, + }, + }, + .completer = { + .id =PKVM_ID_HYP , + .hyp = { + .addr = hyp_addr, + }, + }, + }; + +host_ept_lock (); + + ret =do_donate (&donation); + +host_ept_unlock (); + +return ret; +}
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h +15 -0
@@ -53,4 +53,19 @@ typedef u32 pkvm_id;#define OWNER_ID_HOST 1UL#define OWNER_ID_INV (~(u32)0UL) +/* +* __pkvm_host_donate_hyp() - Donate pages from host to hyp, then host cannot +* access these donated pages. +* +* @hpa: Start hpa of being donated pages, must be continuous. +* @size: The size of memory to be donated. +* +* A range of pages [hpa, hpa + size) will be donated from host to hyp. And +* this will unmap these pages from host ept and set the page owner as hyp_id +* in the pte in host ept. For hyp mmu, it will do nothing as hyp mmu can +* access all the memory by default, but modifying host ept is necessary because +* a page used by pkvm is private and can't be accessed by host. +*/ +int __pkvm_host_donate_hyp (u64 hpa,u64 size); +#endif
Signature · 1 line
2.25.1
RFC PATCH 7/12 pkvm: x86: Implement __pkvm_hyp_donate_host()
In reply to Jason Chen CJFrom: Shaoqin Huang <[email protected]>
Provide __pkvm_hyp_donate_host() to transfer the page ownership from hypervisor to host. This will be used later when pKVM return memory back to host.
--- arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c | 65 +++++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h | 14 ++++++ 2 files changed, 79 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c +65 -0
@@ -30,6 +30,7 @@ struct pkvm_mem_trans_desc {u64 addr; } hyp; }; +u64 prot; };struct pkvm_mem_transition {
@@ -64,6 +65,11 @@ static int host_ept_set_owner_locked(phys_addr_t addr, u64 size, pkvm_id owner_ireturn ret; } +static int host_ept_create_idmap_locked (u64 addr,u64 size,int pgsz_mask,u64 prot) +{ +return pkvm_pgtable_map (pkvm_hyp->host_vm.ept, addr, addr, size, pgsz_mask, prot); +} + static int __check_page_state_walker(struct pkvm_pgtable *pgt, unsigned long vaddr, unsigned long vaddr_end, int level, void *ptep,
@@ -120,6 +126,14 @@ static int host_request_donation(const struct pkvm_mem_transition *tx)return __host_check_page_state_range (addr, size,PKVM_PAGE_OWNED ); } +static int host_ack_donation (const struct pkvm_mem_transition *tx) +{ +u64 addr = tx->completer.host.addr; +u64 size = tx->size; + +return __host_check_page_state_range (addr, size,PKVM_NOPAGE ); +} +static int check_donation (const struct pkvm_mem_transition *tx) {int ret;
@@ -128,6 +142,9 @@ static int check_donation(const struct pkvm_mem_transition *tx)case PKVM_ID_HOST : ret =host_request_donation (tx);break ; +case PKVM_ID_HYP : + ret =0 ; +break ;default : ret = -EINVAL ; }
@@ -136,6 +153,9 @@ static int check_donation(const struct pkvm_mem_transition *tx)return ret;switch (tx->completer.id) { +case PKVM_ID_HOST : + ret =host_ack_donation (tx); +break ;case PKVM_ID_HYP : ret =0 ;break ;
@@ -158,6 +178,15 @@ static int host_initiate_donation(const struct pkvm_mem_transition *tx)return host_ept_set_owner_locked (addr, size, owner_id); } +static int host_complete_donation (const struct pkvm_mem_transition *tx) +{ +u64 addr = tx->completer.host.addr; +u64 size = tx->size; +u64 prot =pkvm_mkstate (tx->completer.prot,PKVM_PAGE_OWNED ); + +return host_ept_create_idmap_locked (addr, size,0 , prot); +} +static int __do_donate (const struct pkvm_mem_transition *tx) {int ret;
@@ -166,6 +195,9 @@ static int __do_donate(const struct pkvm_mem_transition *tx)case PKVM_ID_HOST : ret =host_initiate_donation (tx);break ; +case PKVM_ID_HYP : + ret =0 ; +break ;default : ret = -EINVAL ; }
@@ -174,6 +206,9 @@ static int __do_donate(const struct pkvm_mem_transition *tx)return ret;switch (tx->completer.id) { +case PKVM_ID_HOST : + ret =host_complete_donation (tx); +break ;case PKVM_ID_HYP : ret =0 ;break ;
@@ -233,3 +268,33 @@ int __pkvm_host_donate_hyp(u64 hpa, u64 size)return ret; } + +int __pkvm_hyp_donate_host (u64 hpa,u64 size) +{ +int ret; +u64 hyp_addr = (u64 )__pkvm_va (hpa); +struct pkvm_mem_transition donation = { + .size = size, + .initiator = { + .id =PKVM_ID_HYP , + .hyp = { + .addr = hyp_addr, + }, + }, + .completer = { + .id =PKVM_ID_HOST , + .host = { + .addr = hpa, + }, + .prot =HOST_EPT_DEF_MEM_PROT , + }, + }; + +host_ept_lock (); + + ret =do_donate (&donation); + +host_ept_unlock (); + +return ret; +}
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h +14 -0
@@ -68,4 +68,18 @@ typedef u32 pkvm_id;*/ int __pkvm_host_donate_hyp (u64 hpa,u64 size); +/* +* __pkvm_hyp_donate_host() - Donate pages from hyp to host, then host can +* access these pages. +* +* @hpa: Start hpa of being donated pages, must be continuous. +* @size: The size of memory to be donated. +* +* A range of pages [hpa, hpa + size) will be donated from hyp to host. This +* will create mapping in host ept for these pages, and nothing to do with hyp +* mmu. This is paired with __pkvm_host_donate_hyp(), and same as host reclaiming +* these pages back. +*/ +int __pkvm_hyp_donate_host (u64 hpa,u64 size); +#endif
Signature · 1 line
2.25.1
RFC PATCH 8/12 pkvm: x86: Donate shadow vm & vcpu pages to hypervisor
In reply to Jason Chen CJFrom: Shaoqin Huang <[email protected]>
The shadow vm/vcpu pages are allocated in KVM-high, and then be managed by pKVM hypervisor, so such page's ownership shall be moved from host VM to pKVM hypervisor by __pkvm_host_donate_hyp.
Above is done when doing shadow vm/vcpu initialization, while in shadow vm/vcpu teardown, those pages shall return to host VM, by __pkvm_hyp_donate_host.
--- arch/x86/kvm/vmx/pkvm/hyp/pkvm.c | 50 ++++++++++++++++++++++++---- arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h | 6 ++++ 2 files changed, 49 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/pkvm/hyp/pkvm.c +43 -7
@@ -96,6 +96,7 @@ int __pkvm_init_shadow_vm(unsigned long kvm_va, size_t shadow_size) {struct pkvm_shadow_vm *vm; +int shadow_vm_handle; if (!PAGE_ALIGNED(shadow_pa) || !PAGE_ALIGNED(shadow_size) ||
@@ -103,29 +104,48 @@ int __pkvm_init_shadow_vm(unsigned long kvm_va, + pkvm_shadow_vcpu_array_size())))return -EINVAL ; +if (__pkvm_host_donate_hyp (shadow_pa, shadow_size)) +return -EINVAL ; + vm =pkvm_phys_to_virt (shadow_pa);memset (vm,0 , shadow_size);pkvm_spin_lock_init (&vm->lock); vm->host_kvm_va = kvm_va; + vm->shadow_size = shadow_size;if (pkvm_shadow_ept_init (&vm->sept_desc)) -return -EINVAL ; + goto undonate; + + shadow_vm_handle =allocate_shadow_vm_handle (vm); +if (shadow_vm_handle <0 ) + goto deinit_shadow_ept; + +return shadow_vm_handle; -return allocate_shadow_vm_handle (vm); +deinit_shadow_ept: +pkvm_shadow_ept_deinit (&vm->sept_desc); +undonate: +memset (vm,0 , shadow_size); +__pkvm_hyp_donate_host (shadow_pa, shadow_size); +return -EINVAL ; }unsigned long __pkvm_teardown_shadow_vm (int shadow_vm_handle) {struct pkvm_shadow_vm *vm =free_shadow_vm_handle (shadow_vm_handle); +unsigned long shadow_size;if (!vm)return 0 ;pkvm_shadow_ept_deinit (&vm->sept_desc); -memset (vm,0 ,sizeof (struct pkvm_shadow_vm ) +pkvm_shadow_vcpu_array_size ()); + shadow_size = vm->shadow_size; +memset (vm,0 , shadow_size); + +WARN_ON (__pkvm_hyp_donate_host (pkvm_virt_to_phys (vm), shadow_size));return pkvm_virt_to_phys (vm); }
@@ -321,32 +341,44 @@ s64 __pkvm_init_shadow_vcpu(struct kvm_vcpu *hvcpu, int shadow_vm_handle, (pkvm_hyp->vmcs_config.size > PAGE_SIZE)) return -EINVAL; +if (__pkvm_host_donate_hyp (shadow_pa, shadow_size)) +return -EINVAL ; + shadow_vcpu =pkvm_phys_to_virt (shadow_pa);memset (shadow_vcpu,0 , shadow_size); + shadow_vcpu->shadow_size = shadow_size; ret =read_gva (hvcpu, vcpu_va, &shadow_vcpu->vmx,sizeof (struct vcpu_vmx ), &e);if (ret <0 ) -return -EINVAL ; + goto undonate; vmcs12_va = (unsigned long )shadow_vcpu->vmx.vmcs01.vmcs;if (gva2gpa (hvcpu, vmcs12_va, (gpa_t *)&shadow_vcpu->vmcs12_pa,0 , &e)) -return -EINVAL ; + goto undonate; vm =get_shadow_vm (shadow_vm_handle);if (!vm) -return -EINVAL ; + goto undonate; shadow_vcpu_handle =attach_shadow_vcpu_to_vm (vm, shadow_vcpu);put_shadow_vm (shadow_vm_handle); +if (shadow_vcpu_handle <0 ) + goto undonate; +return shadow_vcpu_handle; +undonate: +memset (shadow_vcpu,0 , shadow_size); +__pkvm_hyp_donate_host (shadow_pa, shadow_size); +return -EINVAL ; }unsigned long __pkvm_teardown_shadow_vcpu (s64 shadow_vcpu_handle) {int shadow_vm_handle =to_shadow_vm_handle (shadow_vcpu_handle);struct shadow_vcpu_state *shadow_vcpu; +unsigned long shadow_size;struct pkvm_shadow_vm *vm =get_shadow_vm (shadow_vm_handle);if (!vm)
@@ -359,7 +391,11 @@ unsigned long __pkvm_teardown_shadow_vcpu(s64 shadow_vcpu_handle)if (!shadow_vcpu)return 0 ; -memset (shadow_vcpu,0 ,sizeof (struct shadow_vcpu_state )); + shadow_size = shadow_vcpu->shadow_size; +memset (shadow_vcpu,0 , shadow_size); +WARN_ON (__pkvm_hyp_donate_host (pkvm_virt_to_phys (shadow_vcpu), + shadow_size)); +return pkvm_virt_to_phys (shadow_vcpu); }
arch/x86/kvm/vmx/pkvm/hyp/pkvm_hyp.h +6 -0
@@ -38,6 +38,9 @@ struct shadow_vcpu_state {struct pkvm_shadow_vm *vm; +/* The donated size of shadow_vcpu. */ +unsigned long shadow_size; +struct hlist_node hnode;unsigned long vmcs12_pa;bool vmcs02_inited;
@@ -93,6 +96,9 @@ struct pkvm_shadow_vm {/* The host's kvm va. */ unsigned long host_kvm_va; +/* The donated size of shadow_vm. */ +unsigned long shadow_size; +/* * VM's shadow EPT. All vCPU shares one mapping. * FIXME: a potential security issue if some vCPUs are
Signature · 1 line
2.25.1
RFC PATCH 9/12 pkvm: x86: Implement do_share() helper for sharing memory
In reply to Jason Chen CJFrom: Shaoqin Huang <[email protected]>
One component sharing a memory region with another can be achieved using a "share" operation, which results in the page state of sharer become from PAGE_OWNED to PAGE_SHARED_OWNED, and the page state of being shared become from NOPAGE to PAGE_SHARED_BORROWED. The two components now can access the page both.
Introduce a do_share() helper for safely sharing a memory region from one component to another. Currently, only host_to_guest sharing is implemented, but the code is easily extended to handle other combinations and the permission checks for each component are reusable.
--- arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c | 155 ++++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h | 18 +++ 2 files changed, 173 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c +155 -0
@@ -17,6 +17,7 @@ struct check_walk_data {enum pkvm_component_id {PKVM_ID_HYP ,PKVM_ID_HOST , +PKVM_ID_GUEST , };struct pkvm_mem_trans_desc {
@@ -29,6 +30,12 @@ struct pkvm_mem_trans_desc {struct {u64 addr; } hyp; + +struct { +struct pkvm_pgtable *pgt; +u64 addr; +u64 phys; + } guest; };u64 prot; };
@@ -118,6 +125,13 @@ static pkvm_id completer_owner_id(const struct pkvm_mem_transition *tx)return __pkvm_owner_id (&tx->completer); } +static int __guest_check_page_state_range (struct pkvm_pgtable *pgt, +u64 addr,u64 size, +enum pkvm_page_state state) +{ +return check_page_state_range (pgt, addr, size, state); +} +static int host_request_donation (const struct pkvm_mem_transition *tx) {u64 addr = tx->initiator.host.addr;
@@ -298,3 +312,144 @@ int __pkvm_hyp_donate_host(u64 hpa, u64 size)return ret; } + +static int host_request_share (const struct pkvm_mem_transition *tx) +{ +u64 addr = tx->initiator.host.addr; +u64 size = tx->size; + +return __host_check_page_state_range (addr, size,PKVM_PAGE_OWNED ); +} + +static int guest_ack_share (const struct pkvm_mem_transition *tx) +{ +u64 addr = tx->completer.guest.addr; +u64 size = tx->size; + +return __guest_check_page_state_range (tx->completer.guest.pgt, addr, + size,PKVM_NOPAGE ); +} + +static int check_share (const struct pkvm_mem_transition *tx) +{ +int ret; + +switch (tx->initiator.id) { +case PKVM_ID_HOST : + ret =host_request_share (tx); +break ; +default : + ret = -EINVAL ; + } + +if (ret) +return ret; + +switch (tx->completer.id) { +case PKVM_ID_GUEST : + ret =guest_ack_share (tx); +break ; +default : + ret = -EINVAL ; + } + +return ret; +} + +static int host_initiate_share (const struct pkvm_mem_transition *tx) +{ +u64 addr = tx->initiator.host.addr; +u64 size = tx->size; +u64 prot =pkvm_mkstate (tx->initiator.prot,PKVM_PAGE_SHARED_OWNED ); + +return host_ept_create_idmap_locked (addr, size,0 , prot); +} + +static int guest_complete_share (const struct pkvm_mem_transition *tx) +{ +struct pkvm_pgtable *pgt = tx->completer.guest.pgt; +u64 addr = tx->completer.guest.addr; +u64 size = tx->size; +u64 phys = tx->completer.guest.phys; +u64 prot = tx->completer.prot; + + prot =pkvm_mkstate (prot,PKVM_PAGE_SHARED_BORROWED ); +return pkvm_pgtable_map (pgt, addr, phys, size,0 , prot); +} + +static int __do_share (const struct pkvm_mem_transition *tx) +{ +int ret; + +switch (tx->initiator.id) { +case PKVM_ID_HOST : + ret =host_initiate_share (tx); +break ; +default : + ret = -EINVAL ; + } + +if (ret) +return ret; + +switch (tx->completer.id) { +case PKVM_ID_GUEST : + ret =guest_complete_share (tx); +break ; +default : + ret = -EINVAL ; + } + +return ret; +} + +/* +* do_share() - The page owner grants access to another component with a given +* set of permissions. +* +* Initiator: OWNED => SHARED_OWNED +* Completer: NOPAGE => SHARED_BORROWED +*/ +static int do_share (const struct pkvm_mem_transition *share) +{ +int ret; + + ret =check_share (share); +if (ret) +return ret; + +return WARN_ON (__do_share (share)); +} + +int __pkvm_host_share_guest (u64 hpa,struct pkvm_pgtable *guest_pgt, +u64 gpa,u64 size,u64 prot) +{ +int ret; +struct pkvm_mem_transition share = { + .size = size, + .initiator = { + .id =PKVM_ID_HOST , + .host = { + .addr = hpa, + }, + .prot =HOST_EPT_DEF_MEM_PROT , + }, + .completer = { + .id =PKVM_ID_GUEST , + .guest = { + .pgt = guest_pgt, + .addr = gpa, + .phys = hpa, + }, + .prot = prot, + }, + }; + +host_ept_lock (); + + ret =do_share (&share); + +host_ept_unlock (); + +return ret; +}
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h +18 -0
@@ -82,4 +82,22 @@ int __pkvm_host_donate_hyp(u64 hpa, u64 size);*/ int __pkvm_hyp_donate_host (u64 hpa,u64 size); +/* +* __pkvm_host_share_guest() - Share pages between host and guest. Host still +* ownes the page and guest will have temporary access to these pages. +* +* @hpa: Start hpa of being shared pages, must be continuous. +* @guest_pgt: The guest ept pagetable. +* @gpa: Start gpa that will be used for mapping into the guest ept. +* @size: The size of pages to be shared. +* @prot: The prot that will be used for creating mapping for guest ept. +* +* A range of pages [hpa, hpa + size) in host ept that their page state +* will be modified from PAGE_OWNED to PAGE_SHARED_OWNED. There will be +* mapping from gfn to pfn to be created in guest ept. The @prot +* and PAGE_SHARED_BORROWED will be used to create such mapping. +*/ +int __pkvm_host_share_guest (u64 hpa,struct pkvm_pgtable *guest_pgt, +u64 gpa,u64 size,u64 prot); +#endif
Signature · 1 line
2.25.1
RFC PATCH 10/12 pkvm: x86: Implement do_unshare() helper for unsharing memory
In reply to Jason Chen CJFrom: Shaoqin Huang <[email protected]>
do_unshare() is paired with do_share(), and it's the reverse of the do_share(), which results in the page sharer take back accessibility of the shared page for another component which being shared. The page state become like:
sharer: PAGE_SHARED_OWNED => OWNED
be shared: PAGE_SHARED_BORROWED => NOPAGE
Introduce a do_unshare() helper for safely unsharing a memory region from one component to another. It will check the page state stored in pte, and make sure the do_unshare() is paired with do_share(), this will make the page being correctly take back and can only be accessed by the page owner after unsharing.
--- arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c | 138 ++++++++++++++++++++++++ arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h | 15 +++ 2 files changed, 153 insertions(+)
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c +138 -0
@@ -453,3 +453,141 @@ int __pkvm_host_share_guest(u64 hpa, struct pkvm_pgtable *guest_pgt,return ret; } + +static int host_request_unshare (const struct pkvm_mem_transition *tx) +{ +u64 addr = tx->initiator.host.addr; +u64 size = tx->size; + +return __host_check_page_state_range (addr, size,PKVM_PAGE_SHARED_OWNED ); +} + +static int guest_ack_unshare (const struct pkvm_mem_transition *tx) +{ +u64 addr = tx->completer.guest.addr; +u64 size = tx->size; + +return __guest_check_page_state_range (tx->completer.guest.pgt, addr, + size,PKVM_PAGE_SHARED_BORROWED ); +} + +int check_unshare (const struct pkvm_mem_transition *tx) +{ +int ret; + +switch (tx->initiator.id) { +case PKVM_ID_HOST : + ret =host_request_unshare (tx); +break ; +default : + ret = -EINVAL ; + } + +if (ret) +return ret; + +switch (tx->completer.id) { +case PKVM_ID_GUEST : + ret =guest_ack_unshare (tx); +break ; +default : + ret = -EINVAL ; + } + +return ret; +} + +static int host_initiate_unshare (const struct pkvm_mem_transition *tx) +{ +u64 addr = tx->initiator.host.addr; +u64 size = tx->size; +u64 prot =pkvm_mkstate (tx->initiator.prot,PKVM_PAGE_OWNED ); + +return host_ept_create_idmap_locked (addr, size,0 , prot); +} + +static int guest_complete_unshare (const struct pkvm_mem_transition *tx) +{ +struct pkvm_pgtable *pgt = tx->completer.guest.pgt; +u64 addr = tx->completer.guest.addr; +u64 phys = tx->completer.guest.phys; +u64 size = tx->size; + +return pkvm_pgtable_unmap_safe (pgt, addr, phys, size); +} + +static int __do_unshare (struct pkvm_mem_transition *tx) +{ +int ret; + +switch (tx->initiator.id) { +case PKVM_ID_HOST : + ret =host_initiate_unshare (tx); +break ; +default : + ret = -EINVAL ; + } + +if (ret) +return ret; + +switch (tx->completer.id) { +case PKVM_ID_GUEST : + ret =guest_complete_unshare (tx); +break ; +default : + ret = -EINVAL ; + } + +return ret; +} + +/* +* do_unshare() - The page owner takes back the page access for another +* component. +* +* Initiator: SHARED_OWNED => OWNED +* Completer: SHARED_BORROWED => NOPAGE +*/ +int do_unshare (struct pkvm_mem_transition *share) +{ +int ret; + + ret =check_unshare (share); +if (ret) +return ret; + +return WARN_ON (__do_unshare (share)); +} + +int __pkvm_host_unshare_guest (u64 hpa,struct pkvm_pgtable *guest_pgt, +u64 gpa,u64 size) +{ +int ret; +struct pkvm_mem_transition share = { + .size = size, + .initiator = { + .id =PKVM_ID_HOST , + .host = { + .addr = hpa, + }, + .prot =HOST_EPT_DEF_MEM_PROT , + }, + .completer = { + .id =PKVM_ID_GUEST , + .guest = { + .pgt = guest_pgt, + .addr = gpa, + .phys = hpa, + }, + }, + }; + +host_ept_lock (); + + ret =do_unshare (&share); + +host_ept_unlock (); + +return ret; +}
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.h +15 -0
@@ -100,4 +100,19 @@ int __pkvm_hyp_donate_host(u64 hpa, u64 size);int __pkvm_host_share_guest (u64 hpa,struct pkvm_pgtable *guest_pgt,u64 gpa,u64 size,u64 prot); +/* +* __pkvm_host_unshare_guest() - Host unshare pages that have been shared to guest +* previously. Guest will not be able to access these pages. +* +* @hpa: Start hpa of being shared pages, must be continuous. +* @guest_pgt: The guest ept pagetable. +* @gpa: Start gpa of shared pages being mapped in guest ept. +* @size: The size of pages to be shared. +* +* Unmap the range [gfn, gfn + nr_pages) in guest ept pagetable. And change +* the page state from PAGE_SHARED_BORROWED to PAGE_OWNED in the host ept. +*/ +int __pkvm_host_unshare_guest (u64 hpa,struct pkvm_pgtable *guest_pgt, +u64 gpa,u64 size); +#endif
Signature · 1 line
2.25.1
RFC PATCH 11/12 pkvm: x86: Add pgtable override helper functions for map/unmap/free leaf
In reply to Jason Chen CJAdd override helper function in pgtable map/unmap/destroy APIs:
- map_leaf_override(): override the final page entry map function for pkvm_pgtable_map()
- unmap_leaf_override(): override the final page entry unmap function for pkvm_pgtable_unmap()
- free_leaf_override(): override the final page entry free function for pkvm_pgtable_destroy()
The pkvm_pgtable_map() calls map_leaf_override() when doing page leaf mapping, if the override helper function exist.
The pkvm_pgtable_unmap() calls unmap_leaf_override() when doing page leaf unmapping, if the override helper function exist.
Meanwhile for pkvm_pgtable_destroy(), it calls free_leaf_override() when doing page leaf free, if the override helper function exist.
This will be used to handle shadow EPT invalidation, destroy and EPT violation in the following patches, as for these operations, shadow EPT has its own page leaf operations, the default one shall be override.
--- arch/x86/kvm/vmx/pkvm/hyp/ept.c | 11 +- arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c | 6 +- arch/x86/kvm/vmx/pkvm/hyp/mmu.c | 5 +- arch/x86/kvm/vmx/pkvm/hyp/pgtable.c | 137 +++++++++++++++--------- arch/x86/kvm/vmx/pkvm/hyp/pgtable.h | 47 +++++++- 5 files changed, 144 insertions(+), 62 deletions(-)
arch/x86/kvm/vmx/pkvm/hyp/ept.c +6 -5
@@ -144,13 +144,14 @@ struct pkvm_pgtable_ops ept_ops = {int pkvm_host_ept_map (unsigned long vaddr_start,unsigned long phys_start,unsigned long size,int pgsz_mask,u64 prot) { -return pkvm_pgtable_map (&host_ept, vaddr_start, phys_start, size, pgsz_mask, prot); +return pkvm_pgtable_map (&host_ept, vaddr_start, phys_start, size, + pgsz_mask, prot,NULL ); }int pkvm_host_ept_unmap (unsigned long vaddr_start,unsigned long phys_start,unsigned long size) { -return pkvm_pgtable_unmap_safe (&host_ept, vaddr_start, phys_start, size); +return pkvm_pgtable_unmap_safe (&host_ept, vaddr_start, phys_start, size,NULL ); }void host_ept_lock (void )
@@ -327,7 +328,7 @@ void pkvm_invalidate_shadow_ept(struct shadow_ept_desc *desc)if (!is_valid_eptp (desc->shadow_eptp)) goto out; -pkvm_pgtable_unmap (sept,0 , size); +pkvm_pgtable_unmap (sept,0 , size,NULL );flush_ept (desc->shadow_eptp); out:
@@ -342,7 +343,7 @@ void pkvm_shadow_ept_deinit(struct shadow_ept_desc *desc)pkvm_spin_lock (&vm->lock);if (desc->shadow_eptp) { -pkvm_pgtable_destroy (sept); +pkvm_pgtable_destroy (sept,NULL );flush_ept (desc->shadow_eptp);
@@ -459,7 +460,7 @@ pkvm_handle_shadow_ept_violation(struct shadow_vcpu_state *shadow_vcpu, u64 l2_gunsigned long gpa =ALIGN_DOWN (l2_gpa, level_size);unsigned long hpa =ALIGN_DOWN (host_gpa2hpa (phys), level_size); -if (!pkvm_pgtable_map (sept, gpa, hpa, level_size,0 , gprot)) +if (!pkvm_pgtable_map (sept, gpa, hpa, level_size,0 , gprot,NULL )) ret =PKVM_HANDLED ; } out:
arch/x86/kvm/vmx/pkvm/hyp/mem_protect.c +3 -3
@@ -74,7 +74,7 @@ static int host_ept_set_owner_locked(phys_addr_t addr, u64 size, pkvm_id owner_istatic int host_ept_create_idmap_locked (u64 addr,u64 size,int pgsz_mask,u64 prot) { -return pkvm_pgtable_map (pkvm_hyp->host_vm.ept, addr, addr, size, pgsz_mask, prot); +return pkvm_pgtable_map (pkvm_hyp->host_vm.ept, addr, addr, size, pgsz_mask, prot,NULL ); } static int
@@ -374,7 +374,7 @@ static int guest_complete_share(const struct pkvm_mem_transition *tx)u64 prot = tx->completer.prot; prot =pkvm_mkstate (prot,PKVM_PAGE_SHARED_BORROWED ); -return pkvm_pgtable_map (pgt, addr, phys, size,0 , prot); +return pkvm_pgtable_map (pgt, addr, phys, size,0 , prot,NULL ); }static int __do_share (const struct pkvm_mem_transition *tx)
@@ -513,7 +513,7 @@ static int guest_complete_unshare(const struct pkvm_mem_transition *tx)u64 phys = tx->completer.guest.phys;u64 size = tx->size; -return pkvm_pgtable_unmap_safe (pgt, addr, phys, size); +return pkvm_pgtable_unmap_safe (pgt, addr, phys, size,NULL ); }static int __do_unshare (struct pkvm_mem_transition *tx)
arch/x86/kvm/vmx/pkvm/hyp/mmu.c +3 -2
@@ -187,7 +187,8 @@ int pkvm_mmu_map(unsigned long vaddr_start, unsigned long phys_start, {int ret; - ret =pkvm_pgtable_map (&hyp_mmu, vaddr_start, phys_start, size, pgsz_mask, prot); + ret =pkvm_pgtable_map (&hyp_mmu, vaddr_start, phys_start, size, + pgsz_mask, prot,NULL );return ret; }
@@ -196,7 +197,7 @@ int pkvm_mmu_unmap(unsigned long vaddr_start, unsigned long size) {int ret; - ret =pkvm_pgtable_unmap (&hyp_mmu, vaddr_start, size); + ret =pkvm_pgtable_unmap (&hyp_mmu, vaddr_start, size,NULL );return ret; }
arch/x86/kvm/vmx/pkvm/hyp/pgtable.c +89 -48
241 lines not on this page · raw message
arch/x86/kvm/vmx/pkvm/hyp/pgtable.h +43 -4
63 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC PATCH 12/12 pkvm: x86: Use page state API in shadow EPT for normal VM
In reply to Jason Chen CJAdd map_leaf & free_leaf override helper functions for shadow EPT, and use page state API in these helper functions to support shadow EPT invalidation, destroy and EPT violation for normal VM.
When map a page for a normal VM in shadow EPT, use the share API to mark this page is shared which is previously owned by the host VM but now is shared between the host VM and the normal VM. And when invalidate or destroy shadow EPT, mark this page as unshared which means owned by the host VM again.
Under the state machine of page state transition, pKVM does not support multiple guest pages mapping to same host page, it's conflict with KSM, so just disable it under pKVM Kconfig.
--- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/vmx/pkvm/hyp/ept.c | 60 +++++++++++++++++++++++++++++++-- arch/x86/kvm/vmx/pkvm/hyp/ept.h | 1 + 3 files changed, 59 insertions(+), 3 deletions(-)
arch/x86/kvm/Kconfig +1 -0
@@ -91,6 +91,7 @@ config PKVM_INTEL bool "pKVM for Intel processors support" depends on KVM_INTEL=y depends on X86_64 + depends on !KSM help Provides support for pKVM on Intel processors.
arch/x86/kvm/vmx/pkvm/hyp/ept.c +57 -3
85 lines not on this page · raw message
arch/x86/kvm/vmx/pkvm/hyp/ept.h +1 -0
@@ -11,6 +11,7 @@ (MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT))#define HOST_EPT_DEF_MMIO_PROT (VMX_EPT_RWX_MASK | \ (MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT)) +#define EPT_PROT_MASK (VMX_EPT_RWX_MASK | VMX_EPT_MT_MASK | VMX_EPT_IPAT_BIT)enum sept_handle_ret {PKVM_NOT_HANDLED ,
Signature · 1 line
2.25.1