KVM: x86/mmu: Introduce pinned SPTEs framework
cover KVM: x86/mmu: Introduce pinned SPTEs framework
SEV currently needs to pin guest memory as it doesn't support migrating encrypted pages. Introduce a framework in KVM's MMU to support pinning pages on demand without requiring additional memory allocations, and with (somewhat hazy) line of sight toward supporting more advanced features for encrypted guest memory, e.g. host page migration.
The idea is to use a software available bit in the SPTE to track that a page has been pinned. The decision to pin a page and the actual pinning managment is handled by vendor code via kvm_x86_ops hooks. There are intentionally two hooks (zap and unzap) introduced that are not needed for SEV. I included them to again show how the flag (probably renamed?) could be used for more than just pin/unpin.
Bugs in the core implementation are pretty much guaranteed. The basic concept has been tested, but in a fairly different incarnation. Most notably, tagging PRESENT SPTEs as PINNED has not been tested, although using the PINNED flag to track zapped (and known to be pinned) SPTEs has been tested. I cobbled this variation together fairly quickly to get the code out there for discussion.
The last patch to pin SEV pages during sev_launch_update_data() is incomplete; it's there to show how we might leverage MMU-based pinning to support pinning pages before the guest is live.
Sean Christopherson (8):
KVM: x86/mmu: Return old SPTE from mmu_spte_clear_track_bits()
KVM: x86/mmu: Use bits 2:0 to check for present SPTEs
KVM: x86/mmu: Refactor handling of not-present SPTEs in mmu_set_spte()
KVM: x86/mmu: Add infrastructure for pinning PFNs on demand
KVM: SVM: Use the KVM MMU SPTE pinning hooks to pin pages on demand
KVM: x86/mmu: Move 'pfn' variable to caller of direct_page_fault()
KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by SEV
KVM: SVM: Pin SEV pages in MMU during sev_launch_update_data()
arch/x86/include/asm/kvm_host.h | 7 ++ arch/x86/kvm/mmu.h | 3 + arch/x86/kvm/mmu/mmu.c | 186 +++++++++++++++++++++++++------- arch/x86/kvm/mmu/paging_tmpl.h | 3 +- arch/x86/kvm/svm/sev.c | 141 +++++++++++++++++++++++- arch/x86/kvm/svm/svm.c | 3 + arch/x86/kvm/svm/svm.h | 3 + 7 files changed, 302 insertions(+), 44 deletions(-)
Signature · 1 line
2.28.0
RFC PATCH 1/8 KVM: x86/mmu: Return old SPTE from mmu_spte_clear_track_bits()
Return the old SPTE when clearing a SPTE and push the "old SPTE present" check to the caller. Tracking pinned SPTEs will use the old SPTE in rmap_remove() to determine whether or not the SPTE is pinned.
--- arch/x86/kvm/mmu/mmu.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +7 -5
@@ -930,9 +930,9 @@ static bool mmu_spte_update(u64 *sptep, u64 new_spte)* Rules for using mmu_spte_clear_track_bits: * It sets the sptep from present to nonpresent, and track the * state bits, it is used to clear the last level sptep. -* Returns non-zero if the PTE was previously valid. +* Returns the old PTE. */ -static int mmu_spte_clear_track_bits (u64 *sptep) +static u64 mmu_spte_clear_track_bits (u64 *sptep) {kvm_pfn_t pfn;u64 old_spte = *sptep;
@@ -943,7 +943,7 @@ static int mmu_spte_clear_track_bits(u64 *sptep) old_spte =__update_clear_spte_slow (sptep,0ull );if (!is_shadow_present_pte (old_spte)) -return 0 ; +return old_spte; pfn =spte_to_pfn (old_spte);
@@ -960,7 +960,7 @@ static int mmu_spte_clear_track_bits(u64 *sptep)if (is_dirty_spte (old_spte))kvm_set_pfn_dirty (pfn); -return 1 ; +return old_spte; }/*
@@ -1484,7 +1484,9 @@ static u64 *rmap_get_next(struct rmap_iterator *iter)static void drop_spte (struct kvm *kvm,u64 *sptep) { -if (mmu_spte_clear_track_bits (sptep)) +u64 old_spte =mmu_spte_clear_track_bits (sptep); + +if (is_shadow_present_pte (old_spte))rmap_remove (kvm, sptep); }
Signature · 1 line
2.28.0
RFC PATCH 2/8 KVM: x86/mmu: Use bits 2:0 to check for present SPTEs
In reply to Sean ChristophersonUse what are effectively EPT's RWX bits to detect present SPTEs instead of simply looking for a non-zero value. This will allow using a non-zero initial value for SPTEs as well as using not-present SPTEs to track metadata for zapped private SPTEs.
--- arch/x86/kvm/mmu/mmu.c | 9 +++++++-- arch/x86/kvm/mmu/paging_tmpl.h | 3 ++- 2 files changed, 9 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +7 -2
@@ -625,9 +625,14 @@ static int is_nx(struct kvm_vcpu *vcpu)return vcpu->arch.efer &EFER_NX ; } -static int is_shadow_present_pte (u64 pte) +static inline bool __is_shadow_present_pte (u64 pte) { -return (pte !=0 ) && !is_mmio_spte (pte); +return !!(pte &0x7 ); +} + +static bool is_shadow_present_pte (u64 pte) +{ +return __is_shadow_present_pte (pte) && !is_mmio_spte (pte); }static int is_large_pte (u64 pte)
arch/x86/kvm/mmu/paging_tmpl.h +2 -1
@@ -1024,7 +1024,8 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)gpa_t pte_gpa;gfn_t gfn; -if (!sp->spt[i]) +if (!__is_shadow_present_pte (sp->spt[i]) && + !is_mmio_spte (sp->spt[i]))continue ; pte_gpa = first_pte_gpa + i *sizeof (pt_element_t);
Signature · 1 line
2.28.0
RFC PATCH 3/8 KVM: x86/mmu: Refactor handling of not-present SPTEs in mmu_set_spte()
In reply to Sean ChristophersonReturn early from mmu_set_spte() if the new SPTE is not-present so as to reduce the indentation of the code that performs metadata updates, e.g. rmap manipulation. Additional metadata updates will soon follow...
--- arch/x86/kvm/mmu/mmu.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
arch/x86/kvm/mmu/mmu.c +8 -6
@@ -3126,12 +3126,14 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, u64 *sptep,if (!was_rmapped &&is_large_pte (*sptep)) ++vcpu->kvm->stat.lpages; -if (is_shadow_present_pte (*sptep)) { -if (!was_rmapped) { - rmap_count =rmap_add (vcpu, sptep, gfn); -if (rmap_count >RMAP_RECYCLE_THRESHOLD ) -rmap_recycle (vcpu, sptep, gfn); - } +/* No additional tracking necessary for not-present SPTEs. */ +if (!is_shadow_present_pte (*sptep)) +return ret; + +if (!was_rmapped) { + rmap_count =rmap_add (vcpu, sptep, gfn); +if (rmap_count >RMAP_RECYCLE_THRESHOLD ) +rmap_recycle (vcpu, sptep, gfn); }return ret;
Signature · 1 line
2.28.0
RFC PATCH 4/8 KVM: x86/mmu: Add infrastructure for pinning PFNs on demand
In reply to Sean Christopherson--- arch/x86/include/asm/kvm_host.h | 7 ++ arch/x86/kvm/mmu/mmu.c | 111 ++++++++++++++++++++++++++------ 2 files changed, 99 insertions(+), 19 deletions(-)
arch/x86/include/asm/kvm_host.h +7 -0
@@ -1138,6 +1138,13 @@ struct kvm_x86_ops {void (*load_mmu_pgd)(struct kvm_vcpu *vcpu,unsigned long cr3); +bool (*pin_spte)(struct kvm_vcpu *vcpu,gfn_t gfn,int level, +kvm_pfn_t pfn); +void (*drop_pinned_spte)(struct kvm *kvm,gfn_t gfn,int level, +kvm_pfn_t pfn); +void (*zap_pinned_spte)(struct kvm *kvm,gfn_t gfn,int level); +void (*unzap_pinned_spte)(struct kvm *kvm,gfn_t gfn,int level); +bool (*has_wbinvd_exit)(void );/* Returns actual tsc_offset set in active VMCS */
arch/x86/kvm/mmu/mmu.c +92 -19
268 lines not on this page · raw message
Signature · 1 line
2.28.0
RFC PATCH 5/8 KVM: SVM: Use the KVM MMU SPTE pinning hooks to pin pages on demand
In reply to Sean Christopherson--- arch/x86/kvm/svm/sev.c | 24 ++++++++++++++++++++++++ arch/x86/kvm/svm/svm.c | 3 +++ arch/x86/kvm/svm/svm.h | 3 +++ 3 files changed, 30 insertions(+)
arch/x86/kvm/svm/sev.c +24 -0
@@ -1193,3 +1193,27 @@ void pre_sev_run(struct vcpu_svm *svm, int cpu) svm->vmcb->control.tlb_ctl =TLB_CONTROL_FLUSH_ASID ;vmcb_mark_dirty (svm->vmcb,VMCB_ASID ); } + +bool sev_pin_spte (struct kvm_vcpu *vcpu,gfn_t gfn,int level,kvm_pfn_t pfn) +{ +if (!sev_guest (vcpu->kvm)) +return false; + +get_page (pfn_to_page (pfn)); + +/* +* Flush any cached lines of the page being added since "ownership" of +* it will be transferred from the host to an encrypted guest. +*/ +clflush_cache_range (__va (pfn <<PAGE_SHIFT ),page_level_size (level)); + +return true; +} + +void sev_drop_pinned_spte (struct kvm *kvm,gfn_t gfn,int level,kvm_pfn_t pfn) +{ +if (WARN_ON_ONCE (!sev_guest (kvm))) +return ; + +put_page (pfn_to_page (pfn)); +}
arch/x86/kvm/svm/svm.c +3 -0
@@ -4150,6 +4150,9 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .need_emulation_on_page_fault = svm_need_emulation_on_page_fault, .apic_init_signal_blocked = svm_apic_init_signal_blocked, + + .pin_spte = sev_pin_spte, + .drop_pinned_spte = sev_drop_pinned_spte, };static struct kvm_x86_init_ops svm_init_ops__initdata = {
arch/x86/kvm/svm/svm.h +3 -0
@@ -489,4 +489,7 @@ void pre_sev_run(struct vcpu_svm *svm, int cpu);int __init sev_hardware_setup (void );void sev_hardware_teardown (void ); +bool sev_pin_spte (struct kvm_vcpu *vcpu,gfn_t gfn,int level,kvm_pfn_t pfn); +void sev_drop_pinned_spte (struct kvm *kvm,gfn_t gfn,int level,kvm_pfn_t pfn); +#endif
Signature · 1 line
2.28.0
RFC PATCH 6/8 KVM: x86/mmu: Move 'pfn' variable to caller of direct_page_fault()
In reply to Sean ChristophersonWhen adding pages prior to boot, SEV needs to pin the resulting host pfn so that the pages that are consumed by sev_launch_update_data() are not moved after the memory is encrypted, which would corrupt the guest data.
--- arch/x86/kvm/mmu/mmu.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +11 -8
@@ -4156,7 +4156,8 @@ static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn, }static int direct_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, -bool prefault,int max_level,bool is_tdp) +bool prefault,int max_level,bool is_tdp, +kvm_pfn_t *pfn) {bool write = error_code &PFERR_WRITE_MASK ;bool exec = error_code &PFERR_FETCH_MASK ;
@@ -4165,7 +4166,6 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,gfn_t gfn = gpa >>PAGE_SHIFT ;unsigned long mmu_seq; -kvm_pfn_t pfn;int r;if (page_fault_handle_page_track (vcpu, error_code, gfn))
@@ -4184,10 +4184,10 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, mmu_seq = vcpu->kvm->mmu_notifier_seq;smp_rmb (); -if (try_async_pf (vcpu, prefault, gfn, gpa, &pfn, write, &map_writable)) +if (try_async_pf (vcpu, prefault, gfn, gpa, pfn, write, &map_writable))return RET_PF_RETRY ; -if (handle_abnormal_pfn (vcpu, is_tdp ?0 : gpa, gfn, pfn,ACC_ALL , &r)) +if (handle_abnormal_pfn (vcpu, is_tdp ?0 : gpa, gfn, *pfn,ACC_ALL , &r))return r; r =RET_PF_RETRY ;
@@ -4197,23 +4197,25 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, r =make_mmu_pages_available (vcpu);if (r) goto out_unlock; - r =__direct_map (vcpu, gpa, write, map_writable, max_level, pfn, + r =__direct_map (vcpu, gpa, write, map_writable, max_level, *pfn, prefault, is_tdp && lpage_disallowed); out_unlock:spin_unlock (&vcpu->kvm->mmu_lock); -kvm_release_pfn_clean (pfn); +kvm_release_pfn_clean (*pfn);return r; }static int nonpaging_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code,bool prefault) { +kvm_pfn_t pfn; +pgprintk ("%s: gva %lx error %x\n" , __func__, gpa, error_code);/* This path builds a PAE pagetable, we can map 2mb pages at maximum. */ return direct_page_fault (vcpu, gpa &PAGE_MASK , error_code, prefault, -PG_LEVEL_2M , false); +PG_LEVEL_2M , false, &pfn); } int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code,
@@ -4252,6 +4254,7 @@ EXPORT_SYMBOL_GPL(kvm_handle_page_fault);int kvm_tdp_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code,bool prefault) { +kvm_pfn_t pfn;int max_level; for (max_level = KVM_MAX_HUGEPAGE_LEVEL;
@@ -4265,7 +4268,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, }return direct_page_fault (vcpu, gpa, error_code, prefault, - max_level, true); + max_level, true, &pfn); } static void nonpaging_init_context(struct kvm_vcpu *vcpu,
Signature · 1 line
2.28.0
RFC PATCH 7/8 KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by SEV
In reply to Sean ChristophersonIntroduce a helper to directly (pun intended) fault-in a TDP page without having to go through the full page fault path. This allows SEV to pin pages before booting the guest, provides the resulting pfn to vendor code if should be needed in the future, and allows the RET_PF_* enums to stay in mmu.c where they belong.
--- arch/x86/kvm/mmu.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 25 +++++++++++++++++++++++++ 2 files changed, 28 insertions(+)
arch/x86/kvm/mmu.h +3 -0
@@ -108,6 +108,9 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,return vcpu->arch.mmu->page_fault (vcpu, cr2_or_gpa, err, prefault); } +kvm_pfn_t kvm_mmu_map_tdp_page (struct kvm_vcpu *vcpu,gpa_t gpa, +u32 error_code,int max_level); +/* * Currently, we have two sorts of write-protection, a) the first one * write-protects guest page to sync the guest modification, b) another one is
arch/x86/kvm/mmu/mmu.c +25 -0
@@ -4271,6 +4271,31 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, max_level, true, &pfn); } +kvm_pfn_t kvm_mmu_map_tdp_page (struct kvm_vcpu *vcpu,gpa_t gpa, +u32 error_code,int max_level) +{ +kvm_pfn_t pfn; +int r; + +if (mmu_topup_memory_caches (vcpu, false)) +return KVM_PFN_ERR_FAULT ; + +/* +* Loop on the page fault path to handle the case where an mmu_notifier +* invalidation triggers RET_PF_RETRY. In the normal page fault path, +* KVM needs to resume the guest in case the invalidation changed any +* of the page fault properties, i.e. the gpa or error code. For this +* path, the gpa and error code are fixed by the caller, and the caller +* expects failure if and only if the page fault can't be fixed. +*/ +do { + r =direct_page_fault (vcpu, gpa, error_code, false, max_level, + true, &pfn); + }while (r ==RET_PF_RETRY && !is_error_noslot_pfn (pfn)); +return pfn; +} +EXPORT_SYMBOL_GPL (kvm_mmu_map_tdp_page); +static void nonpaging_init_context (struct kvm_vcpu *vcpu,struct kvm_mmu *context) {
Signature · 1 line
2.28.0
RFC PATCH 8/8 KVM: SVM: Pin SEV pages in MMU during sev_launch_update_data()
In reply to Sean Christopherson--- arch/x86/kvm/svm/sev.c | 117 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 112 insertions(+), 5 deletions(-)
arch/x86/kvm/svm/sev.c +112 -5
@@ -15,6 +15,7 @@#include <linux/pagemap.h> #include <linux/swap.h> +#include "mmu.h" #include "x86.h" #include "svm.h"
@@ -415,6 +416,107 @@ static unsigned long get_num_contig_pages(unsigned long idx,return pages; } +#define SEV_PFERR (PFERR_WRITE_MASK | PFERR_USER_MASK) + +static void *sev_alloc_pages (unsigned long size,unsigned long *npages) +{ +/* TODO */ + *npages =0 ; +return NULL ; +} + +static struct kvm_memory_slot *hva_to_memslot (struct kvm *kvm, +unsigned long hva) +{ +struct kvm_memslots *slots =kvm_memslots (kvm); +struct kvm_memory_slot *memslot; + +kvm_for_each_memslot (memslot, slots) { +if (hva >= memslot->userspace_addr && + hva < memslot->userspace_addr + + (memslot->npages <<PAGE_SHIFT )) +return memslot; + } + +return NULL ; +} + +static bool hva_to_gpa (struct kvm *kvm,unsigned long hva) +{ +struct kvm_memory_slot *memslot; +gpa_t gpa_offset; + + memslot =hva_to_memslot (kvm, hva); +if (!memslot) +return UNMAPPED_GVA ; + + gpa_offset = hva - memslot->userspace_addr; +return ((memslot->base_gfn <<PAGE_SHIFT ) + gpa_offset); +} + +static struct page **sev_pin_memory_in_mmu (struct kvm *kvm,unsigned long addr, +unsigned long size, +unsigned long *npages) +{ +struct kvm_vcpu *vcpu; +struct page **pages; +unsigned long i; +kvm_pfn_t pfn; +int idx, ret; +gpa_t gpa; + + pages =sev_alloc_pages (size, npages); +if (!pages) +return ERR_PTR (-ENOMEM ); + + vcpu =kvm_get_vcpu (kvm,0 ); +if (mutex_lock_killable (&vcpu->mutex)) { +kvfree (pages); +return ERR_PTR (-EINTR ); + } + +vcpu_load (vcpu); + idx =srcu_read_lock (&kvm->srcu); + +kvm_mmu_load (vcpu); + +for (i =0 ; i < *npages; i++, addr +=PAGE_SIZE ) { +if (signal_pending (current)) { + ret = -ERESTARTSYS ; + goto err; + } + +if (need_resched ()) +cond_resched (); + + gpa =hva_to_gpa (kvm, addr); +if (gpa ==UNMAPPED_GVA ) { + ret = -EFAULT ; + goto err; + } + pfn =kvm_mmu_map_tdp_page (vcpu, gpa,SEV_PFERR ,PG_LEVEL_4K ); +if (is_error_noslot_pfn (pfn)) { + ret = -EFAULT ; + goto err; + } + pages[i] =pfn_to_page (pfn); +get_page (pages[i]); + } + +srcu_read_unlock (&kvm->srcu, idx); +vcpu_put (vcpu); + +mutex_unlock (&vcpu->mutex); +return pages; + +err: +for ( ; i; --i) +put_page (pages[i-1 ]); + +kvfree (pages); +return ERR_PTR (ret); +} +static int sev_launch_update_data (struct kvm *kvm,struct kvm_sev_cmd *argp) {unsigned long vaddr, vaddr_end, next_vaddr, npages, pages, size, i;
@@ -439,9 +541,12 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) vaddr_end = vaddr + size;/* Lock the user memory. */ - inpages =sev_pin_memory (kvm, vaddr, size, &npages,1 ); -if (!inpages) { - ret = -ENOMEM ; +if (atomic_read (&kvm->online_vcpus)) + inpages =sev_pin_memory_in_mmu (kvm, vaddr, size, &npages); +else + inpages =sev_pin_memory (kvm, vaddr, size, &npages,1 ); +if (IS_ERR (inpages)) { + ret =PTR_ERR (inpages); goto e_free; }
@@ -449,9 +554,11 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)* The LAUNCH_UPDATE command will perform in-place encryption of the * memory content (i.e it will write the same memory region with C=1). * It's possible that the cache may contain the data with C=0, i.e., -* unencrypted so invalidate it first. +* unencrypted so invalidate it first. Flushing is automatically +* handled if the pages can be pinned in the MMU. */ -sev_clflush_pages (inpages, npages); +if (!atomic_read (&kvm->online_vcpus)) +sev_clflush_pages (inpages, npages);for (i =0 ; vaddr < vaddr_end; vaddr = next_vaddr, i += pages) {int offset, len;
Signature · 1 line
2.28.0
Sean,
What commit did you base your series on?
Thanks.
-evt(Eric van Tassell)
Quoted 43 lines · Sean Christopherson
SEV currently needs to pin guest memory as it doesn't support migrating encrypted pages. Introduce a framework in KVM's MMU to support pinning pages on demand without requiring additional memory allocations, and with (somewhat hazy) line of sight toward supporting more advanced features for encrypted guest memory, e.g. host page migration.
The idea is to use a software available bit in the SPTE to track that a page has been pinned. The decision to pin a page and the actual pinning managment is handled by vendor code via
kvm_x86_opshooks. There are intentionally two hooks (zap and unzap) introduced that are not needed for SEV. I included them to again show how the flag (probably renamed?) could be used for more than just pin/unpin.Bugs in the core implementation are pretty much guaranteed. The basic concept has been tested, but in a fairly different incarnation. Most notably, tagging PRESENT SPTEs as PINNED has not been tested, although using the PINNED flag to track zapped (and known to be pinned) SPTEs has been tested. I cobbled this variation together fairly quickly to get the code out there for discussion.
The last patch to pin SEV pages during
sev_launch_update_data()is incomplete; it's there to show how we might leverage MMU-based pinning to support pinning pages before the guest is live.Sean Christopherson (8):
KVM: x86/mmu: Return old SPTE frommmu_spte_clear_track_bits()
KVM: x86/mmu: Use bits 2:0 to check for present SPTEs
KVM: x86/mmu: Refactor handling of not-present SPTEs inmmu_set_spte()
KVM: x86/mmu: Add infrastructure for pinning PFNs on demand
KVM: SVM: Use the KVM MMU SPTE pinning hooks to pin pages on demand
KVM: x86/mmu: Move 'pfn' variable to caller ofdirect_page_fault()
KVM: x86/mmu: Introducekvm_mmu_map_tdp_page()for use by SEV
KVM: SVM: Pin SEV pages in MMU duringsev_launch_update_data()arch/x86/include/asm/kvm_host.h | 7 ++ arch/x86/kvm/mmu.h | 3 + arch/x86/kvm/mmu/mmu.c | 186 +++++++++++++++++++++++++------- arch/x86/kvm/mmu/paging_tmpl.h | 3 +- arch/x86/kvm/svm/sev.c | 141 +++++++++++++++++++++++- arch/x86/kvm/svm/svm.c | 3 + arch/x86/kvm/svm/svm.h | 3 + 7 files changed, 302 insertions(+), 44 deletions(-)
Quoted 2 lines · Eric van Tassell
Sean,
What commit did you base your series on?
Gah, sorry.
kvm/queue, commit 9c2475f3e46a1 ("KVM: Using macros instead of magic values").
Thanks for series Sean. Some thoughts
Quoted 5 lines · Sean Christopherson
SEV currently needs to pin guest memory as it doesn't support migrating encrypted pages. Introduce a framework in KVM's MMU to support pinning pages on demand without requiring additional memory allocations, and with (somewhat hazy) line of sight toward supporting more advanced features for encrypted guest memory, e.g. host page migration.
Eric's attempt to do a lazy pinning suffers with the memory allocation problem and your series seems to address it. As you have noticed, currently the SEV enablement in the KVM does not support migrating the encrypted pages. But the recent SEV firmware provides a support to migrate the encrypted pages (e.g host page migration). The support is available in SEV FW >= 0.17.
Quoted 6 lines
The idea is to use a software available bit in the SPTE to track that a page has been pinned. The decision to pin a page and the actual pinning managment is handled by vendor code via
kvm_x86_opshooks. There are intentionally two hooks (zap and unzap) introduced that are not needed for SEV. I included them to again show how the flag (probably renamed?) could be used for more than just pin/unpin.
If using the available software bits for the tracking the pinning is acceptable then it can be used for the non-SEV guests (if needed). I will look through your patch more carefully but one immediate question, when do we unpin the pages? In the case of the SEV, once a page is pinned then it should not be unpinned until the guest terminates. If we unpin the page before the VM terminates then there is a chance the host page migration will kick-in and move the pages. The KVM MMU code may call to drop the spte's during the zap/unzap and this happens a lot during a guest execution and it will lead us to the path where a vendor specific code will unpin the pages during the guest execution and cause a data corruption for the SEV guest.
Quoted 10 lines
Bugs in the core implementation are pretty much guaranteed. The basic concept has been tested, but in a fairly different incarnation. Most notably, tagging PRESENT SPTEs as PINNED has not been tested, although using the PINNED flag to track zapped (and known to be pinned) SPTEs has been tested. I cobbled this variation together fairly quickly to get the code out there for discussion.
The last patch to pin SEV pages during
sev_launch_update_data()is incomplete; it's there to show how we might leverage MMU-based pinning to support pinning pages before the guest is live.
I will add the SEV specific bits and give this a try.
Quoted 20 lines
Sean Christopherson (8):
KVM: x86/mmu: Return old SPTE frommmu_spte_clear_track_bits()
KVM: x86/mmu: Use bits 2:0 to check for present SPTEs
KVM: x86/mmu: Refactor handling of not-present SPTEs inmmu_set_spte()
KVM: x86/mmu: Add infrastructure for pinning PFNs on demand
KVM: SVM: Use the KVM MMU SPTE pinning hooks to pin pages on demand
KVM: x86/mmu: Move 'pfn' variable to caller ofdirect_page_fault()
KVM: x86/mmu: Introducekvm_mmu_map_tdp_page()for use by SEV
KVM: SVM: Pin SEV pages in MMU duringsev_launch_update_data()arch/x86/include/asm/kvm_host.h | 7 ++ arch/x86/kvm/mmu.h | 3 + arch/x86/kvm/mmu/mmu.c | 186 +++++++++++++++++++++++++------- arch/x86/kvm/mmu/paging_tmpl.h | 3 +- arch/x86/kvm/svm/sev.c | 141 +++++++++++++++++++++++- arch/x86/kvm/svm/svm.c | 3 + arch/x86/kvm/svm/svm.h | 3 + 7 files changed, 302 insertions(+), 44 deletions(-)
Quoted 11 lines · Brijesh Singh
Thanks for series Sean. Some thoughts
Quoted 5 lines · Sean Christopherson
SEV currently needs to pin guest memory as it doesn't support migrating encrypted pages. Introduce a framework in KVM's MMU to support pinning pages on demand without requiring additional memory allocations, and with (somewhat hazy) line of sight toward supporting more advanced features for encrypted guest memory, e.g. host page migration.
Quoted 6 lines
Eric's attempt to do a lazy pinning suffers with the memory allocation problem and your series seems to address it. As you have noticed, currently the SEV enablement in the KVM does not support migrating the encrypted pages. But the recent SEV firmware provides a support to migrate the encrypted pages (e.g host page migration). The support is available in SEV FW >= 0.17.
I assume SEV also doesn't support ballooning? Ballooning would be a good first step toward page migration as I think it'd be easier for KVM to support, e.g. only needs to deal with the "zap" and not the "move".
Quoted 7 lines
Quoted 6 lines
The idea is to use a software available bit in the SPTE to track that a page has been pinned. The decision to pin a page and the actual pinning managment is handled by vendor code via
kvm_x86_opshooks. There are intentionally two hooks (zap and unzap) introduced that are not needed for SEV. I included them to again show how the flag (probably renamed?) could be used for more than just pin/unpin.
Quoted 11 lines
If using the available software bits for the tracking the pinning is acceptable then it can be used for the non-SEV guests (if needed). I will look through your patch more carefully but one immediate question, when do we unpin the pages? In the case of the SEV, once a page is pinned then it should not be unpinned until the guest terminates. If we unpin the page before the VM terminates then there is a chance the host page migration will kick-in and move the pages. The KVM MMU code may call to drop the spte's during the zap/unzap and this happens a lot during a guest execution and it will lead us to the path where a vendor specific code will unpin the pages during the guest execution and cause a data corruption for the SEV guest.
The pages are unpinned by:
drop_spte()
|
-> rmap_remove()
|
-> sev_drop_pinned_spte()The intent is to allow unpinning pages when the mm_struct dies, i.e. when the memory is no longer reachable (as opposed to when the last reference to KVM is put), but typing that out, I realize there are dependencies and assumptions that don't hold true for SEV as implemented.
- Parent shadow pages won't be zapped. Recycling MMU pages and zapping all SPs due to memslot updates are the two concerns.
The easy way out for recycling is to not recycle SPs with pinned children, though that may or may not fly with VMM admins.
I'm trying to resolve the memslot issue[*], but confirming that there's no longer an issue with not zapping everything is proving difficult as we haven't yet reproduced the original bug.
- drop_large_spte() won't be invoked. I believe the only semi-legitimate scenario is if the NX huge page workaround is toggled on while a VM is running. Disallowing that if there is an SEV guest seems reasonable?
There might be an issue with the host page size changing, but I don't think that can happen if the page is pinned. That needs more investigation.
[*] https://lkml.kernel.org/r/[email protected]
Quoted 34 lines
Quoted 10 lines
Bugs in the core implementation are pretty much guaranteed. The basic concept has been tested, but in a fairly different incarnation. Most notably, tagging PRESENT SPTEs as PINNED has not been tested, although using the PINNED flag to track zapped (and known to be pinned) SPTEs has been tested. I cobbled this variation together fairly quickly to get the code out there for discussion.
The last patch to pin SEV pages during
sev_launch_update_data()is incomplete; it's there to show how we might leverage MMU-based pinning to support pinning pages before the guest is live.I will add the SEV specific bits and give this a try.
Quoted 20 lines
Sean Christopherson (8):
KVM: x86/mmu: Return old SPTE frommmu_spte_clear_track_bits()
KVM: x86/mmu: Use bits 2:0 to check for present SPTEs
KVM: x86/mmu: Refactor handling of not-present SPTEs inmmu_set_spte()
KVM: x86/mmu: Add infrastructure for pinning PFNs on demand
KVM: SVM: Use the KVM MMU SPTE pinning hooks to pin pages on demand
KVM: x86/mmu: Move 'pfn' variable to caller ofdirect_page_fault()
KVM: x86/mmu: Introducekvm_mmu_map_tdp_page()for use by SEV
KVM: SVM: Pin SEV pages in MMU duringsev_launch_update_data()arch/x86/include/asm/kvm_host.h | 7 ++ arch/x86/kvm/mmu.h | 3 + arch/x86/kvm/mmu/mmu.c | 186 +++++++++++++++++++++++++------- arch/x86/kvm/mmu/paging_tmpl.h | 3 +- arch/x86/kvm/svm/sev.c | 141 +++++++++++++++++++++++- arch/x86/kvm/svm/svm.c | 3 + arch/x86/kvm/svm/svm.h | 3 + 7 files changed, 302 insertions(+), 44 deletions(-)
Quoted 17 lines · Sean Christopherson
Quoted 10 lines · Brijesh Singh
Thanks for series Sean. Some thoughts
Quoted 5 lines · Sean Christopherson
SEV currently needs to pin guest memory as it doesn't support migrating encrypted pages. Introduce a framework in KVM's MMU to support pinning pages on demand without requiring additional memory allocations, and with (somewhat hazy) line of sight toward supporting more advanced features for encrypted guest memory, e.g. host page migration.
Quoted 6 lines
Eric's attempt to do a lazy pinning suffers with the memory allocation problem and your series seems to address it. As you have noticed, currently the SEV enablement in the KVM does not support migrating the encrypted pages. But the recent SEV firmware provides a support to migrate the encrypted pages (e.g host page migration). The support is available in SEV FW >= 0.17.
Quoted 3 lines
I assume SEV also doesn't support ballooning? Ballooning would be a good first step toward page migration as I think it'd be easier for KVM to support, e.g. only needs to deal with the "zap" and not the "move".
Yes, the ballooning does not work with the SEV.
Quoted 23 lines
Quoted 7 lines
Quoted 7 lines
The idea is to use a software available bit in the SPTE to track that a page has been pinned. The decision to pin a page and the actual pinning managment is handled by vendor code via
kvm_x86_opshooks. There are intentionally two hooks (zap and unzap) introduced that are not needed for SEV. I included them to again show how the flag (probably renamed?) could be used for more than just pin/unpin.Quoted 11 lines
If using the available software bits for the tracking the pinning is acceptable then it can be used for the non-SEV guests (if needed). I will look through your patch more carefully but one immediate question, when do we unpin the pages? In the case of the SEV, once a page is pinned then it should not be unpinned until the guest terminates. If we unpin the page before the VM terminates then there is a chance the host page migration will kick-in and move the pages. The KVM MMU code may call to drop the spte's during the zap/unzap and this happens a lot during a guest execution and it will lead us to the path where a vendor specific code will unpin the pages during the guest execution and cause a data corruption for the SEV guest.
The pages are unpinned by:
drop_spte()
|
->rmap_remove()
Quoted 8 lines
| -> sev_drop_pinned_spte()The intent is to allow unpinning pages when the
mm_structdies, i.e. when the memory is no longer reachable (as opposed to when the last reference to KVM is put), but typing that out, I realize there are dependencies and assumptions that don't hold true for SEV as implemented.
So, I tried this RFC with the SEV guest (of course after adding some of the stuff you highlighted below), the guest fails randomly. I have seen a two to three type of failures 1) boot 2) kernbench execution and 3) device addition/removal, the failure signature is not consistent. I believe after addressing some of the dependencies we may able to make some progress but it will add new restriction which did not existed before.
Quoted 54 lines
- Parent shadow pages won't be zapped. Recycling MMU pages and zapping all SPs due to memslot updates are the two concerns.
The easy way out for recycling is to not recycle SPs with pinned children, though that may or may not fly with VMM admins.
I'm trying to resolve the memslot issue[*], but confirming that there's no longer an issue with not zapping everything is proving difficult as we haven't yet reproduced the original bug.
-
drop_large_spte()won't be invoked. I believe the only semi-legitimate scenario is if the NX huge page workaround is toggled on while a VM is running. Disallowing that if there is an SEV guest seems reasonable?There might be an issue with the host page size changing, but I don't think that can happen if the page is pinned. That needs more investigation.
Quoted 32 lines
Quoted 10 lines
Bugs in the core implementation are pretty much guaranteed. The basic concept has been tested, but in a fairly different incarnation. Most notably, tagging PRESENT SPTEs as PINNED has not been tested, although using the PINNED flag to track zapped (and known to be pinned) SPTEs has been tested. I cobbled this variation together fairly quickly to get the code out there for discussion.
The last patch to pin SEV pages during
sev_launch_update_data()is incomplete; it's there to show how we might leverage MMU-based pinning to support pinning pages before the guest is live.I will add the SEV specific bits and give this a try.
Quoted 19 lines
Sean Christopherson (8):
KVM: x86/mmu: Return old SPTE frommmu_spte_clear_track_bits()
KVM: x86/mmu: Use bits 2:0 to check for present SPTEs
KVM: x86/mmu: Refactor handling of not-present SPTEs inmmu_set_spte()
KVM: x86/mmu: Add infrastructure for pinning PFNs on demand
KVM: SVM: Use the KVM MMU SPTE pinning hooks to pin pages on demand
KVM: x86/mmu: Move 'pfn' variable to caller ofdirect_page_fault()
KVM: x86/mmu: Introducekvm_mmu_map_tdp_page()for use by SEV
KVM: SVM: Pin SEV pages in MMU duringsev_launch_update_data()arch/x86/include/asm/kvm_host.h | 7 ++ arch/x86/kvm/mmu.h | 3 + arch/x86/kvm/mmu/mmu.c | 186 +++++++++++++++++++++++++------- arch/x86/kvm/mmu/paging_tmpl.h | 3 +- arch/x86/kvm/svm/sev.c | 141 +++++++++++++++++++++++- arch/x86/kvm/svm/svm.c | 3 + arch/x86/kvm/svm/svm.h | 3 + 7 files changed, 302 insertions(+), 44 deletions(-)
Hi Sean,
Quoted 55 lines · Brijesh Singh
Quoted 16 lines · Sean Christopherson
Quoted 9 lines · Brijesh Singh
Thanks for series Sean. Some thoughts
Quoted 5 lines · Sean Christopherson
SEV currently needs to pin guest memory as it doesn't support migrating encrypted pages. Introduce a framework in KVM's MMU to support pinning pages on demand without requiring additional memory allocations, and with (somewhat hazy) line of sight toward supporting more advanced features for encrypted guest memory, e.g. host page migration.
Quoted 6 lines
Eric's attempt to do a lazy pinning suffers with the memory allocation problem and your series seems to address it. As you have noticed, currently the SEV enablement in the KVM does not support migrating the encrypted pages. But the recent SEV firmware provides a support to migrate the encrypted pages (e.g host page migration). The support is available in SEV FW >= 0.17.
Quoted 3 lines
I assume SEV also doesn't support ballooning? Ballooning would be a good first step toward page migration as I think it'd be easier for KVM to support, e.g. only needs to deal with the "zap" and not the "move".
Yes, the ballooning does not work with the SEV.
Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
The idea is to use a software available bit in the SPTE to track that a page has been pinned. The decision to pin a page and the actual pinning managment is handled by vendor code via
kvm_x86_opshooks. There are intentionally two hooks (zap and unzap) introduced that are not needed for SEV. I included them to again show how the flag (probably renamed?) could be used for more than just pin/unpin.Quoted 11 lines
If using the available software bits for the tracking the pinning is acceptable then it can be used for the non-SEV guests (if needed). I will look through your patch more carefully but one immediate question, when do we unpin the pages? In the case of the SEV, once a page is pinned then it should not be unpinned until the guest terminates. If we unpin the page before the VM terminates then there is a chance the host page migration will kick-in and move the pages. The KVM MMU code may call to drop the spte's during the zap/unzap and this happens a lot during a guest execution and it will lead us to the path where a vendor specific code will unpin the pages during the guest execution and cause a data corruption for the SEV guest.
The pages are unpinned by:
drop_spte()
|
->rmap_remove()Quoted 8 lines
| -> sev_drop_pinned_spte()The intent is to allow unpinning pages when the
mm_structdies, i.e. when the memory is no longer reachable (as opposed to when the last reference to KVM is put), but typing that out, I realize there are dependencies and assumptions that don't hold true for SEV as implemented.
Quoted 27 lines
So, I tried this RFC with the SEV guest (of course after adding some of the stuff you highlighted below), the guest fails randomly. I have seen a two to three type of failures 1) boot 2) kernbench execution and 3) device addition/removal, the failure signature is not consistent. I believe after addressing some of the dependencies we may able to make some progress but it will add new restriction which did not existed before.
Quoted 20 lines
- Parent shadow pages won't be zapped. Recycling MMU pages and zapping all SPs due to memslot updates are the two concerns.
The easy way out for recycling is to not recycle SPs with pinned children, though that may or may not fly with VMM admins.
I'm trying to resolve the memslot issue[*], but confirming that there's no longer an issue with not zapping everything is proving difficult as we haven't yet reproduced the original bug.
-
drop_large_spte()won't be invoked. I believe the only semi-legitimate scenario is if the NX huge page workaround is toggled on while a VM is running. Disallowing that if there is an SEV guest seems reasonable?There might be an issue with the host page size changing, but I don't think that can happen if the page is pinned. That needs more investigation.
We would like to pin the guest memory on #NPF to reduce the boot delay for the SEV guest. Are you planning to proceed with this RFC? With the some fixes, I am able to get the RFC working for the SEV guest. I can share those fixes with you so that you can include them on next revision. One of the main roadblock I see is that the proposed framework has a dependency on the memslot patch you mentioned above. Without the memslot patch we will end up dropping (aka unpinning) spte during memslot updates which is not acceptable for the SEV guest. I don't see any resolution on the memslot patch yet. Any updates are appreciated. I understand that getting memslot issue resolved may be difficult, so I am wondering if in the meantime we should proceed with the xarray approach to track the pinned pages and release them on VM termination.
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 10 lines
Bugs in the core implementation are pretty much guaranteed. The basic concept has been tested, but in a fairly different incarnation. Most notably, tagging PRESENT SPTEs as PINNED has not been tested, although using the PINNED flag to track zapped (and known to be pinned) SPTEs has been tested. I cobbled this variation together fairly quickly to get the code out there for discussion.
The last patch to pin SEV pages during
sev_launch_update_data()is incomplete; it's there to show how we might leverage MMU-based pinning to support pinning pages before the guest is live.I will add the SEV specific bits and give this a try.
Quoted 19 lines
Sean Christopherson (8):
KVM: x86/mmu: Return old SPTE frommmu_spte_clear_track_bits()
KVM: x86/mmu: Use bits 2:0 to check for present SPTEs
KVM: x86/mmu: Refactor handling of not-present SPTEs inmmu_set_spte()
KVM: x86/mmu: Add infrastructure for pinning PFNs on demand
KVM: SVM: Use the KVM MMU SPTE pinning hooks to pin pages on demand
KVM: x86/mmu: Move 'pfn' variable to caller ofdirect_page_fault()
KVM: x86/mmu: Introducekvm_mmu_map_tdp_page()for use by SEV
KVM: SVM: Pin SEV pages in MMU duringsev_launch_update_data()arch/x86/include/asm/kvm_host.h | 7 ++ arch/x86/kvm/mmu.h | 3 + arch/x86/kvm/mmu/mmu.c | 186 +++++++++++++++++++++++++------- arch/x86/kvm/mmu/paging_tmpl.h | 3 +- arch/x86/kvm/svm/sev.c | 141 +++++++++++++++++++++++- arch/x86/kvm/svm/svm.c | 3 + arch/x86/kvm/svm/svm.h | 3 + 7 files changed, 302 insertions(+), 44 deletions(-)