KVM TDX basic feature support
Subsystems (17)
PATCH v8 41/103 KVM: x86/mmu: Add a new is_private member for union kvm_mmu_page_role
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because TDX support introduces private mapping, add a new member in union kvm_mmu_page_role with access functions to check the member.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/include/asm/kvm_host.h | 27 +++++++++++++++++++++++++++ arch/x86/kvm/mmu/mmu_internal.h | 11 +++++++++++ 2 files changed, 38 insertions(+)
arch/x86/include/asm/kvm_host.h +27 -0
@@ -334,7 +334,12 @@ union kvm_mmu_page_role { unsigned ad_disabled:1; unsigned guest_mode:1; unsigned passthrough:1; +#ifdef CONFIG_KVM_MMU_PRIVATE + unsigned is_private:1; + unsigned :4; +#else unsigned :5; +#endif /* * This is left at the top of the word so that
@@ -346,6 +351,28 @@ union kvm_mmu_page_role { }; }; +#ifdef CONFIG_KVM_MMU_PRIVATE +static inline bool kvm_mmu_page_role_is_private (union kvm_mmu_page_role role) +{ +return !!role.is_private; +} + +static inline void kvm_mmu_page_role_set_private (union kvm_mmu_page_role *role) +{ + role->is_private =1 ; +} +#else +static inline bool kvm_mmu_page_role_is_private (union kvm_mmu_page_role role) +{ +return false; +} + +static inline void kvm_mmu_page_role_set_private (union kvm_mmu_page_role *role) +{ +WARN_ON (1 ); +} +#endif +/* * kvm_mmu_extended_role complements kvm_mmu_page_role, tracking properties * relevant to the current MMU configuration. When loading CR0, CR4, or EFER,
arch/x86/kvm/mmu/mmu_internal.h +11 -0
@@ -142,6 +142,17 @@ static inline int kvm_mmu_page_as_id(struct kvm_mmu_page *sp)return kvm_mmu_role_as_id (sp->role); } +static inline bool is_private_sp (const struct kvm_mmu_page *sp) +{ +return kvm_mmu_page_role_is_private (sp->role); +} + +static inline bool is_private_sptep (u64 *sptep) +{ +WARN_ON (!sptep); +return is_private_sp (sptep_to_sp (sptep)); +} +static inline bool kvm_mmu_page_ad_need_write_protect (struct kvm_mmu_page *sp) {/*
Signature · 1 line
2.25.1
Re: [PATCH v8 041/103] KVM: x86/mmu: Add a new is_private member for union kvm_mmu_page_role
Quoted 6 lines · On Sun, Aug 07, 2022 at 03:01:26PM -0700, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Because TDX support introduces private mapping, add a new member in union
kvm_mmu_page_rolewith access functions to check the member.Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Yuan Yao <[email protected]>
Quoted 76 lines
--- arch/x86/include/asm/kvm_host.h | 27 +++++++++++++++++++++++++++ arch/x86/kvm/mmu/mmu_internal.h | 11 +++++++++++ 2 files changed, 38 insertions(+)arch/x86/include/asm/kvm_host.h +27
-0@@ -334,7 +334,12 @@ union kvm_mmu_page_role { unsigned ad_disabled:1; unsigned guest_mode:1; unsigned passthrough:1; +#ifdef CONFIG_KVM_MMU_PRIVATE + unsigned is_private:1; + unsigned :4; +#else unsigned :5; +#endif /* * This is left at the top of the word so that @@ -346,6 +351,28 @@ union kvm_mmu_page_role { }; }; +#ifdef CONFIG_KVM_MMU_PRIVATE +static inline bool kvm_mmu_page_role_is_private (union kvm_mmu_page_role role) +{ +return !!role.is_private; +} + +static inline void kvm_mmu_page_role_set_private (union kvm_mmu_page_role *role) +{ + role->is_private =1 ; +} +#else +static inline bool kvm_mmu_page_role_is_private (union kvm_mmu_page_role role) +{ +return false; +} + +static inline void kvm_mmu_page_role_set_private (union kvm_mmu_page_role *role) +{ +WARN_ON (1 ); +} +#endif +/* * kvm_mmu_extended_role complements kvm_mmu_page_role, tracking properties * relevant to the current MMU configuration. When loading CR0, CR4, or EFER, arch/x86/kvm/mmu/mmu_internal.h +11
-1@@ -142,6 +142,17 @@ static inline int kvm_mmu_page_as_id(struct kvm_mmu_page *sp)return kvm_mmu_role_as_id (sp->role); } +static inline bool is_private_sp (const struct kvm_mmu_page *sp) +{ +return kvm_mmu_page_role_is_private (sp->role); +} + +static inline bool is_private_sptep (u64 *sptep) +{ +WARN_ON (!sptep); +return is_private_sp (sptep_to_sp (sptep)); +} +static inline bool kvm_mmu_page_ad_need_write_protect (struct kvm_mmu_page *sp) {/* -- 2.25.1
PATCH v8 42/103 KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page
In reply to [email protected]From: Isaku Yamahata <[email protected]>
For private GPA, CPU refers a private page table whose contents are encrypted. The dedicated APIs to operate on it (e.g. updating/reading its PTE entry) are used and their cost is expensive.
When KVM resolves KVM page fault, it walks the page tables. To reuse the existing KVM MMU code and mitigate the heavy cost to directly walk protected (encrypted) page table, allocate one more page to copy the protected page table for KVM MMU code to directly walk. Resolve KVM page fault with the existing code, and do additional operations necessary for the protected page table. To distinguish such cases, the existing KVM page table is called a shared page table (i.e. not associated with protected page table), and the page table with protected page table is called a private page table. The relationship is depicted below.
Add a private pointer to struct kvm_mmu_page for protected page table and add helper functions to allocate/initialize/free a protected page table page.
KVM page fault |
| |
V |
-------------+---------- |
| | |
V V |
shared GPA private GPA |
| | |
V V |
shared PT root private PT root | protected PT root
| | | |
V V | V
shared PT private PT ----propagate----> protected PT
| | | |
| \-----------------+------\ |
| | | |
V | V V
shared guest page | private guest page
|
non-encrypted memory | encrypted memory
|PT: page table
- Shared PT is visible to KVM and it is used by CPU.
- Protected PT is used by CPU but it is invisible to KVM.
- Private PT is visible to KVM but not used by CPU. It is used to propagate PT change to the actual protected PT which is used by CPU.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu/mmu.c | 9 ++++ arch/x86/kvm/mmu/mmu_internal.h | 73 +++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 2 + 4 files changed, 85 insertions(+)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -743,6 +743,7 @@ struct kvm_vcpu_arch {struct kvm_mmu_memory_cache mmu_shadow_page_cache;struct kvm_mmu_memory_cache mmu_shadowed_info_cache;struct kvm_mmu_memory_cache mmu_page_header_cache; +struct kvm_mmu_memory_cache mmu_private_sp_cache;/* * QEMU userspace and the guest each have their own FPU state.
arch/x86/kvm/mmu/mmu.c +9 -0
@@ -669,6 +669,13 @@ static int mmu_topup_shadow_page_cache(struct kvm_vcpu *vcpu)struct kvm_mmu_memory_cache *mc = &vcpu->arch.mmu_shadow_page_cache;int start, end, i, r; +if (kvm_gfn_shared_mask (vcpu->kvm)) { + r =kvm_mmu_topup_memory_cache (&vcpu->arch.mmu_private_sp_cache, +PT64_ROOT_MAX_LEVEL ); +if (r) +return r; + } + start =kvm_mmu_memory_cache_nr_free_objects (mc); r =kvm_mmu_topup_memory_cache (mc,PT64_ROOT_MAX_LEVEL );
@@ -718,6 +725,7 @@ static void mmu_free_memory_caches(struct kvm_vcpu *vcpu)kvm_mmu_free_memory_cache (&vcpu->arch.mmu_pte_list_desc_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_shadow_page_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_shadowed_info_cache); +kvm_mmu_free_memory_cache (&vcpu->arch.mmu_private_sp_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_page_header_cache); }
@@ -2162,6 +2170,7 @@ static struct kvm_mmu_page *kvm_mmu_alloc_shadow_page(struct kvm *kvm, sp->shadowed_translation =kvm_mmu_memory_cache_alloc (caches->shadowed_info_cache);set_page_private (virt_to_page (sp->spt), (unsigned long )sp); +kvm_mmu_init_private_sp (sp,NULL );/* * active_mmu_pages must be a FIFO list, as kvm_zap_obsolete_pages()
arch/x86/kvm/mmu/mmu_internal.h +73 -0
85 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.c +2 -0
@@ -71,6 +71,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)static void tdp_mmu_free_sp (struct kvm_mmu_page *sp) { +kvm_mmu_free_private_sp (sp);free_page ((unsigned long )sp->spt);kmem_cache_free (mmu_page_header_cache, sp); }
@@ -300,6 +301,7 @@ static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, tdp_ptep_t sptep, sp->gfn = gfn; sp->ptep = sptep; sp->tdp_mmu_page = true; +kvm_mmu_init_private_sp (sp,NULL );trace_kvm_mmu_get_page (sp, true); }
Signature · 1 line
2.25.1
Re: [PATCH v8 042/103] KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page
Quoted 120 lines · On Sun, Aug 07, 2022 at 03:01:27PM -0700, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
For private GPA, CPU refers a private page table whose contents are encrypted. The dedicated APIs to operate on it (e.g. updating/reading its PTE entry) are used and their cost is expensive.
When KVM resolves KVM page fault, it walks the page tables. To reuse the existing KVM MMU code and mitigate the heavy cost to directly walk protected (encrypted) page table, allocate one more page to copy the protected page table for KVM MMU code to directly walk. Resolve KVM page fault with the existing code, and do additional operations necessary for the protected page table. To distinguish such cases, the existing KVM page table is called a shared page table (i.e. not associated with protected page table), and the page table with protected page table is called a private page table. The relationship is depicted below.
Add a private pointer to struct
kvm_mmu_pagefor protected page table and add helper functions to allocate/initialize/free a protected page table page.KVM page fault | | | V | -------------+---------- | | | | V V | shared GPA private GPA | | | | V V | shared PT root private PT root | protected PT root | | | | V V | V shared PT private PT ----propagate----> protected PT | | | | | \-----------------+------\ | | | | | V | V V shared guest page | private guest page | non-encrypted memory | encrypted memory |PT: page table
- Shared PT is visible to KVM and it is used by CPU.
- Protected PT is used by CPU but it is invisible to KVM.
- Private PT is visible to KVM but not used by CPU. It is used to propagate PT change to the actual protected PT which is used by CPU.Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu/mmu.c | 9 ++++ arch/x86/kvm/mmu/mmu_internal.h | 73 +++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 2 + 4 files changed, 85 insertions(+)arch/x86/include/asm/kvm_host.h +1
-0@@ -743,6 +743,7 @@ struct kvm_vcpu_arch {struct kvm_mmu_memory_cache mmu_shadow_page_cache;struct kvm_mmu_memory_cache mmu_shadowed_info_cache;struct kvm_mmu_memory_cache mmu_page_header_cache; +struct kvm_mmu_memory_cache mmu_private_sp_cache;/* * QEMU userspace and the guest each have their own FPU state. arch/x86/kvm/mmu/mmu.c +9
-0@@ -669,6 +669,13 @@ static int mmu_topup_shadow_page_cache(struct kvm_vcpu *vcpu)struct kvm_mmu_memory_cache *mc = &vcpu->arch.mmu_shadow_page_cache;int start, end, i, r; +if (kvm_gfn_shared_mask (vcpu->kvm)) { + r =kvm_mmu_topup_memory_cache (&vcpu->arch.mmu_private_sp_cache, +PT64_ROOT_MAX_LEVEL ); +if (r) +return r; + } + start =kvm_mmu_memory_cache_nr_free_objects (mc); r =kvm_mmu_topup_memory_cache (mc,PT64_ROOT_MAX_LEVEL );@@ -718,6 +725,7 @@ static void mmu_free_memory_caches(struct kvm_vcpu *vcpu)kvm_mmu_free_memory_cache (&vcpu->arch.mmu_pte_list_desc_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_shadow_page_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_shadowed_info_cache); +kvm_mmu_free_memory_cache (&vcpu->arch.mmu_private_sp_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_page_header_cache); }@@ -2162,6 +2170,7 @@ static struct kvm_mmu_page *kvm_mmu_alloc_shadow_page(struct kvm *kvm, sp->shadowed_translation =kvm_mmu_memory_cache_alloc (caches->shadowed_info_cache);set_page_private (virt_to_page (sp->spt), (unsigned long )sp); +kvm_mmu_init_private_sp (sp,NULL );/* * active_mmu_pages must be a FIFO list, as kvm_zap_obsolete_pages() arch/x86/kvm/mmu/mmu_internal.h +4
-0@@ -82,6 +82,10 @@ struct kvm_mmu_page {*/ u64 *shadowed_translation; +#ifdef CONFIG_KVM_MMU_PRIVATE +/* associated private shadow page, e.g. SEPT page. */ +void *private_sp; +#endif /* Currently serving as active root */ union {int root_count;@@ -153,6 +157,75 @@ static inline bool is_private_sptep(u64 *sptep)return is_private_sp (sptep_to_sp (sptep)); }
arch/x86/kvm/mmu/mmu_internal.h static inline bool is_private_sptep(u64 *sptep) · line 169
Quoted 10 lines
arch/x86/kvm/mmu/mmu_internal.h +10
-0+#ifdef CONFIG_KVM_MMU_PRIVATE +static inline void *kvm_mmu_private_sp (struct kvm_mmu_page *sp) +{ +return sp->private_sp; +} + +static inline void kvm_mmu_init_private_sp (struct kvm_mmu_page *sp,void *private_sp) +{ + sp->private_sp = private_sp; +}
How about "kvm_mmu_set_private_sp()" ? Because it just sets variable. "init" means things more than just setting.
arch/x86/kvm/mmu/mmu_internal.h static inline bool is_private_sptep(u64 *sptep) · line 179
Quoted 10 lines
arch/x86/kvm/mmu/mmu_internal.h +10
-0+ +static inline void kvm_mmu_alloc_private_sp ( +struct kvm_vcpu *vcpu,struct kvm_mmu_memory_cache *private_sp_cache, +struct kvm_mmu_page *sp) +{ +/* +* vcpu == NULL means non-root SPT: +* vcpu == NULL is used to split a large SPT into smaller SPT. Root SPT +* is not a large SPT. +*/
Can we rely on sp->tdp_mmu_root_count to know it's root or not ? Depends on vcpu looks odd, because it's often to split huge pages in per-vcpu level, or things I missed here...
arch/x86/kvm/mmu/mmu_internal.h static inline bool is_private_sptep(u64 *sptep) · line 184
Quoted 5 lines
arch/x86/kvm/mmu/mmu_internal.h +5
-0+bool is_root = vcpu && + vcpu->arch.root_mmu.root_role.level == sp->role.level; + +if (vcpu) + private_sp_cache = &vcpu->arch.mmu_private_sp_cache;
how about totally rely on caller for this, e.g caller to decide uses per-vcpu cache or per-VM cache ?
arch/x86/kvm/mmu/mmu_internal.h static inline bool is_private_sptep(u64 *sptep) · line 185
Quoted 1 line
arch/x86/kvm/mmu/mmu_internal.h +1
-0+WARN_ON (!kvm_mmu_page_role_is_private (sp->role));
Please move to begin of this function because this should be first step to check bad cases.
Quoted 59 lines
arch/x86/kvm/mmu/mmu_internal.h +43
-0+if (is_root) +/* +* Because TDX module assigns root Secure-EPT page and set it to +* Secure-EPTP when TD vcpu is created, secure page table for +* root isn't needed. +*/ + sp->private_sp =NULL ; +else { + sp->private_sp =kvm_mmu_memory_cache_alloc (private_sp_cache); +/* +* Because mmu_private_sp_cache is topped up before staring kvm +* page fault resolving, the allocation above shouldn't fail. +*/ +WARN_ON_ONCE (!sp->private_sp); + } +} + +static inline void kvm_mmu_free_private_sp (struct kvm_mmu_page *sp) +{ +if (sp->private_sp) +free_page ((unsigned long )sp->private_sp); +} +#else +static inline void *kvm_mmu_private_sp (struct kvm_mmu_page *sp) +{ +return NULL ; +} + +static inline void kvm_mmu_init_private_sp (struct kvm_mmu_page *sp,void *private_sp) +{ +} + +static inline void kvm_mmu_alloc_private_sp ( +struct kvm_vcpu *vcpu,struct kvm_mmu_memory_cache *private_sp_cache, +struct kvm_mmu_page *sp) +{ +} + +static inline void kvm_mmu_free_private_sp (struct kvm_mmu_page *sp) +{ +} +#endif +static inline bool kvm_mmu_page_ad_need_write_protect (struct kvm_mmu_page *sp) {/* arch/x86/kvm/mmu/tdp_mmu.c +1
-0@@ -71,6 +71,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)static void tdp_mmu_free_sp (struct kvm_mmu_page *sp) { +kvm_mmu_free_private_sp (sp);free_page ((unsigned long )sp->spt);kmem_cache_free (mmu_page_header_cache, sp); }
arch/x86/kvm/mmu/tdp_mmu.c static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, tdp_ptep_t sptep, · line 304
Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +1
-0@@ -300,6 +301,7 @@ static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, tdp_ptep_t sptep, sp->gfn = gfn; sp->ptep = sptep; sp->tdp_mmu_page = true; +kvm_mmu_init_private_sp (sp,NULL );
This set the private sp allocated by kvm_mmu_alloc_private_sp(), I think it's wrong here.
Quoted 6 lines
arch/x86/kvm/mmu/tdp_mmu.c +0
-1trace_kvm_mmu_get_page (sp, true); } --2.25.1
PATCH v8 43/103 KVM: x86/tdp_mmu: Don't zap private pages for unsupported cases
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX supports only write-back(WB) memory type for private memory architecturally so that (virtualized) memory type change doesn't make sense for private memory. Also currently, page migration isn't supported for TDX yet. (TDX architecturally supports page migration. it's KVM and kernel implementation issue.)
Regarding memory type change (mtrr virtualization and lapic page mapping change), pages are zapped by kvm_zap_gfn_range(). On the next KVM page fault, the SPTE entry with a new memory type for the page is populated. Regarding page migration, pages are zapped by the mmu notifier. On the next KVM page fault, the new migrated page is populated. Don't zap private pages on unmapping for those two cases.
When deleting/moving a KVM memory slot, zap private pages. Typically tearing down VM. Don't invalidate private page tables. i.e. zap only leaf SPTEs for KVM mmu that has a shared bit mask. The existing kvm_tdp_mmu_invalidate_all_roots() depends on role.invalid with read-lock of mmu_lock so that other vcpu can operate on KVM mmu concurrently. It marks the root page table invalid and zaps SPTEs of the root page tables. The TDX module doesn't allow to unlink a protected root page table from the hardware and then allocate a new one for it. i.e. replacing a protected root page table. Instead, zap only leaf SPTEs for KVM mmu with a shared bit mask set.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/mmu/mmu.c | 58 ++++++++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 24 ++++++++++++---- arch/x86/kvm/mmu/tdp_mmu.h | 5 ++-- 3 files changed, 77 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +55 -3
@@ -1589,7 +1589,12 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) flush =kvm_handle_gfn_range (kvm, range, kvm_zap_rmap);if (is_tdp_mmu_enabled (kvm)) - flush =kvm_tdp_mmu_unmap_gfn_range (kvm, range, flush); +/* +* kvm_unmap_gfn_range() is called via mmu notifier. +* For now page migration for private page isn't supported yet, +* don't zap private pages. +*/ + flush =kvm_tdp_mmu_unmap_gfn_range (kvm, range, flush, false);return flush; }
@@ -6039,11 +6044,48 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)return unlikely (!list_empty_careful (&kvm->arch.zapped_obsolete_pages)); } +static void kvm_mmu_zap_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) +{ +bool flush = false; + +write_lock (&kvm->mmu_lock); + +/* +* Zapping non-leaf SPTEs, a.k.a. not-last SPTEs, isn't required, worst +* case scenario we'll have unused shadow pages lying around until they +* are recycled due to age or when the VM is destroyed. +*/ +if (is_tdp_mmu_enabled (kvm)) { +struct kvm_gfn_range range = { + .slot = slot, + .start = slot->base_gfn, + .end = slot->base_gfn + slot->npages, + .may_block = false, + }; + +/* +* this handles both private gfn and shared gfn. +* All private page should be zapped on memslot deletion. +*/ + flush =kvm_tdp_mmu_unmap_gfn_range (kvm, &range, flush, true); + }else { + flush =slot_handle_level (kvm, slot, __kvm_zap_rmap,PG_LEVEL_4K , +KVM_MAX_HUGEPAGE_LEVEL , true); + } +if (flush) +kvm_flush_remote_tlbs (kvm); + +write_unlock (&kvm->mmu_lock); +} +static void kvm_mmu_invalidate_zap_pages_in_memslot (struct kvm *kvm,struct kvm_memory_slot *slot,struct kvm_page_track_notifier_node *node) { -kvm_mmu_zap_all_fast (kvm); +if (kvm_gfn_shared_mask (kvm)) +kvm_mmu_zap_memslot (kvm, slot); +else +kvm_mmu_zap_all_fast (kvm); }int kvm_mmu_init_vm (struct kvm *kvm)
@@ -6145,8 +6187,18 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)if (is_tdp_mmu_enabled (kvm)) {for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) +/* +* zap_private = false. Zap only shared pages. +* +* kvm_zap_gfn_range() is used when PAT memory type was +* changed. Later on the next kvm page fault, populate +* it with updated spte entry. +* Because only WB is supported for private pages, don't +* care of private pages. +*/ flush =kvm_tdp_mmu_zap_leafs (kvm, i, gfn_start, - gfn_end, true, flush); + gfn_end, true, flush, + false); }if (flush)
arch/x86/kvm/mmu/tdp_mmu.c +19 -5
@@ -938,7 +938,8 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)* operation can cause a soft lockup. */ static bool tdp_mmu_zap_leafs (struct kvm *kvm,struct kvm_mmu_page *root, -gfn_t start,gfn_t end,bool can_yield,bool flush) +gfn_t start,gfn_t end,bool can_yield,bool flush, +bool zap_private) {struct tdp_iter iter;
@@ -946,6 +947,10 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,lockdep_assert_held_write (&kvm->mmu_lock); +WARN_ON_ONCE (zap_private && !is_private_sp (root)); +if (!zap_private &&is_private_sp (root)) +return false; +rcu_read_lock ();for_each_tdp_pte_min_level (iter, root,PG_LEVEL_4K , start, end) {
@@ -978,12 +983,13 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,* more SPTEs were zapped since the MMU lock was last acquired. */ bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,int as_id,gfn_t start,gfn_t end, -bool can_yield,bool flush) +bool can_yield,bool flush,bool zap_private) {struct kvm_mmu_page *root;for_each_tdp_mmu_root_yield_safe (kvm, root, as_id) - flush =tdp_mmu_zap_leafs (kvm, root, start, end, can_yield, flush); + flush =tdp_mmu_zap_leafs (kvm, root, start, end, can_yield, flush, + zap_private &&is_private_sp (root));return flush; }
@@ -1043,6 +1049,12 @@ void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm)lockdep_assert_held_write (&kvm->mmu_lock);list_for_each_entry (root, &kvm->arch.tdp_mmu_roots, link) { +/* +* Skip private root since private page table +* is only torn down when VM is destroyed. +*/ +if (is_private_sp (root)) +continue ;if (!root->role.invalid && !WARN_ON_ONCE (!kvm_tdp_mmu_get_root (root))) { root->role.invalid = true;
@@ -1234,11 +1246,13 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return ret; } +/* Used by mmu notifier via kvm_unmap_gfn_range() */ bool kvm_tdp_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range, -bool flush) +bool flush,bool zap_private) {return kvm_tdp_mmu_zap_leafs (kvm, range->slot->as_id, range->start, - range->end, range->may_block, flush); + range->end, range->may_block, flush, + zap_private); } typedef bool (*tdp_handler_t)(struct kvm *kvm, struct tdp_iter *iter,
arch/x86/kvm/mmu/tdp_mmu.h +3 -2
@@ -16,7 +16,8 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root, bool shared);bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,int as_id,gfn_t start, -gfn_t end,bool can_yield,bool flush); +gfn_t end,bool can_yield,bool flush, +bool zap_private);bool kvm_tdp_mmu_zap_sp (struct kvm *kvm,struct kvm_mmu_page *sp);void kvm_tdp_mmu_zap_all (struct kvm *kvm);void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm);
@@ -25,7 +26,7 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm);int kvm_tdp_mmu_map (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);bool kvm_tdp_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range, -bool flush); +bool flush,bool zap_private);bool kvm_tdp_mmu_age_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_tdp_mmu_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_tdp_mmu_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range);
Signature · 1 line
2.25.1
PATCH v8 44/103 KVM: x86/tdp_mmu: Support TDX private mapping for TDP MMU
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Allocate protected page table for private page table, and add hooks to operate on protected page table. This patch adds allocation/free of protected page tables and hooks. When calling hooks to update SPTE entry, freeze the entry, call hooks and unfree the entry to allow concurrent updates on page tables. Which is the advantage of TDP MMU. As kvm_gfn_shared_mask() returns false always, those hooks aren't called yet with this patch.
When the faulting GPA is private, the KVM fault is called private. When resolving private KVM, allocate protected page table and call hooks to operate on protected page table. On the change of the private PTE entry, invoke kvm_x86_ops hook in __handle_changed_spte() to propagate the change to protected page table. The following depicts the relationship.
private KVM page fault |
| |
V |
private GPA | CPU protected EPTP
| | |
V | V
private PT root | protected PT root
| | |
V | V
private PT --hook to propagate-->protected PT
| | |
\--------------------+------\ |
| | |
| V V
| private guest page
|
|
non-encrypted memory | encrypted memory
|PT: page table
The existing KVM TDP MMU code uses atomic update of SPTE. On populating the EPT entry, atomically set the entry. However, it requires TLB shootdown to zap SPTE. To address it, the entry is frozen with the special SPTE value that clears the present bit. After the TLB shootdown, the entry is set to the eventual value (unfreeze).
For protected page table, hooks are called to update protected page table in addition to direct access to the private SPTE. For the zapping case, it works to freeze the SPTE. It can call hooks in addition to TLB shootdown. For populating the private SPTE entry, there can be a race condition without further protection
vcpu 1: populating 2M private SPTE
vcpu 2: populating 4K private SPTE
vcpu 2: TDX SEAMCALL to update 4K protected SPTE => error vcpu 1: TDX SEAMCALL to update 2M protected SPTE
To avoid the race, the frozen SPTE is utilized. Instead of atomic update of the private entry, freeze the entry, call the hook that update protected SPTE, set the entry to the final value.
Support 4K page only at this stage. 2M page support can be done in future patches.
Add is_private member to kvm_page_fault to indicate the fault is private. Also is_private member to struct tdp_inter to propagate it.
Co-developed-by Kai Huang <[email protected]>
Signed-off-by Kai Huang <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
Acked-by Isaku Yamahata <[email protected]>
--- arch/x86/include/asm/kvm-x86-ops.h | 2 + arch/x86/include/asm/kvm_host.h | 20 +++ arch/x86/kvm/mmu/mmu.c | 15 +- arch/x86/kvm/mmu/mmu_internal.h | 35 +++++ arch/x86/kvm/mmu/tdp_iter.h | 2 +- arch/x86/kvm/mmu/tdp_mmu.c | 222 ++++++++++++++++++++++++----- arch/x86/kvm/mmu/tdp_mmu.h | 2 +- virt/kvm/kvm_main.c | 1 + 8 files changed, 259 insertions(+), 40 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +2 -0
@@ -93,6 +93,8 @@ KVM_X86_OP_OPTIONAL_RET0(set_tss_addr) KVM_X86_OP_OPTIONAL_RET0(set_identity_map_addr) KVM_X86_OP_OPTIONAL_RET0(get_mt_mask) KVM_X86_OP(load_mmu_pgd) +KVM_X86_OP_OPTIONAL(free_private_sp) +KVM_X86_OP_OPTIONAL(handle_changed_private_spte) KVM_X86_OP(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier)
arch/x86/include/asm/kvm_host.h +20 -0
@@ -466,6 +466,7 @@ struct kvm_mmu { struct kvm_mmu_page *sp);void (*invlpg)(struct kvm_vcpu *vcpu,gva_t gva,hpa_t root_hpa);struct kvm_mmu_root_info root; +hpa_t private_root_hpa;union kvm_cpu_role cpu_role;union kvm_mmu_page_role root_role;
@@ -1482,6 +1483,20 @@ static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical)return dest_mode_logical ?APIC_DEST_LOGICAL :APIC_DEST_PHYSICAL ; } +struct kvm_spte { +kvm_pfn_t pfn; +bool is_present; +bool is_last; +}; + +struct kvm_spte_change { +gfn_t gfn; +enum pg_level level; +struct kvm_spte old; +struct kvm_spte new; +void *sept_page; +}; +struct kvm_x86_ops {const char *name;
@@ -1594,6 +1609,11 @@ struct kvm_x86_ops {void (*load_mmu_pgd)(struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level); +int (*free_private_sp)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +void *private_sp); +void (*handle_changed_private_spte)( +struct kvm *kvm,const struct kvm_spte_change *change); +bool (*has_wbinvd_exit)(void );u64 (*get_l2_tsc_offset)(struct kvm_vcpu *vcpu);
arch/x86/kvm/mmu/mmu.c +12 -3
@@ -3615,7 +3615,12 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) goto out_unlock;if (is_tdp_mmu_enabled (vcpu->kvm)) { - root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu); +if (kvm_gfn_shared_mask (vcpu->kvm) && + !VALID_PAGE (mmu->private_root_hpa)) { + root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu, true); + mmu->private_root_hpa = root; + } + root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu, false); mmu->root.hpa = root; }else if (shadow_root_level >=PT64_ROOT_4LEVEL ) { root =mmu_alloc_root (vcpu,0 ,0 , shadow_root_level);
@@ -4284,7 +4289,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultunsigned long mmu_seq;int r; - fault->gfn = fault->addr >>PAGE_SHIFT ; + fault->gfn =gpa_to_gfn (fault->addr) & ~kvm_gfn_shared_mask (vcpu->kvm); fault->slot =kvm_vcpu_gfn_to_memslot (vcpu, fault->gfn);if (page_fault_handle_page_track (vcpu, fault))
@@ -5830,6 +5835,7 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) mmu->root.hpa =INVALID_PAGE ; mmu->root.pgd =0 ; + mmu->private_root_hpa =INVALID_PAGE ;for (i =0 ; i <KVM_MMU_NUM_PREV_ROOTS ; i++) mmu->prev_roots[i] =KVM_MMU_ROOT_INFO_INVALID ;
@@ -6064,7 +6070,7 @@ static void kvm_mmu_zap_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) };/* -* this handles both private gfn and shared gfn. +* This handles both private gfn and shared gfn. * All private page should be zapped on memslot deletion. */ flush =kvm_tdp_mmu_unmap_gfn_range (kvm, &range, flush, true);
@@ -6892,6 +6898,9 @@ int kvm_mmu_vendor_module_init(void)void kvm_mmu_destroy (struct kvm_vcpu *vcpu) {kvm_mmu_unload (vcpu); +if (is_tdp_mmu_enabled (vcpu->kvm)) +mmu_free_root_page (vcpu->kvm, &vcpu->arch.mmu->private_root_hpa, +NULL );free_mmu_pages (&vcpu->arch.root_mmu);free_mmu_pages (&vcpu->arch.guest_mmu);mmu_free_memory_caches (vcpu);
arch/x86/kvm/mmu/mmu_internal.h +35 -0
@@ -6,6 +6,8 @@#include <linux/kvm_host.h> #include <asm/kvm_host.h> +#include "mmu.h" +#undef MMU_DEBUG#ifdef MMU_DEBUG
@@ -200,11 +202,30 @@ static inline void kvm_mmu_alloc_private_sp( } } +static inline int kvm_alloc_private_sp_for_split ( +struct kvm_mmu_page *sp,gfp_t gfp) +{ + gfp &= ~__GFP_ZERO; + sp->private_sp = (void *)__get_free_page (gfp); +if (!sp->private_sp) +return -ENOMEM ; +return 0 ; +} +static inline void kvm_mmu_free_private_sp (struct kvm_mmu_page *sp) {if (sp->private_sp)free_page ((unsigned long )sp->private_sp); } + +static inline gfn_t kvm_gfn_for_root (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t gfn) +{ +if (is_private_sp (root)) +return kvm_gfn_private (kvm, gfn); +else +return kvm_gfn_shared (kvm, gfn); +}#else static inline void *kvm_mmu_private_sp (struct kvm_mmu_page *sp) {
@@ -221,9 +242,21 @@ static inline void kvm_mmu_alloc_private_sp( { } +static inline int kvm_alloc_private_sp_for_split ( +struct kvm_mmu_page *sp,gfp_t gfp) +{ +return -ENOMEM ; +} +static inline void kvm_mmu_free_private_sp (struct kvm_mmu_page *sp) { } + +static inline gfn_t kvm_gfn_for_root (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t gfn) +{ +return gfn; +}#endif static inline bool kvm_mmu_page_ad_need_write_protect (struct kvm_mmu_page *sp)
@@ -273,6 +306,7 @@ struct kvm_page_fault {/* Derived from mmu and global state. */ const bool is_tdp;const bool nx_huge_page_workaround_enabled; +const bool is_private;/* * Whether a >4KB mapping can be created or is forbidden due to NX
@@ -355,6 +389,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, .is_tdp = likely(vcpu->arch.mmu->page_fault == kvm_tdp_page_fault), .nx_huge_page_workaround_enabled = is_nx_huge_page_enabled(vcpu->kvm), + .is_private = kvm_is_private_gpa(vcpu->kvm, cr2_or_gpa), .max_level = vcpu->kvm->arch.tdp_max_page_level, .req_level = PG_LEVEL_4K,
arch/x86/kvm/mmu/tdp_iter.h +1 -1
@@ -71,7 +71,7 @@ struct tdp_iter {tdp_ptep_t pt_path[PT64_ROOT_MAX_LEVEL ];/* A pointer to the current SPTE */ tdp_ptep_t sptep; -/* The lowest GFN mapped by the current SPTE */ +/* The lowest GFN (shared bits included) mapped by the current SPTE */ gfn_t gfn;/* The level of the root page given to the iterator */ int root_level;
arch/x86/kvm/mmu/tdp_mmu.c +187 -35
387 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.h +1 -1
@@ -5,7 +5,7 @@#include <linux/kvm_host.h> -hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu); +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu,bool private);__must_check static inline bool kvm_tdp_mmu_get_root (struct kvm_mmu_page *root) {
virt/kvm/kvm_main.c +1 -0
@@ -213,6 +213,7 @@ struct page *kvm_pfn_to_refcounted_page(kvm_pfn_t pfn)return NULL ; } +EXPORT_SYMBOL_GPL (kvm_pfn_to_refcounted_page);/* * Switches to specified vcpu, until a matching vcpu_put()
Signature · 1 line
2.25.1
Re: [PATCH v8 044/103] KVM: x86/tdp_mmu: Support TDX private mapping for TDP MMU
Quoted 442 lines · On Sun, Aug 07, 2022 at 03:01:29PM -0700, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Allocate protected page table for private page table, and add hooks to operate on protected page table. This patch adds allocation/free of protected page tables and hooks. When calling hooks to update SPTE entry, freeze the entry, call hooks and unfree the entry to allow concurrent updates on page tables. Which is the advantage of TDP MMU. As
kvm_gfn_shared_mask()returns false always, those hooks aren't called yet with this patch.When the faulting GPA is private, the KVM fault is called private. When resolving private KVM, allocate protected page table and call hooks to operate on protected page table. On the change of the private PTE entry, invoke
kvm_x86_opshook in__handle_changed_spte()to propagate the change to protected page table. The following depicts the relationship.private KVM page fault | | | V | private GPA | CPU protected EPTP | | | V | V private PT root | protected PT root | | | V | V private PT --hook to propagate-->protected PT | | | \--------------------+------\ | | | | | V V | private guest page | | non-encrypted memory | encrypted memory |PT: page table
The existing KVM TDP MMU code uses atomic update of SPTE. On populating the EPT entry, atomically set the entry. However, it requires TLB shootdown to zap SPTE. To address it, the entry is frozen with the special SPTE value that clears the present bit. After the TLB shootdown, the entry is set to the eventual value (unfreeze).
For protected page table, hooks are called to update protected page table in addition to direct access to the private SPTE. For the zapping case, it works to freeze the SPTE. It can call hooks in addition to TLB shootdown. For populating the private SPTE entry, there can be a race condition without further protection
vcpu 1: populating 2M private SPTE
vcpu 2: populating 4K private SPTE
vcpu 2: TDX SEAMCALL to update 4K protected SPTE => error vcpu 1: TDX SEAMCALL to update 2M protected SPTETo avoid the race, the frozen SPTE is utilized. Instead of atomic update of the private entry, freeze the entry, call the hook that update protected SPTE, set the entry to the final value.
Support 4K page only at this stage. 2M page support can be done in future patches.
Add
is_privatemember tokvm_page_faultto indicate the fault is private. Alsois_privatemember to structtdp_interto propagate it.Co-developed-by Kai Huang <[email protected]>Signed-off-by Kai Huang <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Acked-by Isaku Yamahata <[email protected]>--- arch/x86/include/asm/kvm-x86-ops.h | 2 + arch/x86/include/asm/kvm_host.h | 20 +++ arch/x86/kvm/mmu/mmu.c | 15 +- arch/x86/kvm/mmu/mmu_internal.h | 35 +++++ arch/x86/kvm/mmu/tdp_iter.h | 2 +- arch/x86/kvm/mmu/tdp_mmu.c | 222 ++++++++++++++++++++++++----- arch/x86/kvm/mmu/tdp_mmu.h | 2 +- virt/kvm/kvm_main.c | 1 + 8 files changed, 259 insertions(+), 40 deletions(-)arch/x86/include/asm/kvm-x86-ops.h +2
-0@@ -93,6 +93,8 @@ KVM_X86_OP_OPTIONAL_RET0(set_tss_addr) KVM_X86_OP_OPTIONAL_RET0(set_identity_map_addr) KVM_X86_OP_OPTIONAL_RET0(get_mt_mask) KVM_X86_OP(load_mmu_pgd) +KVM_X86_OP_OPTIONAL(free_private_sp) +KVM_X86_OP_OPTIONAL(handle_changed_private_spte) KVM_X86_OP(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier)arch/x86/include/asm/kvm_host.h +20
-0@@ -466,6 +466,7 @@ struct kvm_mmu { struct kvm_mmu_page *sp);void (*invlpg)(struct kvm_vcpu *vcpu,gva_t gva,hpa_t root_hpa);struct kvm_mmu_root_info root; +hpa_t private_root_hpa;union kvm_cpu_role cpu_role;union kvm_mmu_page_role root_role;@@ -1482,6 +1483,20 @@ static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical)return dest_mode_logical ?APIC_DEST_LOGICAL :APIC_DEST_PHYSICAL ; } +struct kvm_spte { +kvm_pfn_t pfn; +bool is_present; +bool is_last; +}; + +struct kvm_spte_change { +gfn_t gfn; +enum pg_level level; +struct kvm_spte old; +struct kvm_spte new; +void *sept_page; +}; +struct kvm_x86_ops {const char *name;@@ -1594,6 +1609,11 @@ struct kvm_x86_ops {void (*load_mmu_pgd)(struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level); +int (*free_private_sp)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +void *private_sp); +void (*handle_changed_private_spte)( +struct kvm *kvm,const struct kvm_spte_change *change); +bool (*has_wbinvd_exit)(void );u64 (*get_l2_tsc_offset)(struct kvm_vcpu *vcpu);arch/x86/kvm/mmu/mmu.c +12
-3@@ -3615,7 +3615,12 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) goto out_unlock;if (is_tdp_mmu_enabled (vcpu->kvm)) { - root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu); +if (kvm_gfn_shared_mask (vcpu->kvm) && + !VALID_PAGE (mmu->private_root_hpa)) { + root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu, true); + mmu->private_root_hpa = root; + } + root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu, false); mmu->root.hpa = root; }else if (shadow_root_level >=PT64_ROOT_4LEVEL ) { root =mmu_alloc_root (vcpu,0 ,0 , shadow_root_level);@@ -4284,7 +4289,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultunsigned long mmu_seq;int r; - fault->gfn = fault->addr >>PAGE_SHIFT ; + fault->gfn =gpa_to_gfn (fault->addr) & ~kvm_gfn_shared_mask (vcpu->kvm); fault->slot =kvm_vcpu_gfn_to_memslot (vcpu, fault->gfn);if (page_fault_handle_page_track (vcpu, fault))@@ -5830,6 +5835,7 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) mmu->root.hpa =INVALID_PAGE ; mmu->root.pgd =0 ; + mmu->private_root_hpa =INVALID_PAGE ;for (i =0 ; i <KVM_MMU_NUM_PREV_ROOTS ; i++) mmu->prev_roots[i] =KVM_MMU_ROOT_INFO_INVALID ;@@ -6064,7 +6070,7 @@ static void kvm_mmu_zap_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) };/* -* this handles both private gfn and shared gfn. +* This handles both private gfn and shared gfn. * All private page should be zapped on memslot deletion. */ flush =kvm_tdp_mmu_unmap_gfn_range (kvm, &range, flush, true);@@ -6892,6 +6898,9 @@ int kvm_mmu_vendor_module_init(void)void kvm_mmu_destroy (struct kvm_vcpu *vcpu) {kvm_mmu_unload (vcpu); +if (is_tdp_mmu_enabled (vcpu->kvm)) +mmu_free_root_page (vcpu->kvm, &vcpu->arch.mmu->private_root_hpa, +NULL );free_mmu_pages (&vcpu->arch.root_mmu);free_mmu_pages (&vcpu->arch.guest_mmu);mmu_free_memory_caches (vcpu);arch/x86/kvm/mmu/mmu_internal.h +35
-0@@ -6,6 +6,8 @@#include <linux/kvm_host.h> #include <asm/kvm_host.h> +#include "mmu.h" +#undef MMU_DEBUG#ifdef MMU_DEBUG @@ -200,11 +202,30 @@ static inline void kvm_mmu_alloc_private_sp( } } +static inline int kvm_alloc_private_sp_for_split ( +struct kvm_mmu_page *sp,gfp_t gfp) +{ + gfp &= ~__GFP_ZERO; + sp->private_sp = (void *)__get_free_page (gfp); +if (!sp->private_sp) +return -ENOMEM ; +return 0 ; +} +static inline void kvm_mmu_free_private_sp (struct kvm_mmu_page *sp) {if (sp->private_sp)free_page ((unsigned long )sp->private_sp); } + +static inline gfn_t kvm_gfn_for_root (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t gfn) +{ +if (is_private_sp (root)) +return kvm_gfn_private (kvm, gfn); +else +return kvm_gfn_shared (kvm, gfn); +}#else static inline void *kvm_mmu_private_sp (struct kvm_mmu_page *sp) {@@ -221,9 +242,21 @@ static inline void kvm_mmu_alloc_private_sp( { } +static inline int kvm_alloc_private_sp_for_split ( +struct kvm_mmu_page *sp,gfp_t gfp) +{ +return -ENOMEM ; +} +static inline void kvm_mmu_free_private_sp (struct kvm_mmu_page *sp) { } + +static inline gfn_t kvm_gfn_for_root (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t gfn) +{ +return gfn; +}#endif static inline bool kvm_mmu_page_ad_need_write_protect (struct kvm_mmu_page *sp)@@ -273,6 +306,7 @@ struct kvm_page_fault {/* Derived from mmu and global state. */ const bool is_tdp;const bool nx_huge_page_workaround_enabled; +const bool is_private;/* * Whether a >4KB mapping can be created or is forbidden due to NX @@ -355,6 +389,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, .is_tdp = likely(vcpu->arch.mmu->page_fault == kvm_tdp_page_fault), .nx_huge_page_workaround_enabled = is_nx_huge_page_enabled(vcpu->kvm), + .is_private = kvm_is_private_gpa(vcpu->kvm, cr2_or_gpa), .max_level = vcpu->kvm->arch.tdp_max_page_level, .req_level = PG_LEVEL_4K,arch/x86/kvm/mmu/tdp_iter.h +1
-1@@ -71,7 +71,7 @@ struct tdp_iter {tdp_ptep_t pt_path[PT64_ROOT_MAX_LEVEL ];/* A pointer to the current SPTE */ tdp_ptep_t sptep; -/* The lowest GFN mapped by the current SPTE */ +/* The lowest GFN (shared bits included) mapped by the current SPTE */ gfn_t gfn;/* The level of the root page given to the iterator */ int root_level;arch/x86/kvm/mmu/tdp_mmu.c +61
-14@@ -285,6 +285,11 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm_vcpu *vcpu, sp->spt =kvm_mmu_memory_cache_alloc (&vcpu->arch.mmu_shadow_page_cache); sp->role = role; +if (kvm_mmu_page_role_is_private (role)) +kvm_mmu_alloc_private_sp (vcpu,NULL , sp); +else +kvm_mmu_init_private_sp (sp,NULL ); +return sp; }@@ -301,12 +306,12 @@ static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, tdp_ptep_t sptep, sp->gfn = gfn; sp->ptep = sptep; sp->tdp_mmu_page = true; -kvm_mmu_init_private_sp (sp,NULL );trace_kvm_mmu_get_page (sp, true); } -hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu) +static struct kvm_mmu_page *kvm_tdp_mmu_get_vcpu_root (struct kvm_vcpu *vcpu, +bool private) {union kvm_mmu_page_role role = vcpu->arch.mmu->root_role;struct kvm *kvm = vcpu->kvm;@@ -318,6 +323,8 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)* Check for an existing root before allocating a new one. Note, the * role check prevents consuming an invalid root. */ +if (private) +kvm_mmu_page_role_set_private (&role);for_each_tdp_mmu_root (kvm, root,kvm_mmu_role_as_id (role)) {if (root->role.word == role.word &&kvm_tdp_mmu_get_root (root))@@ -334,12 +341,17 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)spin_unlock (&kvm->arch.tdp_mmu_pages_lock); out: -return __pa (root->spt); +return root; +} + +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu,bool private) +{ +return __pa (kvm_tdp_mmu_get_vcpu_root (vcpu, private)->spt); }static void handle_changed_spte (struct kvm *kvm,int as_id,gfn_t gfn, -u64 old_spte,u64 new_spte,int level, -bool shared); +u64 old_spte,u64 new_spte, +union kvm_mmu_page_role role,bool shared);static void handle_changed_spte_acc_track (u64 old_spte,u64 new_spte,int level) {@@ -365,6 +377,8 @@ static void handle_changed_spte_dirty_log(struct kvm *kvm, int as_id, gfn_t gfn,if ((!is_writable_pte (old_spte) || pfn_changed) &&is_writable_pte (new_spte)) { +/* For memory slot operations, use GFN without aliasing */ + gfn = gfn & ~kvm_gfn_shared_mask (kvm); slot =__gfn_to_memslot (__kvm_memslots (kvm, as_id), gfn);mark_page_dirty_in_slot (kvm, slot, gfn); }@@ -489,7 +503,18 @@ static void handle_removed_pt(struct kvm *kvm, tdp_ptep_t pt, bool shared) REMOVED_SPTE, level); }handle_changed_spte (kvm,kvm_mmu_page_as_id (sp), gfn, - old_spte,REMOVED_SPTE , level, shared); + old_spte,REMOVED_SPTE , sp->role, shared); + } + +if (is_private_sp (sp) &&WARN_ON (static_call (kvm_x86_free_private_sp)( + kvm, sp->gfn, sp->role.level, +kvm_mmu_private_sp (sp)))) { +/* +* Failed to unlink Secure EPT page and there is nothing to do +* further. Intentionally leak the page to prevent the kernel +* from accessing the encrypted page. +*/ +kvm_mmu_init_private_sp (sp,NULL ); }call_rcu (&sp->rcu_head, tdp_mmu_free_sp_rcu_callback);@@ -502,7 +527,7 @@ static void handle_removed_pt(struct kvm *kvm, tdp_ptep_t pt, bool shared)* @gfn: the base GFN that was mapped by the SPTE * @old_spte: The value of the SPTE before the change * @new_spte: The value of the SPTE after the change -* @level: the level of the PT the SPTE is part of in the paging structure +* @role: the role of the PT the SPTE is part of in the paging structure * @shared: This operation may not be running under the exclusive use of * the MMU lock and the operation must synchronize with other * threads that might be modifying SPTEs. @@ -511,14 +536,34 @@ static void handle_removed_pt(struct kvm *kvm, tdp_ptep_t pt, bool shared)* This function must be called for all TDP SPTE modifications. */ static void __handle_changed_spte (struct kvm *kvm,int as_id,gfn_t gfn, -u64 old_spte,u64 new_spte,int level, -bool shared) +u64 old_spte,u64 new_spte, +union kvm_mmu_page_role role,bool shared) { +bool is_private =kvm_mmu_page_role_is_private (role); +int level = role.level;bool was_present =is_shadow_present_pte (old_spte);bool is_present =is_shadow_present_pte (new_spte); -bool was_leaf = was_present &&is_last_spte (old_spte, level); -bool is_leaf = is_present &&is_last_spte (new_spte, level); -bool pfn_changed =spte_to_pfn (old_spte) !=spte_to_pfn (new_spte); +bool was_last =is_last_spte (old_spte, level); +bool is_last =is_last_spte (new_spte, level); +bool was_leaf = was_present && was_last; +bool is_leaf = is_present && is_last; +kvm_pfn_t old_pfn =spte_to_pfn (old_spte); +kvm_pfn_t new_pfn =spte_to_pfn (new_spte); +bool pfn_changed = old_pfn != new_pfn; +struct kvm_spte_change change = { + .gfn = gfn, + .level = level, + .old = { + .pfn = old_pfn, + .is_present = was_present, + .is_last = was_last, + }, + .new = { + .pfn = new_pfn, + .is_present = is_present, + .is_last = is_last, + }, + };WARN_ON (level >PT64_ROOT_MAX_LEVEL );WARN_ON (level <PG_LEVEL_4K );@@ -585,7 +630,7 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,if (was_leaf &&is_dirty_spte (old_spte) && (!is_present || !is_dirty_spte (new_spte) || pfn_changed)) -kvm_set_pfn_dirty (spte_to_pfn (old_spte)); +kvm_set_pfn_dirty (old_pfn);/* * Recursively handle child PTs if the change removed a subtree from @@ -594,19 +639,48 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,* pages are kernel allocations and should never be migrated. */ if (was_present && !was_leaf && - (is_leaf || !is_present ||WARN_ON_ONCE (pfn_changed))) + (is_leaf || !is_present ||WARN_ON_ONCE (pfn_changed))) { +WARN_ON (is_private != +is_private_sptep (spte_to_child_pt (old_spte, level)));
arch/x86/kvm/mmu/tdp_mmu.c static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, · line 654
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +9
-0handle_removed_pt (kvm,spte_to_child_pt (old_spte, level), shared); + } + +/* +* Special handling for the private mapping. We are either +* setting up new mapping at middle level page table, or leaf, +* or tearing down existing mapping. +* +* This is after handling lower page table by above +* handle_remove_tdp_mmu_page(). S-EPT requires to remove S-EPT tables
s/handle_remove_tdp_mmu_page/handle_removed_pt
Quoted 277 lines
arch/x86/kvm/mmu/tdp_mmu.c +117
-21+* after removing childrens. +*/ +if (is_private && +/* Ignore change of software only bits. e.g. host_writable */ + (was_leaf != is_leaf || was_present != is_present || pfn_changed)) { +void *sept_page =NULL ; + +if (is_present && !is_leaf) { +struct kvm_mmu_page *sp =to_shadow_page (pfn_to_hpa (new_pfn)); + + sept_page =kvm_mmu_private_sp (sp); +WARN_ON (!sept_page); +WARN_ON (sp->role.level +1 != level); +WARN_ON (sp->gfn != gfn); + } + change.sept_page = sept_page; + +static_call (kvm_x86_handle_changed_private_spte)(kvm, &change); + } }static void handle_changed_spte (struct kvm *kvm,int as_id,gfn_t gfn, -u64 old_spte,u64 new_spte,int level, -bool shared) +u64 old_spte,u64 new_spte, +union kvm_mmu_page_role role,bool shared) { -__handle_changed_spte (kvm, as_id, gfn, old_spte, new_spte, level, - shared); -handle_changed_spte_acc_track (old_spte, new_spte, level); +__handle_changed_spte (kvm, as_id, gfn, old_spte, new_spte, role, shared); +handle_changed_spte_acc_track (old_spte, new_spte, role.level);handle_changed_spte_dirty_log (kvm, as_id, gfn, old_spte, - new_spte, level); + new_spte, role.level); }/* @@ -630,6 +704,24 @@ static inline int tdp_mmu_set_spte_atomic(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { +/* +* For conventional page table, the update flow is +* - update STPE with atomic operation +* - hanlde changed SPTE. __handle_changed_spte() +* NOTE: __handle_changed_spte() (and functions) must be safe against +* concurrent update. It is an exception to zap SPTE. See +* tdp_mmu_zap_spte_atomic(). +* +* For private page table, callbacks are needed to propagate SPTE +* change into the protected page table. In order to atomically update +* both the SPTE and the protected page tables with callbacks, utilize +* freezing SPTE. +* - Freeze the SPTE. Set entry to REMOVED_SPTE. +* - Trigger callbacks for protected page tables. __handle_changed_spte() +* - Unfreeze the SPTE. Set the entry to new_spte. +*/ +bool freeze_spte =is_private_sptep (iter->sptep) && !is_removed_spte (new_spte); +u64 tmp_spte = freeze_spte ?REMOVED_SPTE : new_spte;u64 *sptep =rcu_dereference (iter->sptep);/* @@ -646,13 +738,17 @@ static inline int tdp_mmu_set_spte_atomic(struct kvm *kvm,* Note, fast_pf_fix_direct_spte() can also modify TDP MMU SPTEs and * does not hold the mmu_lock. */ -if (!try_cmpxchg64 (sptep, &iter->old_spte, new_spte)) +if (!try_cmpxchg64 (sptep, &iter->old_spte, tmp_spte))return -EBUSY ; -__handle_changed_spte (kvm, iter->as_id, iter->gfn, iter->old_spte, - new_spte, iter->level, true); +__handle_changed_spte ( + kvm, iter->as_id, iter->gfn, + iter->old_spte, new_spte,sptep_to_sp (sptep)->role, true);handle_changed_spte_acc_track (iter->old_spte, new_spte, iter->level); +if (freeze_spte) +__kvm_tdp_mmu_write_spte (sptep, new_spte); +return 0 ; }@@ -719,9 +815,11 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm,* SPTE had voldatile bits. */ static u64 __tdp_mmu_set_spte (struct kvm *kvm,int as_id,tdp_ptep_t sptep, -u64 old_spte,u64 new_spte,gfn_t gfn,int level, -bool record_acc_track,bool record_dirty_log) +u64 old_spte,u64 new_spte,gfn_t gfn,int level, +bool record_acc_track,bool record_dirty_log) { +union kvm_mmu_page_role role; +lockdep_assert_held_write (&kvm->mmu_lock);/* @@ -735,7 +833,9 @@ static u64 __tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, old_spte =kvm_tdp_mmu_write_spte (sptep, old_spte, new_spte, level); -__handle_changed_spte (kvm, as_id, gfn, old_spte, new_spte, level, false); + role =sptep_to_sp (sptep)->role; + role.level = level; +__handle_changed_spte (kvm, as_id, gfn, old_spte, new_spte, role, false);if (record_acc_track)handle_changed_spte_acc_track (old_spte, new_spte, level);@@ -787,8 +887,11 @@ static inline void tdp_mmu_set_spte_no_dirty_log(struct kvm *kvm,continue ; \ else -#define tdp_mmu_for_each_pte (_iter, _mmu, _start, _end) \ - for_each_tdp_pte(_iter, to_shadow_page(_mmu->root.hpa), _start, _end) +#define tdp_mmu_for_each_pte (_iter, _mmu, _private, _start, _end) \ + for_each_tdp_pte(_iter, \ + to_shadow_page((_private) ? _mmu->private_root_hpa : \ + _mmu->root.hpa), \ + _start, _end)/* * Yield if the MMU lock is contended or this thread needs to return control @@ -951,6 +1054,14 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,if (!zap_private &&is_private_sp (root))return false; +/* +* start and end doesn't have GFN shared bit. This function zaps +* a region including alias. Adjust shared bit of [start, end) if the +* root is shared. +*/ + start =kvm_gfn_for_root (kvm, root, start); + end =kvm_gfn_for_root (kvm, root, end); +rcu_read_lock ();for_each_tdp_pte_min_level (iter, root,PG_LEVEL_4K , start, end) {@@ -1079,10 +1190,19 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu,WARN_ON (sp->role.level != fault->goal_level);if (unlikely (!fault->slot)) new_spte =make_mmio_spte (vcpu, iter->gfn,ACC_ALL ); -else - wrprot =make_spte (vcpu, sp, fault->slot,ACC_ALL , iter->gfn, - fault->pfn, iter->old_spte, fault->prefetch, true, - fault->map_writable, &new_spte); +else { +unsigned long pte_access =ACC_ALL ; +gfn_t gfn_unalias = iter->gfn & ~kvm_gfn_shared_mask (vcpu->kvm); + +/* TDX shared GPAs are no executable, enforce this for the SDV. */ +if (kvm_gfn_shared_mask (vcpu->kvm) && !fault->is_private) + pte_access &= ~ACC_EXEC_MASK ; + + wrprot =make_spte (vcpu, sp, fault->slot, pte_access, + gfn_unalias, fault->pfn, iter->old_spte, + fault->prefetch, true, fault->map_writable, + &new_spte); + }if (new_spte == iter->old_spte) ret =RET_PF_SPURIOUS ;@@ -1181,6 +1301,8 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) {struct kvm_mmu *mmu = vcpu->arch.mmu;struct tdp_iter iter; +gfn_t raw_gfn; +bool is_private = fault->is_private;int ret;kvm_mmu_hugepage_adjust (vcpu, fault);@@ -1189,7 +1311,17 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)rcu_read_lock (); -tdp_mmu_for_each_pte (iter, mmu, fault->gfn, fault->gfn +1 ) { + raw_gfn =gpa_to_gfn (fault->addr); + +if (is_error_noslot_pfn (fault->pfn) || + !kvm_pfn_to_refcounted_page (fault->pfn)) { +if (is_private) { +rcu_read_unlock (); +return -EFAULT ; + } + } + +tdp_mmu_for_each_pte (iter, mmu, is_private, raw_gfn, raw_gfn +1 ) {if (fault->nx_huge_page_workaround_enabled)disallowed_hugepage_adjust (fault, iter.old_spte, iter.level);@@ -1205,6 +1337,12 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) is_large_pte(iter.old_spte)) {if (tdp_mmu_zap_spte_atomic (vcpu->kvm, &iter))break ; +/* +* TODO: large page support. +* Doesn't support large page for TDX now +*/ +WARN_ON (is_private_sptep (iter.sptep)); +/* * The iter must explicitly re-read the spte here @@ -1448,6 +1586,12 @@ static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split( sp->role = role; sp->spt = (void *)__get_free_page (gfp); +if (kvm_mmu_page_role_is_private (role)) { +if (kvm_alloc_private_sp_for_split (sp, gfp)) { +free_page ((unsigned long )sp->spt); + sp->spt =NULL ; + } + }if (!sp->spt) {kmem_cache_free (mmu_page_header_cache, sp);return NULL ;@@ -1463,6 +1607,11 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm,union kvm_mmu_page_role role =tdp_iter_child_role (iter);struct kvm_mmu_page *sp; +WARN_ON (kvm_mmu_page_role_is_private (role) != +is_private_sptep (iter->sptep)); +/* TODO: Large page isn't supported for private SPTE yet. */ +WARN_ON (kvm_mmu_page_role_is_private (role)); +/* * Since we are allocating while under the MMU lock we have to be * careful about GFP flags. Use GFP_NOWAIT to avoid blocking on direct @@ -1897,7 +2046,7 @@ int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes,if (WARN_ON (kvm_gfn_shared_mask (vcpu->kvm)))return leaf; -tdp_mmu_for_each_pte (iter, mmu, gfn, gfn +1 ) { +tdp_mmu_for_each_pte (iter, mmu, false, gfn, gfn +1 ) { leaf = iter.level; sptes[leaf] = iter.old_spte; }@@ -1924,7 +2073,10 @@ u64 *kvm_tdp_mmu_fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, u64 addr,gfn_t gfn = addr >>PAGE_SHIFT ;tdp_ptep_t sptep =NULL ; -tdp_mmu_for_each_pte (iter, mmu, gfn, gfn +1 ) { +/* fast page fault for private GPA isn't supported. */ +WARN_ON_ONCE (kvm_is_private_gpa (vcpu->kvm, addr)); + +tdp_mmu_for_each_pte (iter, mmu, false, gfn, gfn +1 ) { *spte = iter.old_spte; sptep = iter.sptep; }arch/x86/kvm/mmu/tdp_mmu.h +1
-1@@ -5,7 +5,7 @@#include <linux/kvm_host.h> -hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu); +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu,bool private);__must_check static inline bool kvm_tdp_mmu_get_root (struct kvm_mmu_page *root) {virt/kvm/kvm_main.c +1
-1@@ -213,6 +213,7 @@ struct page *kvm_pfn_to_refcounted_page(kvm_pfn_t pfn)return NULL ; } +EXPORT_SYMBOL_GPL (kvm_pfn_to_refcounted_page);/* * Switches to specified vcpu, until a matching vcpu_put() -- 2.25.1
PATCH v8 45/103 The start of TDX KVM patch series: TDX EPT violation
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TDX EPT violation.
Signed-off-by Isaku Yamahata <[email protected]>
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -2
@@ -19,12 +19,12 @@ Patch Layer status * TDX architectural definitions: Applied * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied -* TDX EPT violation: Not yet +* TDX EPT violation: Applying * TD finalization: Not yet * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet * KVM MMU GPA shared bits: Applied * KVM TDP refactoring for TDX: Applied -* KVM TDP MMU hooks: Applying +* KVM TDP MMU hooks: Applied * KVM TDP MMU MapGPA: Not yet
Signature · 1 line
2.25.1
PATCH v8 46/103 KVM: x86/mmu: Disallow dirty logging for x86 TDX
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX doesn't support dirty logging. Report dirty logging isn't supported so that device model, for example qemu, can properly handle it.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/x86.c | 5 +++++ include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 15 ++++++++++++--- 3 files changed, 18 insertions(+), 3 deletions(-)
arch/x86/kvm/x86.c +5 -0
@@ -13356,6 +13356,11 @@ int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size, }EXPORT_SYMBOL_GPL (kvm_sev_es_string_io); +bool kvm_arch_dirty_log_supported (struct kvm *kvm) +{ +return kvm->arch.vm_type !=KVM_X86_TDX_VM ; +} +EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_entry);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_exit);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_fast_mmio);
include/linux/kvm_host.h +1 -0
@@ -1448,6 +1448,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);int kvm_arch_post_init_vm (struct kvm *kvm);void kvm_arch_pre_destroy_vm (struct kvm *kvm);int kvm_arch_create_vm_debugfs (struct kvm *kvm); +bool kvm_arch_dirty_log_supported (struct kvm *kvm);#ifndef __KVM_HAVE_ARCH_VM_ALLOC/*
virt/kvm/kvm_main.c +12 -3
@@ -1511,9 +1511,18 @@ static void kvm_replace_memslot(struct kvm *kvm, } } -static int check_memory_region_flags (const struct kvm_userspace_memory_region *mem) +bool __weak kvm_arch_dirty_log_supported (struct kvm *kvm) { -u32 valid_flags =KVM_MEM_LOG_DIRTY_PAGES ; +return true; +} + +static int check_memory_region_flags (struct kvm *kvm, +const struct kvm_userspace_memory_region *mem) +{ +u32 valid_flags =0 ; + +if (kvm_arch_dirty_log_supported (kvm)) + valid_flags |=KVM_MEM_LOG_DIRTY_PAGES ;#ifdef __KVM_HAVE_READONLY_MEM valid_flags |=KVM_MEM_READONLY ;
@@ -1915,7 +1924,7 @@ int __kvm_set_memory_region(struct kvm *kvm,int as_id, id;int r; - r =check_memory_region_flags (mem); + r =check_memory_region_flags (kvm, mem);if (r)return r;
Signature · 1 line
2.25.1
PATCH v8 47/103 KVM: x86/tdp_mmu: Ignore unsupported mmu operation on private GFNs
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Some KVM MMU operations (dirty page logging, page migration, aging page) aren't supported for private GFNs (yet) with the first generation of TDX. Silently return on unsupported TDX KVM MMU operations.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/mmu/mmu.c | 3 ++ arch/x86/kvm/mmu/tdp_mmu.c | 73 +++++++++++++++++++++++++++++++++++--- arch/x86/kvm/x86.c | 3 ++ 3 files changed, 74 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -0
@@ -6566,6 +6566,9 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, sp =sptep_to_sp (sptep); pfn =spte_to_pfn (*sptep); +/* Private page dirty logging is not supported yet. */ +KVM_BUG_ON (is_private_sptep (sptep), kvm); +/* * We cannot do huge page mapping for indirect shadow pages, * which are found on the last rmap (level = 1) when not using
arch/x86/kvm/mmu/tdp_mmu.c +68 -5
136 lines not on this page · raw message
arch/x86/kvm/x86.c +3 -0
@@ -12434,6 +12434,9 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,u32 new_flags = new ? new->flags :0 ;bool log_dirty_pages = new_flags &KVM_MEM_LOG_DIRTY_PAGES ; +if (!kvm_arch_dirty_log_supported (kvm) && log_dirty_pages) +return ; +/* * Update CPU dirty logging if dirty logging is being toggled. This * applies to all operations.
Signature · 1 line
2.25.1
PATCH v8 48/103 KVM: VMX: Split out guts of EPT violation to common/exposed function
In reply to [email protected]From: Sean Christopherson <[email protected]>
The difference of TDX EPT violation is how to retrieve information, GPA, and exit qualification. To share the code to handle EPT violation, split out the guts of EPT violation handler so that VMX/TDX exit handler can call it after retrieving GPA and exit qualification.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
Reviewed-by Kai Huang <[email protected]>
--- arch/x86/kvm/vmx/common.h | 33 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 29 +++++------------------------ 2 files changed, 38 insertions(+), 24 deletions(-) create mode 100644 arch/x86/kvm/vmx/common.h
arch/x86/kvm/vmx/common.h new file +33 -0
@@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_X86_VMX_COMMON_H +#define __KVM_X86_VMX_COMMON_H + +#include <linux/kvm_host.h> + +#include "mmu.h" + +static inline int __vmx_handle_ept_violation (struct kvm_vcpu *vcpu,gpa_t gpa, +unsigned long exit_qualification) +{ +u64 error_code; + +/* Is it a read fault? */ + error_code = (exit_qualification &EPT_VIOLATION_ACC_READ ) + ?PFERR_USER_MASK :0 ; +/* Is it a write fault? */ + error_code |= (exit_qualification &EPT_VIOLATION_ACC_WRITE ) + ?PFERR_WRITE_MASK :0 ; +/* Is it a fetch fault? */ + error_code |= (exit_qualification &EPT_VIOLATION_ACC_INSTR ) + ?PFERR_FETCH_MASK :0 ; +/* ept page table entry is present? */ + error_code |= (exit_qualification &EPT_VIOLATION_RWX_MASK ) + ?PFERR_PRESENT_MASK :0 ; + + error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ? +PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; + +return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); +} + +#endif /* __KVM_X86_VMX_COMMON_H */
arch/x86/kvm/vmx/vmx.c +5 -24
@@ -50,6 +50,7 @@#include <asm/vmx.h> #include "capabilities.h" +#include "common.h" #include "cpuid.h" #include "evmcs.h" #include "hyperv.h"
@@ -5646,11 +5647,10 @@ static int handle_task_switch(struct kvm_vcpu *vcpu)static int handle_ept_violation (struct kvm_vcpu *vcpu) { -unsigned long exit_qualification; -gpa_t gpa; -u64 error_code; +unsigned long exit_qualification =vmx_get_exit_qual (vcpu); +gpa_t gpa =vmcs_read64 (GUEST_PHYSICAL_ADDRESS ); - exit_qualification =vmx_get_exit_qual (vcpu); +trace_kvm_page_fault (gpa, exit_qualification);/* * EPT violation happened while executing iret from NMI,
@@ -5663,25 +5663,6 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu) (exit_qualification & INTR_INFO_UNBLOCK_NMI))vmcs_set_bits (GUEST_INTERRUPTIBILITY_INFO ,GUEST_INTR_STATE_NMI ); - gpa =vmcs_read64 (GUEST_PHYSICAL_ADDRESS ); -trace_kvm_page_fault (gpa, exit_qualification); - -/* Is it a read fault? */ - error_code = (exit_qualification &EPT_VIOLATION_ACC_READ ) - ?PFERR_USER_MASK :0 ; -/* Is it a write fault? */ - error_code |= (exit_qualification &EPT_VIOLATION_ACC_WRITE ) - ?PFERR_WRITE_MASK :0 ; -/* Is it a fetch fault? */ - error_code |= (exit_qualification &EPT_VIOLATION_ACC_INSTR ) - ?PFERR_FETCH_MASK :0 ; -/* ept page table entry is present? */ - error_code |= (exit_qualification &EPT_VIOLATION_RWX_MASK ) - ?PFERR_PRESENT_MASK :0 ; - - error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ? -PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; - vcpu->arch.exit_qualification = exit_qualification;/*
@@ -5695,7 +5676,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)if (unlikely (allow_smaller_maxphyaddr &&kvm_vcpu_is_illegal_gpa (vcpu, gpa)))return kvm_emulate_instruction (vcpu,0 ); -return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); +return __vmx_handle_ept_violation (vcpu, gpa, exit_qualification); }static int handle_ept_misconfig (struct kvm_vcpu *vcpu)
Signature · 1 line
2.25.1
Re: [PATCH v8 048/103] KVM: VMX: Split out guts of EPT violation to common/exposed function
Quoted 11 lines · On Sun, Aug 07, 2022 at 03:01:33PM -0700, [email protected] wrote:
From: Sean Christopherson <[email protected]>
The difference of TDX EPT violation is how to retrieve information, GPA, and exit qualification. To share the code to handle EPT violation, split out the guts of EPT violation handler so that VMX/TDX exit handler can call it after retrieving GPA and exit qualification.
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Reviewed-by Paolo Bonzini <[email protected]>Reviewed-by Kai Huang <[email protected]>
Reviewed-by Yuan Yao <[email protected]>
Quoted 110 lines
--- arch/x86/kvm/vmx/common.h | 33 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 29 +++++------------------------ 2 files changed, 38 insertions(+), 24 deletions(-) create mode 100644 arch/x86/kvm/vmx/common.harch/x86/kvm/vmx/common.h new file +33
-0@@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_X86_VMX_COMMON_H +#define __KVM_X86_VMX_COMMON_H + +#include <linux/kvm_host.h> + +#include "mmu.h" + +static inline int __vmx_handle_ept_violation (struct kvm_vcpu *vcpu,gpa_t gpa, +unsigned long exit_qualification) +{ +u64 error_code; + +/* Is it a read fault? */ + error_code = (exit_qualification &EPT_VIOLATION_ACC_READ ) + ?PFERR_USER_MASK :0 ; +/* Is it a write fault? */ + error_code |= (exit_qualification &EPT_VIOLATION_ACC_WRITE ) + ?PFERR_WRITE_MASK :0 ; +/* Is it a fetch fault? */ + error_code |= (exit_qualification &EPT_VIOLATION_ACC_INSTR ) + ?PFERR_FETCH_MASK :0 ; +/* ept page table entry is present? */ + error_code |= (exit_qualification &EPT_VIOLATION_RWX_MASK ) + ?PFERR_PRESENT_MASK :0 ; + + error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ? +PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; + +return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); +} + +#endif /* __KVM_X86_VMX_COMMON_H */ arch/x86/kvm/vmx/vmx.c +5
-25@@ -50,6 +50,7 @@#include <asm/vmx.h> #include "capabilities.h" +#include "common.h" #include "cpuid.h" #include "evmcs.h" #include "hyperv.h" @@ -5646,11 +5647,10 @@ static int handle_task_switch(struct kvm_vcpu *vcpu)static int handle_ept_violation (struct kvm_vcpu *vcpu) { -unsigned long exit_qualification; -gpa_t gpa; -u64 error_code; +unsigned long exit_qualification =vmx_get_exit_qual (vcpu); +gpa_t gpa =vmcs_read64 (GUEST_PHYSICAL_ADDRESS ); - exit_qualification =vmx_get_exit_qual (vcpu); +trace_kvm_page_fault (gpa, exit_qualification);/* * EPT violation happened while executing iret from NMI, @@ -5663,25 +5663,6 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu) (exit_qualification & INTR_INFO_UNBLOCK_NMI))vmcs_set_bits (GUEST_INTERRUPTIBILITY_INFO ,GUEST_INTR_STATE_NMI ); - gpa =vmcs_read64 (GUEST_PHYSICAL_ADDRESS ); -trace_kvm_page_fault (gpa, exit_qualification); - -/* Is it a read fault? */ - error_code = (exit_qualification &EPT_VIOLATION_ACC_READ ) - ?PFERR_USER_MASK :0 ; -/* Is it a write fault? */ - error_code |= (exit_qualification &EPT_VIOLATION_ACC_WRITE ) - ?PFERR_WRITE_MASK :0 ; -/* Is it a fetch fault? */ - error_code |= (exit_qualification &EPT_VIOLATION_ACC_INSTR ) - ?PFERR_FETCH_MASK :0 ; -/* ept page table entry is present? */ - error_code |= (exit_qualification &EPT_VIOLATION_RWX_MASK ) - ?PFERR_PRESENT_MASK :0 ; - - error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ? -PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; - vcpu->arch.exit_qualification = exit_qualification;/* @@ -5695,7 +5676,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)if (unlikely (allow_smaller_maxphyaddr &&kvm_vcpu_is_illegal_gpa (vcpu, gpa)))return kvm_emulate_instruction (vcpu,0 ); -return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); +return __vmx_handle_ept_violation (vcpu, gpa, exit_qualification); }static int handle_ept_misconfig (struct kvm_vcpu *vcpu) --2.25.1
PATCH v8 49/103 KVM: VMX: Move setting of EPT MMU masks to common VT-x code
In reply to [email protected]From: Sean Christopherson <[email protected]>
EPT MMU masks are used commonly for VMX and TDX. The value needs to be initialized in common code before both VMX/TDX-specific initialization code.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/main.c | 5 +++++ arch/x86/kvm/vmx/vmx.c | 4 ---- 2 files changed, 5 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/main.c +5 -0
@@ -4,6 +4,7 @@#include "x86_ops.h" #include "vmx.h" #include "nested.h" +#include "mmu.h" #include "pmu.h" #include "tdx.h"
@@ -26,6 +27,10 @@ static __init int vt_hardware_setup(void) enable_tdx = enable_tdx && !tdx_hardware_setup (&vt_x86_ops); +if (enable_ept) +kvm_mmu_set_ept_masks (enable_ept_ad_bits, +cpu_has_vmx_ept_execute_only ()); +return 0 ; }
arch/x86/kvm/vmx/vmx.c +0 -4
@@ -8209,10 +8209,6 @@ __init int vmx_hardware_setup(void)set_bit (0 , vmx_vpid_bitmap);/* 0 is reserved for host */ -if (enable_ept) -kvm_mmu_set_ept_masks (enable_ept_ad_bits, -cpu_has_vmx_ept_execute_only ()); -/* * Setup shadow_me_value/shadow_me_mask to include MKTME KeyID * bits to shadow_zero_check.
Signature · 1 line
2.25.1
Re: [PATCH v8 049/103] KVM: VMX: Move setting of EPT MMU masks to common VT-x code
Quoted 5 lines · On Sun, Aug 07, 2022 at 03:01:34PM -0700, [email protected] wrote:
From: Sean Christopherson <[email protected]>
EPT MMU masks are used commonly for VMX and TDX. The value needs to be initialized in common code before both VMX/TDX-specific initialization code.
The "VMX/TDX-specific initialization code" is not clear enough, the patch moves the EPT MMU masks initialization AFTER vmx_hardware_setup() and tdx_hardware_setup(), and these 2 hardware_setup functions are ahead of other VMX/TDX initializtion code. I guess we can drop this patch ?
Quoted 49 lines
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/kvm/vmx/main.c | 5 +++++ arch/x86/kvm/vmx/vmx.c | 4 ---- 2 files changed, 5 insertions(+), 4 deletions(-)arch/x86/kvm/vmx/main.c +5
-0@@ -4,6 +4,7 @@#include "x86_ops.h" #include "vmx.h" #include "nested.h" +#include "mmu.h" #include "pmu.h" #include "tdx.h" @@ -26,6 +27,10 @@ static __init int vt_hardware_setup(void) enable_tdx = enable_tdx && !tdx_hardware_setup (&vt_x86_ops); +if (enable_ept) +kvm_mmu_set_ept_masks (enable_ept_ad_bits, +cpu_has_vmx_ept_execute_only ()); +return 0 ; }arch/x86/kvm/vmx/vmx.c +0
-5@@ -8209,10 +8209,6 @@ __init int vmx_hardware_setup(void)set_bit (0 , vmx_vpid_bitmap);/* 0 is reserved for host */ -if (enable_ept) -kvm_mmu_set_ept_masks (enable_ept_ad_bits, -cpu_has_vmx_ept_execute_only ()); -/* * Setup shadow_me_value/shadow_me_mask to include MKTME KeyID * bits to shadow_zero_check. -- 2.25.1
PATCH v8 50/103 KVM: TDX: Add load_mmu_pgd method for TDX
In reply to [email protected]From: Sean Christopherson <[email protected]>
For virtual IO, the guest TD shares guest pages with VMM without encryption. Shared EPT is used to map guest pages in unprotected way.
Add the VMCS field encoding for the shared EPTP, which will be used by TDX to have separate EPT walks for private GPAs (existing EPTP) versus shared GPAs (new shared EPTP).
Set shared EPT pointer value for the TDX guest to initialize TDX MMU.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/include/asm/vmx.h | 1 + arch/x86/kvm/vmx/main.c | 11 ++++++++++- arch/x86/kvm/vmx/tdx.c | 5 +++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ 4 files changed, 20 insertions(+), 1 deletion(-)
arch/x86/include/asm/vmx.h +1 -0
@@ -234,6 +234,7 @@ enum vmcs_field {TSC_MULTIPLIER_HIGH =0x00002033 ,TERTIARY_VM_EXEC_CONTROL =0x00002034 ,TERTIARY_VM_EXEC_CONTROL_HIGH =0x00002035 , +SHARED_EPT_POINTER =0x0000203C ,PID_POINTER_TABLE =0x00002042 ,PID_POINTER_TABLE_HIGH =0x00002043 , GUEST_PHYSICAL_ADDRESS = 0x00002400,
arch/x86/kvm/vmx/main.c +10 -1
@@ -110,6 +110,15 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)return vmx_vcpu_reset (vcpu, init_event); } +static void vt_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa, +int pgd_level) +{ +if (is_td_vcpu (vcpu)) +return tdx_load_mmu_pgd (vcpu, root_hpa, pgd_level); + +vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -228,7 +237,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .write_tsc_offset = vmx_write_tsc_offset, .write_tsc_multiplier = vmx_write_tsc_multiplier, - .load_mmu_pgd = vmx_load_mmu_pgd, + .load_mmu_pgd = vt_load_mmu_pgd, .check_intercept = vmx_check_intercept, .handle_exit_irqoff = vmx_handle_exit_irqoff,
arch/x86/kvm/vmx/tdx.c +5 -0
@@ -518,6 +518,11 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) vcpu->kvm->vm_bugged = true; } +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) +{ +td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK ); +} +int tdx_dev_ioctl (void __user *argp) {struct kvm_tdx_capabilities __user *user_caps;
arch/x86/kvm/vmx/x86_ops.h +4 -0
@@ -145,6 +145,8 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp); + +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return 0 ; }static inline bool tdx_is_vm_type_supported (unsigned long type) {return false; }
@@ -162,6 +164,8 @@ static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; } + +static inline void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level) {}#endif #endif /* __KVM_X86_VMX_X86_OPS_H */
Signature · 1 line
2.25.1
Re: [PATCH v8 050/103] KVM: TDX: Add load_mmu_pgd method for TDX
Quoted 9 lines · On Sun, Aug 07, 2022 at 03:01:35PM -0700, [email protected] wrote:
From: Sean Christopherson <[email protected]>
For virtual IO, the guest TD shares guest pages with VMM without encryption. Shared EPT is used to map guest pages in unprotected way.
Add the VMCS field encoding for the shared EPTP, which will be used by TDX to have separate EPT walks for private GPAs (existing EPTP) versus shared GPAs (new shared EPTP).
Quoted 5 lines
Set shared EPT pointer value for the TDX guest to initialize TDX MMU.
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Reviewed-by Paolo Bonzini <[email protected]>
Reviewed-by Yuan Yao <[email protected]>
Quoted 89 lines
--- arch/x86/include/asm/vmx.h | 1 + arch/x86/kvm/vmx/main.c | 11 ++++++++++- arch/x86/kvm/vmx/tdx.c | 5 +++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ 4 files changed, 20 insertions(+), 1 deletion(-)arch/x86/include/asm/vmx.h +1
-0@@ -234,6 +234,7 @@ enum vmcs_field {TSC_MULTIPLIER_HIGH =0x00002033 ,TERTIARY_VM_EXEC_CONTROL =0x00002034 ,TERTIARY_VM_EXEC_CONTROL_HIGH =0x00002035 , +SHARED_EPT_POINTER =0x0000203C ,PID_POINTER_TABLE =0x00002042 ,PID_POINTER_TABLE_HIGH =0x00002043 , GUEST_PHYSICAL_ADDRESS = 0x00002400,arch/x86/kvm/vmx/main.c +10
-1@@ -110,6 +110,15 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)return vmx_vcpu_reset (vcpu, init_event); } +static void vt_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa, +int pgd_level) +{ +if (is_td_vcpu (vcpu)) +return tdx_load_mmu_pgd (vcpu, root_hpa, pgd_level); + +vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))@@ -228,7 +237,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .write_tsc_offset = vmx_write_tsc_offset, .write_tsc_multiplier = vmx_write_tsc_multiplier, - .load_mmu_pgd = vmx_load_mmu_pgd, + .load_mmu_pgd = vt_load_mmu_pgd, .check_intercept = vmx_check_intercept, .handle_exit_irqoff = vmx_handle_exit_irqoff,arch/x86/kvm/vmx/tdx.c +5
-0@@ -518,6 +518,11 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) vcpu->kvm->vm_bugged = true; } +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) +{ +td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK ); +} +int tdx_dev_ioctl (void __user *argp) {struct kvm_tdx_capabilities __user *user_caps;arch/x86/kvm/vmx/x86_ops.h +4
-1@@ -145,6 +145,8 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp); + +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return 0 ; }static inline bool tdx_is_vm_type_supported (unsigned long type) {return false; }@@ -162,6 +164,8 @@ static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; } + +static inline void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level) {}#endif #endif /* __KVM_X86_VMX_X86_OPS_H */ --2.25.1
PATCH v8 51/103 KVM: TDX: don't request KVM_REQ_APIC_PAGE_RELOAD
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX doesn't need APIC page depending on vapic and its callback is WARN_ON_ONCE(is_tdx). To avoid unnecessary overhead and WARN_ON_ONCE(), skip requesting KVM_REQ_APIC_PAGE_RELOAD when TD.
WARNING: arch/x86/kvm/vmx/main.c:696 vt_set_apic_access_page_addr+0x3c/0x50 [kvm_intel] RIP: 0010:vt_set_apic_access_page_addr+0x3c/0x50 [kvm_intel] Call Trace: vcpu_enter_guest+0x145d/0x24d0 [kvm] kvm_arch_vcpu_ioctl_run+0x25d/0xcc0 [kvm] kvm_vcpu_ioctl+0x414/0xa30 [kvm] __x64_sys_ioctl+0xc0/0x100 do_syscall_64+0x39/0xc0 entry_SYSCALL_64_after_hwframe+0x44/0xae
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/x86.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/x86.c +3 -1
@@ -10224,7 +10224,9 @@ void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm,* Update it when it becomes invalid. */ apic_address =gfn_to_hva (kvm,APIC_DEFAULT_PHYS_BASE >>PAGE_SHIFT ); -if (start <= apic_address && apic_address < end) +/* TDX doesn't need APIC page. */ +if (kvm->arch.vm_type !=KVM_X86_TDX_VM && + start <= apic_address && apic_address < end)kvm_make_all_cpus_request (kvm,KVM_REQ_APIC_PAGE_RELOAD ); }
Signature · 1 line
2.25.1
PATCH v8 52/103 KVM: x86/VMX: introduce vmx tlb_remote_flush and tlb_remote_flush_with_range
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This is preparation for TDX to define its own tlb_remote_flush and tlb_remote_flush_with_range. Currently vmx code defines tlb_remote_flush and tlb_remote_flush_with_range defined as NULL by default and only when nested hyper-v guest case, they are defined to non-NULL methods.
To make TDX code to override those two methods consistently with other methods, define vmx_tlb_remote_flush and vmx_tlb_remote_flush_with_range as nop and call hyper-v code only when nested hyper-v guest case.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/kvm_onhyperv.c | 5 ++++- arch/x86/kvm/kvm_onhyperv.h | 1 + arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/svm/svm_onhyperv.h | 1 + arch/x86/kvm/vmx/main.c | 2 ++ arch/x86/kvm/vmx/vmx.c | 34 ++++++++++++++++++++++++++++----- arch/x86/kvm/vmx/x86_ops.h | 3 +++ 7 files changed, 41 insertions(+), 7 deletions(-)
arch/x86/kvm/kvm_onhyperv.c +4 -1
@@ -93,11 +93,14 @@ int hv_remote_flush_tlb(struct kvm *kvm) }EXPORT_SYMBOL_GPL (hv_remote_flush_tlb); +bool hv_use_remote_flush_tlb__ro_after_init ; +EXPORT_SYMBOL_GPL (hv_use_remote_flush_tlb); +void hv_track_root_tdp (struct kvm_vcpu *vcpu,hpa_t root_tdp) {struct kvm_arch *kvm_arch = &vcpu->kvm->arch; -if (kvm_x86_ops.tlb_remote_flush == hv_remote_flush_tlb) { +if (hv_use_remote_flush_tlb) {spin_lock (&kvm_arch->hv_root_tdp_lock); vcpu->arch.hv_root_tdp = root_tdp;if (root_tdp != kvm_arch->hv_root_tdp)
arch/x86/kvm/kvm_onhyperv.h +1 -0
@@ -10,6 +10,7 @@int hv_remote_flush_tlb_with_range (struct kvm *kvm,struct kvm_tlb_range *range);int hv_remote_flush_tlb (struct kvm *kvm); +extern bool hv_use_remote_flush_tlb__ro_after_init ;void hv_track_root_tdp (struct kvm_vcpu *vcpu,hpa_t root_tdp);#else /* !CONFIG_HYPERV */ static inline void hv_track_root_tdp (struct kvm_vcpu *vcpu,hpa_t root_tdp)
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -242,7 +242,7 @@ static void kvm_flush_remote_tlbs_with_range(struct kvm *kvm, {int ret = -ENOTSUPP ; -if (range && kvm_x86_ops.tlb_remote_flush_with_range) +if (range &&kvm_available_flush_tlb_with_range ()) ret =static_call (kvm_x86_tlb_remote_flush_with_range)(kvm, range);if (ret)
arch/x86/kvm/svm/svm_onhyperv.h +1 -0
@@ -36,6 +36,7 @@ static inline void svm_hv_hardware_setup(void) svm_x86_ops.tlb_remote_flush = hv_remote_flush_tlb; svm_x86_ops.tlb_remote_flush_with_range = hv_remote_flush_tlb_with_range; + hv_use_remote_flush_tlb = true; }if (ms_hyperv.nested_features &HV_X64_NESTED_DIRECT_FLUSH ) {
arch/x86/kvm/vmx/main.c +2 -0
@@ -187,6 +187,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .flush_tlb_all = vmx_flush_tlb_all, .flush_tlb_current = vmx_flush_tlb_current, + .tlb_remote_flush = vmx_tlb_remote_flush, + .tlb_remote_flush_with_range = vmx_tlb_remote_flush_with_range, .flush_tlb_gva = vmx_flush_tlb_gva, .flush_tlb_guest = vmx_flush_tlb_guest,
arch/x86/kvm/vmx/vmx.c +29 -5
@@ -3076,6 +3076,33 @@ void vmx_flush_tlb_current(struct kvm_vcpu *vcpu)vpid_sync_context (vmx_get_current_vpid (vcpu)); } +int vmx_tlb_remote_flush (struct kvm *kvm) +{ +#if IS_ENABLED (CONFIG_HYPERV ) +if (hv_use_remote_flush_tlb) +return hv_remote_flush_tlb (kvm); +#endif +/* +* fallback to KVM_REQ_TLB_FLUSH. +* See kvm_arch_flush_remote_tlb() and kvm_flush_remote_tlbs(). +*/ +return -EOPNOTSUPP ; +} + +int vmx_tlb_remote_flush_with_range (struct kvm *kvm, +struct kvm_tlb_range *range) +{ +#if IS_ENABLED (CONFIG_HYPERV ) +if (hv_use_remote_flush_tlb) +return hv_remote_flush_tlb_with_range (kvm, range); +#endif +/* +* fallback to tlb_remote_flush. See +* kvm_flush_remote_tlbs_with_range() +*/ +return -EOPNOTSUPP ; +} +void vmx_flush_tlb_gva (struct kvm_vcpu *vcpu,gva_t addr) {/*
@@ -8176,11 +8203,8 @@ __init int vmx_hardware_setup(void)#if IS_ENABLED (CONFIG_HYPERV )if (ms_hyperv.nested_features &HV_X64_NESTED_GUEST_MAPPING_FLUSH - && enable_ept) { - vt_x86_ops.tlb_remote_flush = hv_remote_flush_tlb; - vt_x86_ops.tlb_remote_flush_with_range = - hv_remote_flush_tlb_with_range; - } + && enable_ept) + hv_use_remote_flush_tlb = true;#endif if (!cpu_has_vmx_ple ()) {
arch/x86/kvm/vmx/x86_ops.h +3 -0
@@ -91,6 +91,9 @@ void vmx_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags);bool vmx_get_if_flag (struct kvm_vcpu *vcpu);void vmx_flush_tlb_all (struct kvm_vcpu *vcpu);void vmx_flush_tlb_current (struct kvm_vcpu *vcpu); +int vmx_tlb_remote_flush (struct kvm *kvm); +int vmx_tlb_remote_flush_with_range (struct kvm *kvm, +struct kvm_tlb_range *range);void vmx_flush_tlb_gva (struct kvm_vcpu *vcpu,gva_t addr);void vmx_flush_tlb_guest (struct kvm_vcpu *vcpu);void vmx_set_interrupt_shadow (struct kvm_vcpu *vcpu,int mask);
Signature · 1 line
2.25.1
PATCH v8 53/103 KVM: TDX: TDP MMU TDX support
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Implement hooks of TDP MMU for TDX backend. TLB flush, TLB shootdown, propagating the change private EPT entry to Secure EPT and freeing Secure EPT page. TLB flush handles both shared EPT and private EPT. It flushes shared EPT same as VMX. It also waits for the TDX TLB shootdown. For the hook to free Secure EPT page, unlinks the Secure EPT page from the Secure EPT so that the page can be freed to OS.
Propagate the entry change to Secure EPT. The possible entry changes are present -> non-present(zapping) and non-present -> present(population). On population just link the Secure EPT page or the private guest page to the Secure EPT by TDX SEAMCALL. Because TDP MMU allows concurrent zapping/population, zapping requires synchronous TLB shoot down with the frozen EPT entry. It zaps the secure entry, increments TLB counter, sends IPI to remote vcpus to trigger TLB flush, and then unlinks the private guest page from the Secure EPT. For simplicity, batched zapping with exclude lock is handled as concurrent zapping. Although it's inefficient, it can be optimized in the future.
For MMIO SPTE, the spte value changes as follows.
initial value (suppress VE bit is set)
-> Guest issues MMIO and triggers EPT violation
-> KVM updates SPTE value to MMIO value (suppress VE bit is cleared) -> Guest MMIO resumes. It triggers VE exception in guest TD -> Guest VE handler issues TDG.VP.VMCALL<MMIO>
-> KVM handles MMIO
-> Guest VE handler resumes its execution after MMIO instruction
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/mmu/spte.c | 3 +- arch/x86/kvm/vmx/main.c | 61 ++++++- arch/x86/kvm/vmx/tdx.c | 320 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 21 +++ arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/kvm/vmx/x86_ops.h | 5 +- 6 files changed, 403 insertions(+), 9 deletions(-)
arch/x86/kvm/mmu/spte.c +2 -1
@@ -58,7 +58,8 @@ u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access)u64 spte =generation_mmio_spte_mask (gen);u64 gpa = gfn <<PAGE_SHIFT ; -WARN_ON_ONCE (!vcpu->kvm->arch.shadow_mmio_value); +WARN_ON_ONCE (!vcpu->kvm->arch.shadow_mmio_value && + !kvm_gfn_shared_mask (vcpu->kvm)); access &= shadow_mmio_access_mask; spte |= vcpu->kvm->arch.shadow_mmio_value | access;
arch/x86/kvm/vmx/main.c +55 -6
73 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.c +320 -0
358 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +21 -0
@@ -26,8 +26,23 @@ struct kvm_tdx {int hkid;bool finalized; +atomic_t tdh_mem_track;u64 tsc_offset; + +/* +* Some SEAMCALLs try to lock TD resources (e.g. Secure-EPT) they use or +* update. If TDX module fails to obtain the lock, it returns +* TDX_OPERAND_BUSY error without spinning. It's VMM/OS responsibility +* to retry or guarantee no contention because TDX module has the +* restriction on cpu cycles it can spend and VMM/OS knows better +* vcpu scheduling. +* +* TDP MMU uses read lock of kvm.arch.mmu_lock so TDP MMU code can be +* run concurrently with multiple vCPUs. Lock to prevent seamcalls from +* running concurrently when TDP MMU is enabled. +*/ +spinlock_t seamcall_lock; };struct vcpu_tdx {
@@ -168,6 +183,12 @@ static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 fielreturn out.r8; } +static __always_inline int pg_level_to_tdx_sept_level(enum pg_level level) +{ +WARN_ON (level ==PG_LEVEL_NONE ); +return level -1 ; +} +#else static inline int tdx_module_setup (void ) {return -ENODEV ; };
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -145,7 +145,7 @@ module_param_named(preemption_timer, enable_preemption_timer, bool, S_IRUGO);extern bool __read_mostly allow_smaller_maxphyaddr;module_param (allow_smaller_maxphyaddr, bool,S_IRUGO ); -u64 __ro_after_init vmx_shadow_mmio_mask; +static u64 __ro_after_init vmx_shadow_mmio_mask;#define KVM_VM_CR0_ALWAYS_OFF (X86_CR0_NW | X86_CR0_CD)#define KVM_VM_CR0_ALWAYS_ON_UNRESTRICTED_GUEST X86_CR0_NE
arch/x86/kvm/vmx/x86_ops.h +4 -1
@@ -13,7 +13,6 @@ void hv_vp_assist_page_exit(void);void __init vmx_init_early (void );int __init vmx_init (void );void vmx_exit (void ); -extern u64 __ro_after_init vmx_shadow_mmio_mask;__init int vmx_cpu_has_kvm_support (void );__init int vmx_disabled_by_bios (void );
@@ -149,6 +148,8 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp); +void tdx_flush_tlb (struct kvm_vcpu *vcpu); +int tdx_sept_tlb_remote_flush (struct kvm *kvm);void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return 0 ; }
@@ -168,6 +169,8 @@ static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; } +static inline void tdx_flush_tlb (struct kvm_vcpu *vcpu) {} +static inline int tdx_sept_tlb_remote_flush (struct kvm *kvm) {return 0 ; }static inline void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level) {}#endif
Signature · 1 line
2.25.1
Sean Christopherson
maintainerRe: [PATCH v8 053/103] KVM: TDX: TDP MMU TDX support
Quoted 1 line · On Sun, Aug 07, 2022, [email protected] wrote:
+static void
tdx_unpin_pfn(struct kvm *kvm, kvm_pfn_t pfn)
Why does this helper exist? KVM should not be pinning private pages, that should be the purview of the private backing fd.
arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 552
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5
-0+{ +struct page *page =pfn_to_page (pfn); + +put_page (page); +WARN_ON (!page_count (page) &&to_kvm_tdx (kvm)->hkid >0 );
For all patches, please use the ONCE variant or KVM_BUG_ON() where appropriate. An unlimited WARN/WARN_ON() is all but guaranteed to spam the kernel log and bring the host to its knees if it fires.
arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 565
Quoted 13 lines
arch/x86/kvm/vmx/tdx.c +13
-0+} + +static void __tdx_sept_set_private_spte (struct kvm *kvm,gfn_t gfn, +enum pg_level level,kvm_pfn_t pfn) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +hpa_t hpa =pfn_to_hpa (pfn); +gpa_t gpa =gfn_to_gpa (gfn); +struct tdx_module_output out; +u64 err; + +if (WARN_ON_ONCE (is_error_noslot_pfn (pfn) || + !kvm_pfn_to_refcounted_page (pfn)))
I'm terribly confused. The cover letter says:
- fd-based private page v7 is integrated. This is mostly same to Chao's patches.
which I interpreted as meaning this series relies on UPM. But if that's the case, why is KVM manually requiring private memory to be backed by "struct page" and then manually pinning pages?
arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 573
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8
-0+return ; + +/* TODO: handle large pages. */ +if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) +return ; + +/* To prevent page migration, do nothing on mmu notifier. */ +get_page (pfn_to_page (pfn));
Again, what's going on here?
arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 581
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8
-0+ +if (likely (is_td_finalized (kvm_tdx))) { + err =tdh_mem_page_aug (kvm_tdx->tdr.pa, gpa, hpa, &out); +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_MEM_PAGE_AUG , err, &out); +put_page (pfn_to_page (pfn)); + } +return ;
This return is pointless.
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4
-0+ } +} + +static void tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,
arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 595
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+ enum pg_level level, kvm_pfn_t pfn) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); + +spin_lock (&kvm_tdx->seamcall_lock); +__tdx_sept_set_private_spte (kvm, gfn, level, pfn); +spin_unlock (&kvm_tdx->seamcall_lock); +} + +static void tdx_sept_drop_private_spte(
Please fix this style for all patches. And not just in function prototypes, but also in function calls and probably several other places as well. The _strongly_ preferred style is:
static void tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
enum pg_level level, kvm_pfn_t pfn)
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5
-0+ struct kvm *kvm, gfn_t gfn, enum pg_level level, kvm_pfn_t pfn) +{ + int tdx_level = pg_level_to_tdx_sept_level(level); +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +gpa_t gpa =gfn_to_gpa (gfn);
arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 610
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+hpa_t hpa =pfn_to_hpa (pfn); +hpa_t hpa_with_hkid; +struct tdx_module_output out; +u64 err =0 ; + +/* TODO: handle large pages. */ +if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) +return ; + +spin_lock (&kvm_tdx->seamcall_lock);
Taking a spinlock completely defeats the purpose of taking mmu_lock for read. This is definitely not an acceptable approach long term, and even for initial upstreaming I'm hesitant to punt on adding proper retry logic.
arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 620
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+if (is_hkid_assigned (kvm_tdx)) { + err =tdh_mem_page_remove (kvm_tdx->tdr.pa, gpa, tdx_level, &out); +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_MEM_PAGE_REMOVE , err, &out); + goto unlock; + } + + hpa_with_hkid =set_hkid_to_hpa (hpa, (u16 )kvm_tdx->hkid); + err =tdh_phymem_page_wbinvd (hpa_with_hkid); +if (WARN_ON_ONCE (err)) {
Why is the above a KVM_BUG_ON() but this one is not?
Quoted 17 lines
arch/x86/kvm/vmx/tdx.c +17
-0+pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err,NULL ); + goto unlock; + } + }else +/* +* The HKID assigned to this TD was already freed and cache +* was already flushed. We don't have to flush again. +*/ + err =tdx_reclaim_page ((unsigned long )__va (hpa), hpa, false,0 ); + +unlock: +spin_unlock (&kvm_tdx->seamcall_lock); + +if (!err) +tdx_unpin_pfn (kvm, pfn); +} +
...
arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 769
Quoted 2 lines
arch/x86/kvm/vmx/tdx.c +2
-0+static void tdx_handle_changed_private_spte ( +struct kvm *kvm,const struct kvm_spte_change *change)
I haven't look at this in detail, and probably won't for a few weeks, but my initial reaction is that I really, really don't like funnelling both insertion and deletion into a single helper. E.g. it should be trivial to pass the source_pa for PAGE.ADD via kvm_page_fault instead of shoving it into the struct kvm_tdx (or struct vcpu_tdx). But smushing the two things together makes that painfully difficult.
Re: [PATCH v8 053/103] KVM: TDX: TDP MMU TDX support
Quoted 5 lines · Sean Christopherson
Quoted 1 line · On Sun, Aug 07, 2022, [email protected] wrote:
+static void
tdx_unpin_pfn(struct kvm *kvm, kvm_pfn_t pfn)Why does this helper exist? KVM should not be pinning private pages, that should be the purview of the private backing fd.
Agreed. Similar comment made to v7 as well:
https://lore.kernel.org/all/[email protected]/t/#m995b4255e0b5a0759c31e4f4cf7be40cdf5ee1a4
Signature · 2 lines
Thanks, -Kai
PATCH v8 54/103 The start of TDX KVM patch series: KVM TDP MMU MapGPA
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of KVM TDP MMU MapGPA.
Signed-off-by Isaku Yamahata <[email protected]>
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -1
@@ -11,6 +11,7 @@ What qemu can do - TDX VM TYPE is exposed to Qemu. - Qemu can create/destroy guest of TDX vm type. - Qemu can create/destroy vcpu of TDX vm type. +- Qemu can populate initial guest memory image. Patch Layer status ------------------
@@ -19,7 +20,7 @@ Patch Layer status * TDX architectural definitions: Applied * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied -* TDX EPT violation: Applying +* TDX EPT violation: Applied * TD finalization: Not yet * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet
Signature · 1 line
2.25.1
PATCH v8 55/103 KVM: Add functions to track whether GFN is private or shared
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX KVM support needs to track whether GFN is private or shared. Introduce CONFIG_HAVE_KVM_PRIVATE_MEM_ATTR to add functions to track whether GFN is private or shared. Use xarray to track it for memory efficiency in normal usage.
Suggested-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/Kconfig | 1 + include/linux/kvm_host.h | 16 ++++++++++++ virt/kvm/Kconfig | 3 +++ virt/kvm/kvm_main.c | 55 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 75 insertions(+)
arch/x86/kvm/Kconfig +1 -0
@@ -48,6 +48,7 @@ config KVM select SRCU select INTERVAL_TREE select HAVE_KVM_PM_NOTIFIER if PM + select HAVE_KVM_PRIVATE_MEM_ATTR if KVM_MMU_PRIVATE help Support hosting fully virtualized guest machines using hardware virtualization extensions. You will need a fairly recent
include/linux/kvm_host.h +16 -0
@@ -786,8 +786,24 @@ struct kvm {struct notifier_block pm_notifier;#endif char stats_id[KVM_STATS_NAME_SIZE ]; +#ifdef CONFIG_HAVE_KVM_PRIVATE_MEM_ATTR +struct xarray mem_attr_array; +#endif }; +#ifdef CONFIG_HAVE_KVM_PRIVATE_MEM_ATTR +#define KVM_MEM_ATTR_SHARED 0x1 +#define KVM_MEM_ATTR_PRIVATE 0x2 + +/* memory attr on [start, end) */ +int kvm_vm_reserve_mem_attr (struct kvm *kvm,gfn_t start,gfn_t end); +int kvm_vm_set_mem_attr (struct kvm *kvm,int attr,gfn_t start,gfn_t end); +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return !xa_load (&kvm->mem_attr_array, gfn); +} +#endif +#define kvm_err (fmt, ...) \ pr_err("kvm [%i]: " fmt, task_pid_nr(current), ## __VA_ARGS__)#define kvm_info (fmt, ...) \
virt/kvm/Kconfig +3 -0
@@ -72,3 +72,6 @@ config KVM_XFER_TO_GUEST_WORK config HAVE_KVM_PM_NOTIFIER bool + +config HAVE_KVM_PRIVATE_MEM_ATTR + bool
virt/kvm/kvm_main.c +55 -0
73 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v8 56/103 KVM: x86/mmu: Let vcpu re-try when faulting page type conflict
In reply to [email protected]From: Xiaoyao Li <[email protected]>
When it gets a private page fault on a shared page, or vice verse, let vcpu retry and vcpu will keep faulting until other vcpu maps the gpa to matched page type.
Signed-off-by Xiaoyao Li <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/mmu/mmu.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/kvm/mmu/mmu.c +4 -0
@@ -4230,6 +4230,10 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return RET_PF_EMULATE ; } +if (kvm_gfn_shared_mask (vcpu->kvm) && + (kvm_mem_is_private (vcpu->kvm, fault->gfn) != fault->is_private)) +return RET_PF_RETRY ; + async = false; fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, &async, fault->write, &fault->map_writable,
Signature · 1 line
2.25.1
PATCH v8 57/103 KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by TDX
In reply to [email protected]From: Sean Christopherson <[email protected]>
Introduce a helper to directly (pun intended) fault-in a TDP page without having to go through the full page fault path. This allows TDX to get the resulting pfn and also allows the RET_PF_* enums to stay in mmu.c where they belong.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/mmu.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+)
arch/x86/kvm/mmu.h +3 -0
@@ -152,6 +152,9 @@ static inline void kvm_mmu_load_pgd(struct kvm_vcpu *vcpu) vcpu->arch.mmu->root_role.level); } +kvm_pfn_t kvm_mmu_map_tdp_page (struct kvm_vcpu *vcpu,gpa_t gpa, +u32 error_code,int max_level); +/* * Check if a given access (described through the I/D, W/R and U/S bits of a * page fault error code pfec) causes a permission fault with the given PTE
arch/x86/kvm/mmu/mmu.c +39 -0
@@ -4416,6 +4416,45 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return direct_page_fault (vcpu, fault); } +kvm_pfn_t kvm_mmu_map_tdp_page (struct kvm_vcpu *vcpu,gpa_t gpa, +u32 error_code,int max_level) +{ +int r; +struct kvm_page_fault fault = (struct kvm_page_fault ) { + .addr = gpa, + .error_code = error_code, + .exec = error_code &PFERR_FETCH_MASK , + .write = error_code &PFERR_WRITE_MASK , + .present = error_code &PFERR_PRESENT_MASK , + .rsvd = error_code &PFERR_RSVD_MASK , + .user = error_code &PFERR_USER_MASK , + .prefetch = false, + .is_tdp = true, + .nx_huge_page_workaround_enabled =is_nx_huge_page_enabled (vcpu->kvm), + .is_private =kvm_is_private_gpa (vcpu->kvm, gpa), + }; + +if (mmu_topup_memory_caches (vcpu, false)) +return KVM_PFN_ERR_FAULT ; + +/* +* Loop on the page fault path to handle the case where an mmu_notifier +* invalidation triggers RET_PF_RETRY. In the normal page fault path, +* KVM needs to resume the guest in case the invalidation changed any +* of the page fault properties, i.e. the gpa or error code. For this +* path, the gpa and error code are fixed by the caller, and the caller +* expects failure if and only if the page fault can't be fixed. +*/ +do { + fault.max_level = max_level; + fault.req_level =PG_LEVEL_4K ; + fault.goal_level =PG_LEVEL_4K ; + r =direct_page_fault (vcpu, &fault); + }while (r ==RET_PF_RETRY && !is_error_noslot_pfn (fault.pfn)); +return fault.pfn; +} +EXPORT_SYMBOL_GPL (kvm_mmu_map_tdp_page); +static void nonpaging_init_context (struct kvm_mmu *context) { context->page_fault = nonpaging_page_fault;
Signature · 1 line
2.25.1
PATCH v8 58/103 KVM: x86/tdp_mmu: implement MapGPA hypercall for TDX
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The TDX Guest-Hypervisor communication interface(GHCI) specification defines MapGPA hypercall for guest TD to request the host VMM to map given GPA range as private or shared.
It means the guest TD uses the GPA as shared (or private). The GPA won't be used as private (or shared). VMM should enforce GPA usage. VMM doesn't have to map the GPA on the hypercall request.
- Zap the aliased region.
If shared (or private) GPA is requested, zap private (or shared) GPA (modulo shared bit).
- Record the request GPA is shared (or private) by kvm.mem_attr_array.
- Don't map GPA. The GPA is mapped on the next EPT violation.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/mmu.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 50 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 40 ++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.h | 3 +++ 4 files changed, 96 insertions(+)
arch/x86/kvm/mmu.h +3 -0
@@ -217,6 +217,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end);int kvm_arch_write_log_dirty (struct kvm_vcpu *vcpu); +int kvm_mmu_map_gpa (struct kvm_vcpu *vcpu,gfn_t *startp,gfn_t end, +bool map_private); +int kvm_mmu_post_init_vm (struct kvm *kvm);void kvm_mmu_pre_destroy_vm (struct kvm *kvm);
arch/x86/kvm/mmu/mmu.c +50 -0
56 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.c +40 -0
@@ -2086,6 +2086,46 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm,return spte_set; } +int kvm_tdp_mmu_map_gpa (struct kvm_vcpu *vcpu, +gfn_t *startp,gfn_t end,bool map_private) +{ +struct kvm_mmu *mmu = vcpu->arch.mmu; +struct kvm *kvm = vcpu->kvm; +struct kvm_mmu_page *root; +gfn_t start = *startp; +bool flush = false; +int i; + +lockdep_assert_held_write (&kvm->mmu_lock); +WARN_ON (start &kvm_gfn_shared_mask (kvm)); +WARN_ON (end &kvm_gfn_shared_mask (kvm)); + +if (!VALID_PAGE (mmu->root.hpa) || !VALID_PAGE (mmu->private_root_hpa)) +return -EINVAL ; + +kvm_inc_notifier_count (kvm, start, end); +for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { +for_each_tdp_mmu_root_yield_safe (kvm, root, i) { +if (is_private_sp (root) == map_private) +continue ; + +/* +* TODO: If necessary, return to the caller with -EAGAIN +* instead of yield-and-resume within +* tdp_mmu_zap_leafs(). +*/ + flush =tdp_mmu_zap_leafs ( + kvm, root, start, end,/*can_yield=*/ false, + flush,/*zap_private=*/ is_private_sp (root)); + } + } +if (flush) +kvm_flush_remote_tlbs_with_address (kvm, start, end - start); +kvm_dec_notifier_count (kvm, start, end); + +return 0 ; +} +/* * Return the level of the lowest level SPTE added to sptes. * That SPTE may be non-present.
arch/x86/kvm/mmu/tdp_mmu.h +3 -0
@@ -51,6 +51,9 @@ void kvm_tdp_mmu_try_split_huge_pages(struct kvm *kvm, gfn_t start, gfn_t end, int target_level, bool shared); +int kvm_tdp_mmu_map_gpa (struct kvm_vcpu *vcpu, +gfn_t *startp,gfn_t end,bool map_private); +static inline void kvm_tdp_mmu_walk_lockless_begin (void ) {rcu_read_lock ();
Signature · 1 line
2.25.1
PATCH v8 59/103 The start of TDX KVM patch series: TD finalization
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TD finalization.
Signed-off-by Isaku Yamahata <[email protected]>
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -2
@@ -21,11 +21,11 @@ Patch Layer status * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied * TDX EPT violation: Applied -* TD finalization: Not yet +* TD finalization: Applying * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet * KVM MMU GPA shared bits: Applied * KVM TDP refactoring for TDX: Applied * KVM TDP MMU hooks: Applied -* KVM TDP MMU MapGPA: Not yet +* KVM TDP MMU MapGPA: Applied
Signature · 1 line
2.25.1
PATCH v8 60/103 KVM: TDX: Create initial guest memory
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because the guest memory is protected in TDX, the creation of the initial guest memory requires a dedicated TDX module API, tdh_mem_page_add, instead of directly copying the memory contents into the guest memory in the case of the default VM type. KVM MMU page fault handler callback, private_page_add, handles it.
Define new subcommand, KVM_TDX_INIT_MEM_REGION, of VM-scoped KVM_MEMORY_ENCRYPT_OP. It assigns the guest page, copies the initial memory contents into the guest memory, encrypts the guest memory. At the same time, optionally it extends memory measurement of the TDX guest. It calls the KVM MMU page fault(EPT-violation) handler to trigger the callbacks for it.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/include/uapi/asm/kvm.h | 9 ++ arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/vmx/tdx.c | 147 +++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 2 + tools/arch/x86/include/uapi/asm/kvm.h | 9 ++ 5 files changed, 163 insertions(+), 5 deletions(-)
arch/x86/include/uapi/asm/kvm.h +9 -0
@@ -540,6 +540,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_CAPABILITIES =0 ,KVM_TDX_INIT_VM ,KVM_TDX_INIT_VCPU , +KVM_TDX_INIT_MEM_REGION , KVM_TDX_CMD_NR_MAX, };
@@ -617,4 +618,12 @@ struct kvm_tdx_init_vm { }; }; +#define KVM_TDX_MEASURE_MEMORY_REGION (1UL << 0) + +struct kvm_tdx_init_mem_region { +__u64 source_addr; +__u64 gpa; +__u64 nr_pages; +}; +#endif /* _ASM_X86_KVM_H */
arch/x86/kvm/mmu/mmu.c +1 -0
@@ -5423,6 +5423,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) out:return r; } +EXPORT_SYMBOL (kvm_mmu_load);void kvm_mmu_unload (struct kvm_vcpu *vcpu) {
arch/x86/kvm/vmx/tdx.c +142 -5
187 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +2 -0
@@ -25,6 +25,8 @@ struct kvm_tdx {u64 xfam;int hkid; +hpa_t source_pa; +bool finalized;atomic_t tdh_mem_track;
tools/arch/x86/include/uapi/asm/kvm.h +9 -0
@@ -534,6 +534,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_CAPABILITIES =0 ,KVM_TDX_INIT_VM ,KVM_TDX_INIT_VCPU , +KVM_TDX_INIT_MEM_REGION , KVM_TDX_CMD_NR_MAX, };
@@ -611,4 +612,12 @@ struct kvm_tdx_init_vm { }; }; +#define KVM_TDX_MEASURE_MEMORY_REGION (1UL << 0) + +struct kvm_tdx_init_mem_region { +__u64 source_addr; +__u64 gpa; +__u64 nr_pages; +}; +#endif /* _ASM_X86_KVM_H */
Signature · 1 line
2.25.1
PATCH v8 61/103 KVM: TDX: Finalize VM initialization
In reply to [email protected]From: Isaku Yamahata <[email protected]>
To protect the initial contents of the guest TD, the TDX module measures the guest TD during the build process as SHA-384 measurement. The measurement of the guest TD contents needs to be completed to make the guest TD ready to run.
Add a new subcommand, KVM_TDX_FINALIZE_VM, for VM-scoped KVM_MEMORY_ENCRYPT_OP to finalize the measurement and mark the TDX VM ready to run.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/vmx/tdx.c | 21 +++++++++++++++++++++ tools/arch/x86/include/uapi/asm/kvm.h | 1 + 3 files changed, 23 insertions(+)
arch/x86/include/uapi/asm/kvm.h +1 -0
@@ -541,6 +541,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_INIT_VM ,KVM_TDX_INIT_VCPU ,KVM_TDX_INIT_MEM_REGION , +KVM_TDX_FINALIZE_VM , KVM_TDX_CMD_NR_MAX, };
arch/x86/kvm/vmx/tdx.c +21 -0
@@ -1221,6 +1221,24 @@ static int tdx_init_mem_region(struct kvm *kvm, struct kvm_tdx_cmd *cmd)return ret; } +static int tdx_td_finalizemr (struct kvm *kvm) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +u64 err; + +if (!is_td_initialized (kvm) ||is_td_finalized (kvm_tdx)) +return -EINVAL ; + + err =tdh_mr_finalize (kvm_tdx->tdr.pa); +if (WARN_ON_ONCE (err)) { +pr_tdx_error (TDH_MR_FINALIZE , err,NULL ); +return -EIO ; + } + + kvm_tdx->finalized = true; +return 0 ; +} +int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {struct kvm_tdx_cmd tdx_cmd;
@@ -1240,6 +1258,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)case KVM_TDX_INIT_MEM_REGION : r =tdx_init_mem_region (kvm, &tdx_cmd);break ; +case KVM_TDX_FINALIZE_VM : + r =tdx_td_finalizemr (kvm); +break ;default : r = -EINVAL ; goto out;
tools/arch/x86/include/uapi/asm/kvm.h +1 -0
@@ -535,6 +535,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_INIT_VM ,KVM_TDX_INIT_VCPU ,KVM_TDX_INIT_MEM_REGION , +KVM_TDX_FINALIZE_VM , KVM_TDX_CMD_NR_MAX, };
Signature · 1 line
2.25.1
PATCH v8 62/103 The start of TDX KVM patch series: TD vcpu enter/exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TD vcpu enter/exit.
Signed-off-by Isaku Yamahata <[email protected]>
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +3 -2
@@ -12,6 +12,7 @@ What qemu can do - Qemu can create/destroy guest of TDX vm type. - Qemu can create/destroy vcpu of TDX vm type. - Qemu can populate initial guest memory image. +- Qemu can finalize guest TD. Patch Layer status ------------------
@@ -21,8 +22,8 @@ Patch Layer status * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied * TDX EPT violation: Applied -* TD finalization: Applying -* TD vcpu enter/exit: Not yet +* TD finalization: Applied +* TD vcpu enter/exit: Applying * TD vcpu interrupts/exit/hypercall: Not yet * KVM MMU GPA shared bits: Applied
Signature · 1 line
2.25.1
PATCH v8 63/103 KVM: TDX: Add helper assembly function to TDX vcpu
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX defines an API to run TDX vcpu with its own ABI. Define an assembly helper function to run TDX vcpu to hide the special ABI so that C code can call it with function call ABI.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/vmenter.S | 146 +++++++++++++++++++++++++++++++++++++ 1 file changed, 146 insertions(+)
arch/x86/kvm/vmx/vmenter.S +146 -0
@@ -3,6 +3,7 @@ #include <asm/asm.h> #include <asm/asm-offsets.h> #include <asm/bitsperlong.h> +#include <asm/errno.h> #include <asm/kvm_vcpu_regs.h> #include <asm/nospec-branch.h> #include <asm/percpu.h>
@@ -31,6 +32,13 @@ #define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE #endif +#ifdef CONFIG_INTEL_TDX_HOST +#define TDENTER 0 +#define EXIT_REASON_TDCALL 77 +#define TDENTER_ERROR_BIT 63 +#define seamcall .byte 0x66,0x0f,0x01,0xcf +#endif + .section .noinstr.text, "ax" /**
@@ -358,3 +366,141 @@ SYM_FUNC_START(vmx_do_interrupt_nmi_irqoff) pop %_ASM_BP RET SYM_FUNC_END(vmx_do_interrupt_nmi_irqoff) + +#ifdef CONFIG_INTEL_TDX_HOST + +.pushsection .noinstr.text, "ax" + +/** + * __tdx_vcpu_run - Call SEAMCALL(TDENTER) to run a TD vcpu + * @tdvpr: physical address of TDVPR + * @regs: void * (to registers of TDVCPU) + * @gpr_mask: non-zero if guest registers need to be loaded prior to TDENTER + * + * Returns: + * TD-Exit Reason + * + * Note: KVM doesn't support using XMM in its hypercalls, it's the HyperV + * code's responsibility to save/restore XMM registers on TDVMCALL. + */ +SYM_FUNC_START(__tdx_vcpu_run) + push %rbp + mov %rsp, %rbp + + push %r15 + push %r14 + push %r13 + push %r12 + push %rbx + + /* Save @regs, which is needed after TDENTER to capture output. */ + push %rsi + + /* Load @tdvpr to RCX */ + mov %rdi, %rcx + + /* No need to load guest GPRs if the last exit wasn't a TDVMCALL. */ + test %dx, %dx + je 1f + + /* Load @regs to RAX, which will be clobbered with $TDENTER anyways. */ + mov %rsi, %rax + + mov VCPU_RBX(%rax), %rbx + mov VCPU_RDX(%rax), %rdx + mov VCPU_RBP(%rax), %rbp + mov VCPU_RSI(%rax), %rsi + mov VCPU_RDI(%rax), %rdi + + mov VCPU_R8 (%rax), %r8 + mov VCPU_R9 (%rax), %r9 + mov VCPU_R10(%rax), %r10 + mov VCPU_R11(%rax), %r11 + mov VCPU_R12(%rax), %r12 + mov VCPU_R13(%rax), %r13 + mov VCPU_R14(%rax), %r14 + mov VCPU_R15(%rax), %r15 + + /* Load TDENTER to RAX. This kills the @regs pointer! */ +1: mov $TDENTER, %rax + +2: seamcall + + /* Skip to the exit path if TDENTER failed. */ + bt $TDENTER_ERROR_BIT, %rax + jc 4f + + /* Temporarily save the TD-Exit reason. */ + push %rax + + /* check if TD-exit due to TDVMCALL */ + cmp $EXIT_REASON_TDCALL, %ax + + /* Reload @regs to RAX. */ + mov 8(%rsp), %rax + + /* Jump on non-TDVMCALL */ + jne 3f + + /* Save all output from SEAMCALL(TDENTER) */ + mov %rbx, VCPU_RBX(%rax) + mov %rbp, VCPU_RBP(%rax) + mov %rsi, VCPU_RSI(%rax) + mov %rdi, VCPU_RDI(%rax) + mov %r10, VCPU_R10(%rax) + mov %r11, VCPU_R11(%rax) + mov %r12, VCPU_R12(%rax) + mov %r13, VCPU_R13(%rax) + mov %r14, VCPU_R14(%rax) + mov %r15, VCPU_R15(%rax) + +3: mov %rcx, VCPU_RCX(%rax) + mov %rdx, VCPU_RDX(%rax) + mov %r8, VCPU_R8 (%rax) + mov %r9, VCPU_R9 (%rax) + + /* + * Clear all general purpose registers except RSP and RAX to prevent + * speculative use of the guest's values. + */ + xor %rbx, %rbx + xor %rcx, %rcx + xor %rdx, %rdx + xor %rsi, %rsi + xor %rdi, %rdi + xor %rbp, %rbp + xor %r8, %r8 + xor %r9, %r9 + xor %r10, %r10 + xor %r11, %r11 + xor %r12, %r12 + xor %r13, %r13 + xor %r14, %r14 + xor %r15, %r15 + + /* Restore the TD-Exit reason to RAX for return. */ + pop %rax + + /* "POP" @regs. */ +4: add $8, %rsp + pop %rbx + pop %r12 + pop %r13 + pop %r14 + pop %r15 + + pop %rbp + RET + +5: cmpb $0, kvm_rebooting + je 6f + mov $-EFAULT, %rax + jmp 4b +6: ud2 + _ASM_EXTABLE(2b, 5b) + +SYM_FUNC_END(__tdx_vcpu_run) + +.popsection + +#endif
Signature · 1 line
2.25.1
PATCH v8 64/103 KVM: TDX: Implement TDX vcpu enter/exit path
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This patch implements running TDX vcpu. Once vcpu runs on the logical processor (LP), the TDX vcpu is associated with it. When the TDX vcpu moves to another LP, the TDX vcpu needs to flush its status on the LP. When destroying TDX vcpu, it needs to complete flush and flush cpu memory cache. Track which LP the TDX vcpu run and flush it as necessary.
Do nothing on sched_in event as TDX doesn't support pause loop.
TDX vcpu execution requires restoring PMU debug store after returning back to KVM because the TDX module unconditionally resets the value. To reuse the existing code, export perf_restore_debug_store.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/main.c | 21 +++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 32 ++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 33 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ arch/x86/kvm/x86.c | 1 + 5 files changed, 87 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +19 -2
@@ -110,6 +110,23 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)return vmx_vcpu_reset (vcpu, init_event); } +static int vt_vcpu_pre_run (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +/* Unconditionally continue to vcpu_run(). */ +return 1 ; + +return vmx_vcpu_pre_run (vcpu); +} + +static fastpath_t vt_vcpu_run (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_vcpu_run (vcpu); + +return vmx_vcpu_run (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -241,8 +258,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .flush_tlb_gva = vt_flush_tlb_gva, .flush_tlb_guest = vt_flush_tlb_guest, - .vcpu_pre_run = vmx_vcpu_pre_run, - .vcpu_run = vmx_vcpu_run, + .vcpu_pre_run = vt_vcpu_pre_run, + .vcpu_run = vt_vcpu_run, .handle_exit = vmx_handle_exit, .skip_emulated_instruction = vmx_skip_emulated_instruction, .update_emulated_instruction = vmx_update_emulated_instruction,
arch/x86/kvm/vmx/tdx.c +32 -0
@@ -11,6 +11,9 @@#include "x86.h" #include "mmu.h" +#include <trace/events/kvm.h> +#include "trace.h" +#undef pr_fmt#define pr_fmt (fmt) "tdx: " fmt
@@ -539,6 +542,35 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) vcpu->kvm->vm_bugged = true; } +u64 __tdx_vcpu_run (hpa_t tdvpr,void *regs,u32 regs_mask); + +static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu, +struct vcpu_tdx *tdx) +{ +guest_enter_irqoff (); + tdx->exit_reason.full =__tdx_vcpu_run (tdx->tdvpr.pa, vcpu->arch.regs,0 ); +guest_exit_irqoff (); +} + +fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); + +if (unlikely (vcpu->kvm->vm_bugged)) { + tdx->exit_reason.full =TDX_NON_RECOVERABLE_VCPU ; +return EXIT_FASTPATH_NONE ; + } + +trace_kvm_entry (vcpu); + +tdx_vcpu_enter_exit (vcpu, tdx); + + vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ; +trace_kvm_exit (vcpu,KVM_ISA_VMX ); + +return EXIT_FASTPATH_NONE ; +} +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) {td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK );
arch/x86/kvm/vmx/tdx.h +33 -0
@@ -47,12 +47,45 @@ struct kvm_tdx {spinlock_t seamcall_lock; }; +union tdx_exit_reason { +struct { +/* 31:0 mirror the VMX Exit Reason format */ +u64 basic :16 ; +u64 reserved16 :1 ; +u64 reserved17 :1 ; +u64 reserved18 :1 ; +u64 reserved19 :1 ; +u64 reserved20 :1 ; +u64 reserved21 :1 ; +u64 reserved22 :1 ; +u64 reserved23 :1 ; +u64 reserved24 :1 ; +u64 reserved25 :1 ; +u64 bus_lock_detected :1 ; +u64 enclave_mode :1 ; +u64 smi_pending_mtf :1 ; +u64 smi_from_vmx_root :1 ; +u64 reserved30 :1 ; +u64 failed_vmentry :1 ; + +/* 63:32 are TDX specific */ +u64 details_l1 :8 ; +u64 class :8 ; +u64 reserved61_48 :14 ; +u64 non_recoverable :1 ; +u64 error :1 ; + }; +u64 full; +}; +struct vcpu_tdx {struct kvm_vcpu vcpu;struct tdx_td_page tdvpr;struct tdx_td_page *tdvpx; +union tdx_exit_reason exit_reason; +bool vcpu_initialized;/*
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -144,6 +144,7 @@ void tdx_vm_free(struct kvm *kvm);int tdx_vcpu_create (struct kvm_vcpu *vcpu);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event); +fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -165,6 +166,7 @@ static inline void tdx_vm_free(struct kvm *kvm) {}static inline int tdx_vcpu_create (struct kvm_vcpu *vcpu) {return -EOPNOTSUPP ; }static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {} +static inline fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) {return EXIT_FASTPATH_NONE ; }static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
arch/x86/kvm/x86.c +1 -0
@@ -300,6 +300,7 @@ const struct kvm_stats_header kvm_vcpu_stats_header = { };u64 __read_mostly host_xcr0; +EXPORT_SYMBOL_GPL (host_xcr0);static struct kmem_cache *x86_emulator_cache;
Signature · 1 line
2.25.1
PATCH v8 65/103 KVM: TDX: vcpu_run: save/restore host state(host kernel gs)
In reply to [email protected]From: Isaku Yamahata <[email protected]>
On entering/exiting TDX vcpu, Preserved or clobbered CPU state is different from VMX case. Add TDX hooks to save/restore host/guest CPU state. Save/restore kernel GS base MSR.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/main.c | 28 +++++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 39 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 4 ++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ 4 files changed, 73 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +26 -2
@@ -110,6 +110,30 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)return vmx_vcpu_reset (vcpu, init_event); } +static void vt_prepare_switch_to_guest (struct kvm_vcpu *vcpu) +{ +/* +* All host state is saved/restored across SEAMCALL/SEAMRET, and the +* guest state of a TD is obviously off limits. Deferring MSRs and DRs +* is pointless because the TDX module needs to load *something* so as +* not to expose guest state. +*/ +if (is_td_vcpu (vcpu)) { +tdx_prepare_switch_to_guest (vcpu); +return ; + } + +vmx_prepare_switch_to_guest (vcpu); +} + +static void vt_vcpu_put (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_vcpu_put (vcpu); + +return vmx_vcpu_put (vcpu); +} +static int vt_vcpu_pre_run (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -223,9 +247,9 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_free = vt_vcpu_free, .vcpu_reset = vt_vcpu_reset, - .prepare_switch_to_guest = vmx_prepare_switch_to_guest, + .prepare_switch_to_guest = vt_prepare_switch_to_guest, .vcpu_load = vmx_vcpu_load, - .vcpu_put = vmx_vcpu_put, + .vcpu_put = vt_vcpu_put, .update_exception_bitmap = vmx_update_exception_bitmap, .get_msr_feature = vmx_get_msr_feature,
arch/x86/kvm/vmx/tdx.c +39 -0
@@ -1,5 +1,6 @@// SPDX-License-Identifier: GPL-2.0 #include <linux/cpu.h> +#include <linux/mmu_context.h> #include <asm/tdx.h>
@@ -450,6 +451,9 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.guest_state_protected = !(to_kvm_tdx (vcpu->kvm)->attributes &TDX_TD_ATTRIBUTE_DEBUG ); + tdx->host_state_need_save = true; + tdx->host_state_need_restore = false; +return 0 ; free_tdvpx:
@@ -463,6 +467,39 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu)return ret; } +void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); + +if (!tdx->host_state_need_save) +return ; + +if (likely (is_64bit_mm (current->mm))) + tdx->msr_host_kernel_gs_base = current->thread.gsbase; +else + tdx->msr_host_kernel_gs_base =read_msr (MSR_KERNEL_GS_BASE ); + + tdx->host_state_need_save = false; +} + +static void tdx_prepare_switch_to_host (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); + + tdx->host_state_need_save = true; +if (!tdx->host_state_need_restore) +return ; + +wrmsrl (MSR_KERNEL_GS_BASE , tdx->msr_host_kernel_gs_base); + tdx->host_state_need_restore = false; +} + +void tdx_vcpu_put (struct kvm_vcpu *vcpu) +{ +vmx_vcpu_pi_put (vcpu); +tdx_prepare_switch_to_host (vcpu); +} +void tdx_vcpu_free (struct kvm_vcpu *vcpu) {struct vcpu_tdx *tdx =to_tdx (vcpu);
@@ -565,6 +602,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (vcpu, tdx); + tdx->host_state_need_restore = true; + vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ;trace_kvm_exit (vcpu,KVM_ISA_VMX );
arch/x86/kvm/vmx/tdx.h +4 -0
@@ -88,6 +88,10 @@ struct vcpu_tdx {bool vcpu_initialized; +bool host_state_need_save; +bool host_state_need_restore; +u64 msr_host_kernel_gs_base; +/* * Dummy to make pmu_intel not corrupt memory. * TODO: Support PMU for TDX. Future work.
arch/x86/kvm/vmx/x86_ops.h +4 -0
@@ -145,6 +145,8 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event);fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu); +void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu); +void tdx_vcpu_put (struct kvm_vcpu *vcpu);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -167,6 +169,8 @@ static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTSUPP; }static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {}static inline fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) {return EXIT_FASTPATH_NONE ; } +static inline void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {} +static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {}static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v8 66/103 KVM: TDX: restore host xsave state when exit from the guest TD
In reply to [email protected]From: Isaku Yamahata <[email protected]>
On exiting from the guest TD, xsave state is clobbered. Restore xsave state on TD exit.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
arch/x86/kvm/vmx/tdx.c +18 -0
@@ -2,6 +2,7 @@#include <linux/cpu.h> #include <linux/mmu_context.h> +#include <asm/fpu/xcr.h> #include <asm/tdx.h> #include "capabilities.h"
@@ -579,6 +580,22 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) vcpu->kvm->vm_bugged = true; } +static void tdx_restore_host_xsave_state (struct kvm_vcpu *vcpu) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (vcpu->kvm); + +if (static_cpu_has (X86_FEATURE_XSAVE ) && + host_xcr0 != (kvm_tdx->xfam & kvm_caps.supported_xcr0)) +xsetbv (XCR_XFEATURE_ENABLED_MASK , host_xcr0); +if (static_cpu_has (X86_FEATURE_XSAVES ) && +/* PT can be exposed to TD guest regardless of KVM's XSS support */ + host_xss != (kvm_tdx->xfam & (kvm_caps.supported_xss |XFEATURE_MASK_PT ))) +wrmsrl (MSR_IA32_XSS , host_xss); +if (static_cpu_has (X86_FEATURE_PKU ) && + (kvm_tdx->xfam &XFEATURE_MASK_PKRU )) +write_pkru (vcpu->arch.host_pkru); +} +u64 __tdx_vcpu_run (hpa_t tdvpr,void *regs,u32 regs_mask); static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu,
@@ -602,6 +619,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (vcpu, tdx); +tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true; vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ;
Signature · 1 line
2.25.1
PATCH v8 67/103 KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o wrmsr
In reply to [email protected]From: Chao Gao <[email protected]>
Several MSRs are constant and only used in userspace(ring 3). But VMs may have different values. KVM uses kvm_set_user_return_msr() to switch to guest's values and leverages user return notifier to restore them when the kernel is to return to userspace. To eliminate unnecessary wrmsr, KVM also caches the value it wrote to an MSR last time.
TDX module unconditionally resets some of these MSRs to architectural INIT state on TD exit. It makes the cached values in kvm_user_return_msrs are inconsistent with values in hardware. This inconsistency needs to be fixed. Otherwise, it may mislead kvm_on_user_return() to skip restoring some MSRs to the host's values. kvm_set_user_return_msr() can help correct this case, but it is not optimal as it always does a wrmsr. So, introduce a variation of kvm_set_user_return_msr() to update cached values and skip that wrmsr.
Signed-off-by Chao Gao <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -2072,6 +2072,7 @@ int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low,int kvm_add_user_return_msr (u32 msr);int kvm_find_user_return_msr (u32 msr);int kvm_set_user_return_msr (unsigned index,u64 val,u64 mask); +void kvm_user_return_update_cache (unsigned int index,u64 val);static inline bool kvm_is_supported_user_return_msr (u32 msr) {
arch/x86/kvm/x86.c +20 -5
@@ -425,6 +425,15 @@ static void kvm_user_return_msr_cpu_online(void) } } +static void kvm_user_return_register_notifier (struct kvm_user_return_msrs *msrs) +{ +if (!msrs->registered) { + msrs->urn.on_user_return = kvm_on_user_return; +user_return_notifier_register (&msrs->urn); + msrs->registered = true; + } +} +int kvm_set_user_return_msr (unsigned slot,u64 value,u64 mask) {unsigned int cpu =smp_processor_id ();
@@ -439,15 +448,21 @@ int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask)return 1 ; msrs->values[slot].curr = value; -if (!msrs->registered) { - msrs->urn.on_user_return = kvm_on_user_return; -user_return_notifier_register (&msrs->urn); - msrs->registered = true; - } +kvm_user_return_register_notifier (msrs);return 0 ; }EXPORT_SYMBOL_GPL (kvm_set_user_return_msr); +/* Update the cache, "curr", and register the notifier */ +void kvm_user_return_update_cache (unsigned int slot,u64 value) +{ +struct kvm_user_return_msrs *msrs =this_cpu_ptr (user_return_msrs); + + msrs->values[slot].curr = value; +kvm_user_return_register_notifier (msrs); +} +EXPORT_SYMBOL_GPL (kvm_user_return_update_cache); +static void drop_user_return_notifiers (void ) {unsigned int cpu =smp_processor_id ();
Signature · 1 line
2.25.1
PATCH v8 68/103 KVM: TDX: restore user ret MSRs
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Several user ret MSRs are clobbered on TD exit. Restore those values on TD exit and before returning to ring 3.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 43 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
arch/x86/kvm/vmx/tdx.c +43 -0
@@ -580,6 +580,28 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) vcpu->kvm->vm_bugged = true; } +struct tdx_uret_msr { +u32 msr; +unsigned int slot; +u64 defval; +}; + +static struct tdx_uret_msr tdx_uret_msrs[] = { + {.msr =MSR_SYSCALL_MASK ,}, + {.msr =MSR_STAR ,}, + {.msr =MSR_LSTAR ,}, + {.msr =MSR_TSC_AUX ,}, +}; + +static void tdx_user_return_update_cache (void ) +{ +int i; + +for (i =0 ; i <ARRAY_SIZE (tdx_uret_msrs); i++) +kvm_user_return_update_cache (tdx_uret_msrs[i].slot, + tdx_uret_msrs[i].defval); +} +static void tdx_restore_host_xsave_state (struct kvm_vcpu *vcpu) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (vcpu->kvm);
@@ -619,6 +641,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (vcpu, tdx); +tdx_user_return_update_cache ();tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true;
@@ -1454,6 +1477,26 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)return -ENODEV ; } +for (i =0 ; i <ARRAY_SIZE (tdx_uret_msrs); i++) { +/* +* Here it checks if MSRs (tdx_uret_msrs) can be saved/restored +* before returning to user space. +* +* this_cpu_ptr(user_return_msrs)->registerd isn't checked +* because the registration is done at vcpu runtime by +* kvm_set_user_return_msr(). +* Here is setting up cpu feature before running vcpu, +* registered is alreays false. +*/ + tdx_uret_msrs[i].slot =kvm_find_user_return_msr (tdx_uret_msrs[i].msr); +if (tdx_uret_msrs[i].slot ==-1 ) { +/* If any MSR isn't supported, it is a KVM bug */ +pr_err ("MSR %x isn't included by kvm_find_user_return_msr\n" , + tdx_uret_msrs[i].msr); +return -EIO ; + } + } + max_pkgs =topology_max_packages (); tdx_mng_key_config_lock =kcalloc (max_pkgs,sizeof (*tdx_mng_key_config_lock),GFP_KERNEL );
Signature · 1 line
2.25.1
PATCH v8 69/103 The start of TDX KVM patch series: TD vcpu exits/interrupts/hypercalls
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TD vcpu exits, interrupts, and hypercalls.
Signed-off-by Isaku Yamahata <[email protected]>
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -1
@@ -13,6 +13,7 @@ What qemu can do - Qemu can create/destroy vcpu of TDX vm type. - Qemu can populate initial guest memory image. - Qemu can finalize guest TD. +- Qemu can start to run vcpu. But vcpu can not make progress yet. Patch Layer status ------------------
@@ -23,7 +24,7 @@ Patch Layer status * TD vcpu creation/destruction: Applied * TDX EPT violation: Applied * TD finalization: Applied -* TD vcpu enter/exit: Applying +* TD vcpu enter/exit: Applied * TD vcpu interrupts/exit/hypercall: Not yet * KVM MMU GPA shared bits: Applied
Signature · 1 line
2.25.1
PATCH v8 70/103 KVM: TDX: complete interrupts after tdexit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This corresponds to VMX __vmx_complete_interrupts(). Because TDX virtualize vAPIC, KVM only needs to care NMI injection.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/kvm/vmx/tdx.c +10 -0
@@ -580,6 +580,14 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) vcpu->kvm->vm_bugged = true; } +static void tdx_complete_interrupts (struct kvm_vcpu *vcpu) +{ +/* Avoid costly SEAMCALL if no nmi was injected */ +if (vcpu->arch.nmi_injected) + vcpu->arch.nmi_injected =td_management_read8 (to_tdx (vcpu), +TD_VCPU_PEND_NMI ); +} +struct tdx_uret_msr {u32 msr;unsigned int slot;
@@ -648,6 +656,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ;trace_kvm_exit (vcpu,KVM_ISA_VMX ); +tdx_complete_interrupts (vcpu); +return EXIT_FASTPATH_NONE ; }
Signature · 1 line
2.25.1
PATCH v8 71/103 KVM: TDX: restore debug store when TD exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because debug store is clobbered, restore it on TD exit.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/events/intel/ds.c | 1 + arch/x86/kvm/vmx/tdx.c | 1 + 2 files changed, 2 insertions(+)
arch/x86/events/intel/ds.c +1 -0
@@ -2256,3 +2256,4 @@ void perf_restore_debug_store(void)wrmsrl (MSR_IA32_DS_AREA , (unsigned long )ds); } +EXPORT_SYMBOL_GPL (perf_restore_debug_store);
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -650,6 +650,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (vcpu, tdx);tdx_user_return_update_cache (); +perf_restore_debug_store ();tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true;
Signature · 1 line
2.25.1
PATCH v8 72/103 KVM: TDX: handle vcpu migration over logical processor
In reply to [email protected]From: Isaku Yamahata <[email protected]>
For vcpu migration, in the case of VMX, VCMS is flushed on the source pcpu, and load it on the target pcpu. There are corresponding TDX SEAMCALL APIs, call them on vcpu migration. The logic is mostly same as VMX except the TDX SEAMCALLs are used.
When shutting down the machine, (VMX or TDX) vcpus needs to be shutdown on each pcpu. Do the similar for TDX with TDX SEAMCALL APIs.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/main.c | 43 +++++++++++-- arch/x86/kvm/vmx/tdx.c | 121 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 2 + arch/x86/kvm/vmx/x86_ops.h | 6 ++ 4 files changed, 168 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/main.c +39 -4
@@ -17,6 +17,25 @@ static bool vt_is_vm_type_supported(unsigned long type) (enable_tdx &&tdx_is_vm_type_supported (type)); } +static int vt_hardware_enable (void ) +{ +int ret; + + ret =vmx_hardware_enable (); +if (ret) +return ret; + +tdx_hardware_enable (); +return 0 ; +} + +static void vt_hardware_disable (void ) +{ +/* Note, TDX *and* VMX need to be disabled if TDX is enabled. */ +tdx_hardware_disable (); +vmx_hardware_disable (); +} +static __init int vt_hardware_setup (void ) {int ret;
@@ -151,6 +170,14 @@ static fastpath_t vt_vcpu_run(struct kvm_vcpu *vcpu)return vmx_vcpu_run (vcpu); } +static void vt_vcpu_load (struct kvm_vcpu *vcpu,int cpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_vcpu_load (vcpu, cpu); + +return vmx_vcpu_load (vcpu, cpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -209,6 +236,14 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); } +static void vt_sched_in (struct kvm_vcpu *vcpu,int cpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_sched_in (vcpu, cpu); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -231,8 +266,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .hardware_unsetup = vt_hardware_unsetup, .check_processor_compatibility = vmx_check_processor_compatibility, - .hardware_enable = vmx_hardware_enable, - .hardware_disable = vmx_hardware_disable, + .hardware_enable = vt_hardware_enable, + .hardware_disable = vt_hardware_disable, .has_emulated_msr = vmx_has_emulated_msr, .is_vm_type_supported = vt_is_vm_type_supported,
@@ -248,7 +283,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_reset = vt_vcpu_reset, .prepare_switch_to_guest = vt_prepare_switch_to_guest, - .vcpu_load = vmx_vcpu_load, + .vcpu_load = vt_vcpu_load, .vcpu_put = vt_vcpu_put, .update_exception_bitmap = vmx_update_exception_bitmap,
@@ -336,7 +371,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .request_immediate_exit = vmx_request_immediate_exit, - .sched_in = vmx_sched_in, + .sched_in = vt_sched_in, .cpu_dirty_log_size = PML_ENTITY_NUM, .update_cpu_dirty_logging = vmx_update_cpu_dirty_logging,
arch/x86/kvm/vmx/tdx.c +121 -0
166 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +2 -0
@@ -84,6 +84,8 @@ struct vcpu_tdx {struct tdx_td_page tdvpr;struct tdx_td_page *tdvpx; +struct list_head cpu_list; +union tdx_exit_reason exit_reason;bool vcpu_initialized;
arch/x86/kvm/vmx/x86_ops.h +6 -0
@@ -135,6 +135,8 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu);int __init tdx_hardware_setup (struct kvm_x86_ops *x86_ops);bool tdx_is_vm_type_supported (unsigned long type);void tdx_hardware_unsetup (void ); +void tdx_hardware_enable (void ); +void tdx_hardware_disable (void );int tdx_dev_ioctl (void __user *argp);int tdx_vm_init (struct kvm *kvm);
@@ -147,6 +149,7 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu);void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu);void tdx_vcpu_put (struct kvm_vcpu *vcpu); +void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -158,6 +161,8 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level);static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return 0 ; }static inline bool tdx_is_vm_type_supported (unsigned long type) {return false; }static inline void tdx_hardware_unsetup (void ) {} +static inline void tdx_hardware_enable (void ) {} +static inline void tdx_hardware_disable (void ) {}static inline int tdx_dev_ioctl (void __user *argp) {return -EOPNOTSUPP ; };static inline int tdx_vm_init (struct kvm *kvm) {return -EOPNOTSUPP ; }
@@ -171,6 +176,7 @@ static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}static inline fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) {return EXIT_FASTPATH_NONE ; }static inline void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {} +static inline void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {}static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v8 73/103 KVM: x86: Add a switch_db_regs flag to handle TDX's auto-switched behavior
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add a flag, KVM_DEBUGREG_AUTO_SWITCHED_GUEST, to skip saving/restoring DRs irrespective of any other flags. TDX-SEAM unconditionally saves and restores guest DRs and reset to architectural INIT state on TD exit. So, KVM needs to save host DRs before TD enter without restoring guest DRs and restore host DRs after TD exit.
Opportunistically convert the KVM_DEBUGREG_* definitions to use BIT().
Reported-by Xiaoyao Li <[email protected]>
Signed-off-by Sean Christopherson <[email protected]>
Co-developed-by Chao Gao <[email protected]>
Signed-off-by Chao Gao <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/include/asm/kvm_host.h | 9 +++++++-- arch/x86/kvm/vmx/tdx.c | 1 + arch/x86/kvm/x86.c | 11 ++++++++--- 3 files changed, 16 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +7 -2
@@ -582,8 +582,13 @@ struct kvm_pmu {struct kvm_pmu_ops ;enum { -KVM_DEBUGREG_BP_ENABLED =1 , -KVM_DEBUGREG_WONT_EXIT =2 , +KVM_DEBUGREG_BP_ENABLED =BIT (0 ), +KVM_DEBUGREG_WONT_EXIT =BIT (1 ), +/* +* Guest debug registers are saved/restored by hardware on exit from +* or enter guest. KVM needn't switch them. +*/ +KVM_DEBUGREG_AUTO_SWITCH =BIT (2 ), };struct kvm_mtrr_range {
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -532,6 +532,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.efer =EFER_SCE |EFER_LME |EFER_LMA |EFER_NX ; + vcpu->arch.switch_db_regs =KVM_DEBUGREG_AUTO_SWITCH ; vcpu->arch.cr0_guest_owned_bits =-1ul ; vcpu->arch.cr4_guest_owned_bits =-1ul ;
arch/x86/kvm/x86.c +8 -3
@@ -10509,7 +10509,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)if (vcpu->arch.guest_fpu.xfd_err)wrmsrl (MSR_IA32_XFD_ERR , vcpu->arch.guest_fpu.xfd_err); -if (unlikely (vcpu->arch.switch_db_regs)) { +if (unlikely (vcpu->arch.switch_db_regs & ~KVM_DEBUGREG_AUTO_SWITCH )) {set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg (vcpu->arch.eff_db[1 ],1 );
@@ -10552,6 +10552,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)*/ if (unlikely (vcpu->arch.switch_db_regs &KVM_DEBUGREG_WONT_EXIT )) {WARN_ON (vcpu->guest_debug &KVM_GUESTDBG_USE_HW_BP ); +WARN_ON (vcpu->arch.switch_db_regs &KVM_DEBUGREG_AUTO_SWITCH );static_call (kvm_x86_sync_dirty_debug_regs)(vcpu);kvm_update_dr0123 (vcpu);kvm_update_dr7 (vcpu);
@@ -10564,8 +10565,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)* care about the messed up debug address registers. But if * we have some of them active, restore the old state. */ -if (hw_breakpoint_active ()) -hw_breakpoint_restore (); +if (hw_breakpoint_active ()) { +if (!(vcpu->arch.switch_db_regs &KVM_DEBUGREG_AUTO_SWITCH )) +hw_breakpoint_restore (); +else +set_debugreg (__this_cpu_read (cpu_dr7),7 ); + } vcpu->arch.last_vmentry_cpu = vcpu->cpu; vcpu->arch.last_guest_tsc =kvm_read_l1_tsc (vcpu,rdtsc ());
Signature · 1 line
2.25.1
PATCH v8 74/103 KVM: TDX: Add support for find pending IRQ in a protected local APIC
In reply to [email protected]From: Sean Christopherson <[email protected]>
Add flag and hook to KVM's local APIC management to support determining whether or not a TDX guest as a pending IRQ. For TDX vCPUs, the virtual APIC page is owned by the TDX module and cannot be accessed by KVM. As a result, registers that are virtualized by the CPU, e.g. PPR, cannot be read or written by KVM. To deliver interrupts for TDX guests, KVM must send an IRQ to the CPU on the posted interrupt notification vector. And to determine if TDX vCPU has a pending interrupt, KVM must check if there is an outstanding notification.
Return "no interrupt" in kvm_apic_has_interrupt() if the guest APIC is protected to short-circuit the various other flows that try to pull an IRQ out of the vAPIC, the only valid operation is querying _if_ an IRQ is pending, KVM can't do anything based on _which_ IRQ is pending.
Intentionally omit sanity checks from other flows, e.g. PPR update, so as not to degrade non-TDX guests with unecessary checks. A well-behaved KVM and userspace will never reach those flows for TDX guests, but reaching them is not fatal if something does go awry.
Note, this doesn't handle interrupts that have been delivered to the vCPU but not yet recognized by the core, i.e. interrupts that are sitting in vmcs.GUEST_INTR_STATUS. Querying that state requires a SEAMCALL and will be supported in a future patch.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/irq.c | 3 +++ arch/x86/kvm/lapic.c | 3 +++ arch/x86/kvm/lapic.h | 2 ++ arch/x86/kvm/vmx/main.c | 11 +++++++++++ arch/x86/kvm/vmx/tdx.c | 6 ++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ 8 files changed, 29 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -112,6 +112,7 @@ KVM_X86_OP_OPTIONAL(pi_update_irte) KVM_X86_OP_OPTIONAL(pi_start_assignment) KVM_X86_OP_OPTIONAL(apicv_post_state_restore) KVM_X86_OP_OPTIONAL_RET0(dy_apicv_has_pending_interrupt) +KVM_X86_OP_OPTIONAL(protected_apic_has_interrupt) KVM_X86_OP_OPTIONAL(set_hv_timer) KVM_X86_OP_OPTIONAL(cancel_hv_timer) KVM_X86_OP(setup_mce)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1661,6 +1661,7 @@ struct kvm_x86_ops {void (*pi_start_assignment)(struct kvm *kvm);void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);bool (*dy_apicv_has_pending_interrupt)(struct kvm_vcpu *vcpu); +bool (*protected_apic_has_interrupt)(struct kvm_vcpu *vcpu);int (*set_hv_timer)(struct kvm_vcpu *vcpu,u64 guest_deadline_tsc,bool *expired);
arch/x86/kvm/irq.c +3 -0
@@ -100,6 +100,9 @@ int kvm_cpu_has_interrupt(struct kvm_vcpu *v)if (kvm_cpu_has_extint (v))return 1 ; +if (lapic_in_kernel (v) && v->arch.apic->guest_apic_protected) +return static_call (kvm_x86_protected_apic_has_interrupt)(v); +return kvm_apic_has_interrupt (v) !=-1 ;/* LAPIC */ }EXPORT_SYMBOL_GPL (kvm_cpu_has_interrupt);
arch/x86/kvm/lapic.c +3 -0
@@ -2622,6 +2622,9 @@ int kvm_apic_has_interrupt(struct kvm_vcpu *vcpu)if (!kvm_apic_present (vcpu))return -1 ; +if (apic->guest_apic_protected) +return -1 ; +__apic_update_ppr (apic, &ppr);return apic_has_interrupt_for_ppr (apic, ppr); }
arch/x86/kvm/lapic.h +2 -0
@@ -65,6 +65,8 @@ struct kvm_lapic {bool sw_enabled;bool irr_pending;bool lvt0_in_nmi_mode; +/* Select registers in the vAPIC cannot be read/written. */ +bool guest_apic_protected;/* Number of bits set in ISR. */ s16 isr_count;/* The highest vector set in ISR; if -1 - invalid, must scan ISR. */
arch/x86/kvm/vmx/main.c +11 -0
@@ -46,6 +46,9 @@ static __init int vt_hardware_setup(void) enable_tdx = enable_tdx && !tdx_hardware_setup (&vt_x86_ops); +if (!enable_tdx) + vt_x86_ops.protected_apic_has_interrupt =NULL ; +if (enable_ept)kvm_mmu_set_ept_masks (enable_ept_ad_bits,cpu_has_vmx_ept_execute_only ());
@@ -178,6 +181,13 @@ static void vt_vcpu_load(struct kvm_vcpu *vcpu, int cpu)return vmx_vcpu_load (vcpu, cpu); } +static bool vt_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) +{ +KVM_BUG_ON (!is_td_vcpu (vcpu), vcpu->kvm); + +return tdx_protected_apic_has_interrupt (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -348,6 +358,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .sync_pir_to_irr = vmx_sync_pir_to_irr, .deliver_interrupt = vmx_deliver_interrupt, .dy_apicv_has_pending_interrupt = pi_has_pending_interrupt, + .protected_apic_has_interrupt = vt_protected_apic_has_interrupt, .set_tss_addr = vmx_set_tss_addr, .set_identity_map_addr = vmx_set_identity_map_addr,
arch/x86/kvm/vmx/tdx.c +6 -0
@@ -513,6 +513,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu)return -EINVAL ;fpstate_set_confidential (&vcpu->arch.guest_fpu); + vcpu->arch.apic->guest_apic_protected = true; ret =tdx_alloc_td_page (&tdx->tdvpr);if (ret)
@@ -577,6 +578,11 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu)local_irq_enable (); } +bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) +{ +return pi_has_pending_interrupt (vcpu); +} +void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {struct vcpu_tdx *tdx =to_tdx (vcpu);
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -150,6 +150,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu);void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu);void tdx_vcpu_put (struct kvm_vcpu *vcpu);void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu); +bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -177,6 +178,7 @@ static inline fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) { return EXIT_FASTPstatic inline void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {} +static inline bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) {return false; }static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v8 75/103 KVM: x86: Assume timer IRQ was injected if APIC state is proteced
In reply to [email protected]From: Sean Christopherson <[email protected]>
If APIC state is protected, i.e. the vCPU is a TDX guest, assume a timer IRQ was injected when deciding whether or not to busy wait in the "timer advanced" path. The "real" vIRR is not readable/writable, so trying to query for a pending timer IRQ will return garbage.
Note, TDX can scour the PIR if it wants to be more precise and skip the "wait" call entirely.
Signed-off-by Sean Christopherson <[email protected]>
--- arch/x86/kvm/lapic.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c +10 -1
@@ -1606,8 +1606,17 @@ static void apic_update_lvtt(struct kvm_lapic *apic)static bool lapic_timer_int_injected (struct kvm_vcpu *vcpu) {struct kvm_lapic *apic = vcpu->arch.apic; -u32 reg =kvm_lapic_get_reg (apic,APIC_LVTT ); +u32 reg; +/* +* Assume a timer IRQ was "injected" if the APIC is protected. KVM's +* copy of the vIRR is bogus, it's the responsibility of the caller to +* precisely check whether or not a timer IRQ is pending. +*/ +if (apic->guest_apic_protected) +return true; + + reg =kvm_lapic_get_reg (apic,APIC_LVTT );if (kvm_apic_hw_enabled (apic)) {int vec = reg &APIC_VECTOR_MASK ;void *bitmap = apic->regs +APIC_ISR ;
Signature · 1 line
2.25.1
PATCH v8 76/103 KVM: TDX: remove use of struct vcpu_vmx from posted_interrupt.c
In reply to [email protected]From: Isaku Yamahata <[email protected]>
As TDX will use posted_interrupt.c, the use of struct vcpu_vmx is a blocker. Because the members of struct pi_desc pi_desc and struct list_head pi_wakeup_list are only used in posted_interrupt.c, introduce common structure, struct vcpu_pi, make vcpu_vmx and vcpu_tdx has same layout in the top of structure.
To minimize the diff size, avoid code conversion like, vmx->pi_desc => vmx->common->pi_desc. Instead add compile time check if the layout is expected.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/posted_intr.c | 41 ++++++++++++++++++++++++++-------- arch/x86/kvm/vmx/posted_intr.h | 11 +++++++++ arch/x86/kvm/vmx/tdx.c | 1 + arch/x86/kvm/vmx/tdx.h | 8 +++++++ arch/x86/kvm/vmx/vmx.h | 14 +++++++----- 5 files changed, 60 insertions(+), 15 deletions(-)
arch/x86/kvm/vmx/posted_intr.c +32 -9
@@ -9,6 +9,7 @@#include "posted_intr.h" #include "trace.h" #include "vmx.h" +#include "tdx.h" /* * Maintain a per-CPU list of vCPUs that need to be awakened by wakeup_handler()
@@ -29,9 +30,29 @@ static DEFINE_PER_CPU(struct list_head, wakeup_vcpus_on_cpu);*/ static DEFINE_PER_CPU(raw_spinlock_t, wakeup_vcpus_on_cpu_lock); +/* +* The layout of the head of struct vcpu_vmx and struct vcpu_tdx must match with +* struct vcpu_pi. +*/ +static_assert (offsetof(struct vcpu_pi , pi_desc) == + offsetof(struct vcpu_vmx , pi_desc)); +static_assert (offsetof(struct vcpu_pi , pi_wakeup_list) == + offsetof(struct vcpu_vmx , pi_wakeup_list)); +#ifdef CONFIG_INTEL_TDX_HOST +static_assert (offsetof(struct vcpu_pi , pi_desc) == + offsetof(struct vcpu_tdx , pi_desc)); +static_assert (offsetof(struct vcpu_pi , pi_wakeup_list) == + offsetof(struct vcpu_tdx , pi_wakeup_list)); +#endif + +static inline struct vcpu_pi *vcpu_to_pi (struct kvm_vcpu *vcpu) +{ +return (struct vcpu_pi *)vcpu; +} +static inline struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu) { -return &(to_vmx (vcpu)->pi_desc); +return &vcpu_to_pi (vcpu)->pi_desc; }static int pi_try_set_control (struct pi_desc *pi_desc,u64 *pold,u64 new)
@@ -50,8 +71,8 @@ static int pi_try_set_control(struct pi_desc *pi_desc, u64 *pold, u64 new)void vmx_vcpu_pi_load (struct kvm_vcpu *vcpu,int cpu) { -struct pi_desc *pi_desc =vcpu_to_pi_desc (vcpu); -struct vcpu_vmx *vmx =to_vmx (vcpu); +struct vcpu_pi *vcpu_pi =vcpu_to_pi (vcpu); +struct pi_desc *pi_desc = &vcpu_pi->pi_desc;struct pi_desc old, new;unsigned long flags;unsigned int dest;
@@ -88,7 +109,7 @@ void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)*/ if (pi_desc->nv ==POSTED_INTR_WAKEUP_VECTOR ) {raw_spin_lock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); -list_del (&vmx->pi_wakeup_list); +list_del (&vcpu_pi->pi_wakeup_list);raw_spin_unlock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); }
@@ -143,15 +164,15 @@ static bool vmx_can_use_vtd_pi(struct kvm *kvm)*/ static void pi_enable_wakeup_handler (struct kvm_vcpu *vcpu) { -struct pi_desc *pi_desc =vcpu_to_pi_desc (vcpu); -struct vcpu_vmx *vmx =to_vmx (vcpu); +struct vcpu_pi *vcpu_pi =vcpu_to_pi (vcpu); +struct pi_desc *pi_desc = &vcpu_pi->pi_desc;struct pi_desc old, new;unsigned long flags;local_irq_save (flags);raw_spin_lock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); -list_add_tail (&vmx->pi_wakeup_list, +list_add_tail (&vcpu_pi->pi_wakeup_list, &per_cpu (wakeup_vcpus_on_cpu, vcpu->cpu));raw_spin_unlock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu));
@@ -188,7 +209,8 @@ static bool vmx_needs_pi_wakeup(struct kvm_vcpu *vcpu)* notification vector is switched to the one that calls * back to the pi_wakeup_handler() function. */ -return vmx_can_use_ipiv (vcpu) ||vmx_can_use_vtd_pi (vcpu->kvm); +return (vmx_can_use_ipiv (vcpu) && !is_td_vcpu (vcpu)) || +vmx_can_use_vtd_pi (vcpu->kvm); }void vmx_vcpu_pi_put (struct kvm_vcpu *vcpu)
@@ -198,7 +220,8 @@ void vmx_vcpu_pi_put(struct kvm_vcpu *vcpu)if (!vmx_needs_pi_wakeup (vcpu))return ; -if (kvm_vcpu_is_blocking (vcpu) && !vmx_interrupt_blocked (vcpu)) +if (kvm_vcpu_is_blocking (vcpu) && + (is_td_vcpu (vcpu) || !vmx_interrupt_blocked (vcpu)))pi_enable_wakeup_handler (vcpu);/*
arch/x86/kvm/vmx/posted_intr.h +11 -0
@@ -94,6 +94,17 @@ static inline bool pi_test_sn(struct pi_desc *pi_desc) (unsigned long *)&pi_desc->control); } +struct vcpu_pi { +struct kvm_vcpu vcpu; + +/* Posted interrupt descriptor */ +struct pi_desc pi_desc; + +/* Used if this vCPU is waiting for PI notification wakeup. */ +struct list_head pi_wakeup_list; +/* Until here common layout betwwn vcpu_vmx and vcpu_tdx. */ +}; +void vmx_vcpu_pi_load (struct kvm_vcpu *vcpu,int cpu);void vmx_vcpu_pi_put (struct kvm_vcpu *vcpu);void pi_wakeup_handler (void );
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -514,6 +514,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu)fpstate_set_confidential (&vcpu->arch.guest_fpu); vcpu->arch.apic->guest_apic_protected = true; +INIT_LIST_HEAD (&tdx->pi_wakeup_list); ret =tdx_alloc_td_page (&tdx->tdvpr);if (ret)
arch/x86/kvm/vmx/tdx.h +8 -0
@@ -4,6 +4,7 @@#ifdef CONFIG_INTEL_TDX_HOST +#include "posted_intr.h" #include "pmu_intel.h" #include "tdx_ops.h"
@@ -81,6 +82,13 @@ union tdx_exit_reason {struct vcpu_tdx {struct kvm_vcpu vcpu; +/* Posted interrupt descriptor */ +struct pi_desc pi_desc; + +/* Used if this vCPU is waiting for PI notification wakeup. */ +struct list_head pi_wakeup_list; +/* Until here same layout to struct vcpu_pi. */ +struct tdx_td_page tdvpr;struct tdx_td_page *tdvpx;
arch/x86/kvm/vmx/vmx.h +8 -6
@@ -248,6 +248,14 @@ struct nested_vmx {struct vcpu_vmx {struct kvm_vcpu vcpu; + +/* Posted interrupt descriptor */ +struct pi_desc pi_desc; + +/* Used if this vCPU is waiting for PI notification wakeup. */ +struct list_head pi_wakeup_list; +/* Until here same layout to struct vcpu_pi. */ +u8 fail;u8 x2apic_msr_bitmap_mode;
@@ -317,12 +325,6 @@ struct vcpu_vmx {union vmx_exit_reason exit_reason; -/* Posted interrupt descriptor */ -struct pi_desc pi_desc; - -/* Used if this vCPU is waiting for PI notification wakeup. */ -struct list_head pi_wakeup_list; -/* Support for a guest hypervisor (nested VMX) */ struct nested_vmx nested;
Signature · 1 line
2.25.1
PATCH v8 77/103 KVM: TDX: Implement interrupt injection
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX supports interrupt inject into vcpu with posted interrupt. Wire up the corresponding kvm x86 operations to posted interrupt. Move kvm_vcpu_trigger_posted_interrupt() from vmx.c to common.h to share the code.
VMX can inject interrupt by setting interrupt information field, VM_ENTRY_INTR_INFO_FIELD, of VMCS. TDX supports interrupt injection only by posted interrupt. Ignore the execution path to access VM_ENTRY_INTR_INFO_FIELD.
As cpu state is protected and apicv is enabled for the TDX guest, VMM can inject interrupt by updating posted interrupt descriptor. Treat interrupt can be injected always.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/common.h | 71 ++++++++++++++++++++++++++ arch/x86/kvm/vmx/main.c | 92 ++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/posted_intr.c | 2 +- arch/x86/kvm/vmx/posted_intr.h | 2 + arch/x86/kvm/vmx/tdx.c | 25 +++++++++ arch/x86/kvm/vmx/vmx.c | 67 +------------------------ arch/x86/kvm/vmx/x86_ops.h | 7 ++- 7 files changed, 189 insertions(+), 77 deletions(-)
arch/x86/kvm/vmx/common.h +71 -0
@@ -4,6 +4,7 @@#include <linux/kvm_host.h> +#include "posted_intr.h" #include "mmu.h" static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t gpa,
@@ -30,4 +31,74 @@ static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t gpa, return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0); } +static inline void kvm_vcpu_trigger_posted_interrupt (struct kvm_vcpu *vcpu, +int pi_vec) +{ +#ifdef CONFIG_SMP +if (vcpu->mode ==IN_GUEST_MODE ) { +/* +* The vector of the virtual has already been set in the PIR. +* Send a notification event to deliver the virtual interrupt +* unless the vCPU is the currently running vCPU, i.e. the +* event is being sent from a fastpath VM-Exit handler, in +* which case the PIR will be synced to the vIRR before +* re-entering the guest. +* +* When the target is not the running vCPU, the following +* possibilities emerge: +* +* Case 1: vCPU stays in non-root mode. Sending a notification +* event posts the interrupt to the vCPU. +* +* Case 2: vCPU exits to root mode and is still runnable. The +* PIR will be synced to the vIRR before re-entering the guest. +* Sending a notification event is ok as the host IRQ handler +* will ignore the spurious event. +* +* Case 3: vCPU exits to root mode and is blocked. vcpu_block() +* has already synced PIR to vIRR and never blocks the vCPU if +* the vIRR is not empty. Therefore, a blocked vCPU here does +* not wait for any requested interrupts in PIR, and sending a +* notification event also results in a benign, spurious event. +*/ + +if (vcpu !=kvm_get_running_vcpu ()) + apic->send_IPI_mask (get_cpu_mask (vcpu->cpu), pi_vec); +return ; + } +#endif +/* +* The vCPU isn't in the guest; wake the vCPU in case it is blocking, +* otherwise do nothing as KVM will grab the highest priority pending +* IRQ via ->sync_pir_to_irr() in vcpu_enter_guest(). +*/ +kvm_vcpu_wake_up (vcpu); +} + +/* +* Send interrupt to vcpu via posted interrupt way. +* 1. If target vcpu is running(non-root mode), send posted interrupt +* notification to vcpu and hardware will sync PIR to vIRR atomically. +* 2. If target vcpu isn't running(root mode), kick it to pick up the +* interrupt from PIR in next vmentry. +*/ +static inline void __vmx_deliver_posted_interrupt ( +struct kvm_vcpu *vcpu,struct pi_desc *pi_desc,int vector) +{ +if (pi_test_and_set_pir (vector, pi_desc)) +return ; + +/* If a previous notification has sent the IPI, nothing to do. */ +if (pi_test_and_set_on (pi_desc)) +return ; + +/* +* The implied barrier in pi_test_and_set_on() pairs with the smp_mb_*() +* after setting vcpu->mode in vcpu_enter_guest(), thus the vCPU is +* guaranteed to see PID.ON=1 and sync the PIR to IRR if triggering a +* posted interrupt "fails" because vcpu->mode != IN_GUEST_MODE. +*/ +kvm_vcpu_trigger_posted_interrupt (vcpu,POSTED_INTR_VECTOR ); +} +#endif /* __KVM_X86_VMX_COMMON_H */
arch/x86/kvm/vmx/main.c +83 -9
126 lines not on this page · raw message
arch/x86/kvm/vmx/posted_intr.c +1 -1
@@ -50,7 +50,7 @@ static inline struct vcpu_pi *vcpu_to_pi(struct kvm_vcpu *vcpu)return (struct vcpu_pi *)vcpu; } -static inline struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu) +struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu) {return &vcpu_to_pi (vcpu)->pi_desc; }
arch/x86/kvm/vmx/posted_intr.h +2 -0
@@ -105,6 +105,8 @@ struct vcpu_pi {/* Until here common layout betwwn vcpu_vmx and vcpu_tdx. */ }; +struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu); +void vmx_vcpu_pi_load (struct kvm_vcpu *vcpu,int cpu);void vmx_vcpu_pi_put (struct kvm_vcpu *vcpu);void pi_wakeup_handler (void );
arch/x86/kvm/vmx/tdx.c +25 -0
@@ -7,6 +7,7 @@#include "capabilities.h" #include "x86_ops.h" +#include "common.h" #include "mmu.h" #include "tdx.h" #include "vmx.h"
@@ -543,6 +544,9 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.guest_state_protected = !(to_kvm_tdx (vcpu->kvm)->attributes &TDX_TD_ATTRIBUTE_DEBUG ); + tdx->pi_desc.nv =POSTED_INTR_VECTOR ; + tdx->pi_desc.sn =1 ; + tdx->host_state_need_save = true; tdx->host_state_need_restore = false;
@@ -563,6 +567,7 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) {struct vcpu_tdx *tdx =to_tdx (vcpu); +vmx_vcpu_pi_load (vcpu, cpu);if (vcpu->cpu == cpu)return ;
@@ -776,6 +781,12 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu); +if (pi_test_on (&tdx->pi_desc)) { + apic->send_IPI_self (POSTED_INTR_VECTOR ); + +kvm_wait_lapic_expire (vcpu); + } +tdx_vcpu_enter_exit (vcpu, tdx);tdx_user_return_update_cache ();
@@ -1119,6 +1130,16 @@ static void tdx_handle_changed_private_spte( } } +void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode, +int trig_mode,int vector) +{ +struct kvm_vcpu *vcpu = apic->vcpu; +struct vcpu_tdx *tdx =to_tdx (vcpu); + +/* TDX supports only posted interrupt. No lapic emulation. */ +__vmx_deliver_posted_interrupt (vcpu, &tdx->pi_desc, vector); +} +int tdx_dev_ioctl (void __user *argp) {struct kvm_tdx_capabilities __user *user_caps;
@@ -1553,6 +1574,10 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)return -EIO ; } +td_vmcs_write16 (tdx,POSTED_INTR_NV ,POSTED_INTR_VECTOR ); +td_vmcs_write64 (tdx,POSTED_INTR_DESC_ADDR ,__pa (&tdx->pi_desc)); +td_vmcs_setbit32 (tdx,PIN_BASED_VM_EXEC_CONTROL ,PIN_BASED_POSTED_INTR ); + tdx->vcpu_initialized = true;return 0 ; }
arch/x86/kvm/vmx/vmx.c +1 -66
85 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +6 -1
@@ -53,7 +53,6 @@ int vmx_check_intercept(struct kvm_vcpu *vcpu,bool vmx_apic_init_signal_blocked (struct kvm_vcpu *vcpu);void vmx_migrate_timers (struct kvm_vcpu *vcpu);void vmx_set_virtual_apic_mode (struct kvm_vcpu *vcpu); -void vmx_apicv_post_state_restore (struct kvm_vcpu *vcpu);bool vmx_check_apicv_inhibit_reasons (enum kvm_apicv_inhibit reason);void vmx_hwapic_irr_update (struct kvm_vcpu *vcpu,int max_irr);void vmx_hwapic_isr_update (int max_isr);
@@ -152,6 +151,9 @@ void tdx_vcpu_put(struct kvm_vcpu *vcpu);void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu);bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu); +void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode, +int trig_mode,int vector); +int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -180,6 +182,9 @@ static inline void tdx_vcpu_put(struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {}static inline bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) {return false; } +static inline void tdx_deliver_interrupt ( +struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector) {} +static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v8 78/103 KVM: TDX: Implements vcpu request_immediate_exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Now we are able to inject interrupts into TDX vcpu, it's ready to block TDX vcpu. Wire up kvm x86 methods for blocking/unblocking vcpu for TDX. To unblock on pending events, request immediate exit methods is also needed.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/main.c +9 -1
@@ -328,6 +328,14 @@ static void vt_enable_irq_window(struct kvm_vcpu *vcpu)vmx_enable_irq_window (vcpu); } +static void vt_request_immediate_exit (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return __kvm_request_immediate_exit (vcpu); + +vmx_request_immediate_exit (vcpu); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -454,7 +462,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .check_intercept = vmx_check_intercept, .handle_exit_irqoff = vmx_handle_exit_irqoff, - .request_immediate_exit = vmx_request_immediate_exit, + .request_immediate_exit = vt_request_immediate_exit, .sched_in = vt_sched_in,
Signature · 1 line
2.25.1
PATCH v8 79/103 KVM: TDX: Implement methods to inject NMI
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX vcpu control structure defines one bit for pending NMI for VMM to inject NMI by setting the bit without knowing TDX vcpu NMI states. Because the vcpu state is protected, VMM can't know about NMI states of TDX vcpu. The TDX module handles actual injection and NMI states transition.
Add methods for NMI and treat NMI can be injected always.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/main.c | 62 +++++++++++++++++++++++++++++++++++--- arch/x86/kvm/vmx/tdx.c | 5 +++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ 3 files changed, 64 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/main.c +57 -5
@@ -264,6 +264,58 @@ static void vt_flush_tlb_guest(struct kvm_vcpu *vcpu)vmx_flush_tlb_guest (vcpu); } +static void vt_inject_nmi (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_inject_nmi (vcpu); + +vmx_inject_nmi (vcpu); +} + +static int vt_nmi_allowed (struct kvm_vcpu *vcpu,bool for_injection) +{ +/* +* The TDX module manages NMI windows and NMI reinjection, and hides NMI +* blocking, all KVM can do is throw an NMI over the wall. +*/ +if (is_td_vcpu (vcpu)) +return true; + +return vmx_nmi_allowed (vcpu, for_injection); +} + +static bool vt_get_nmi_mask (struct kvm_vcpu *vcpu) +{ +/* +* Assume NMIs are always unmasked. KVM could query PEND_NMI and treat +* NMIs as masked if a previous NMI is still pending, but SEAMCALLs are +* expensive and the end result is unchanged as the only relevant usage +* of get_nmi_mask() is to limit the number of pending NMIs, i.e. it +* only changes whether KVM or the TDX module drops an NMI. +*/ +if (is_td_vcpu (vcpu)) +return false; + +return vmx_get_nmi_mask (vcpu); +} + +static void vt_set_nmi_mask (struct kvm_vcpu *vcpu,bool masked) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_nmi_mask (vcpu, masked); +} + +static void vt_enable_nmi_window (struct kvm_vcpu *vcpu) +{ +/* Refer the comment in vt_get_nmi_mask(). */ +if (is_td_vcpu (vcpu)) +return ; + +vmx_enable_nmi_window (vcpu); +} +static void vt_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) {
@@ -418,14 +470,14 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .get_interrupt_shadow = vt_get_interrupt_shadow, .patch_hypercall = vmx_patch_hypercall, .inject_irq = vt_inject_irq, - .inject_nmi = vmx_inject_nmi, + .inject_nmi = vt_inject_nmi, .queue_exception = vmx_queue_exception, .cancel_injection = vt_cancel_injection, .interrupt_allowed = vt_interrupt_allowed, - .nmi_allowed = vmx_nmi_allowed, - .get_nmi_mask = vmx_get_nmi_mask, - .set_nmi_mask = vmx_set_nmi_mask, - .enable_nmi_window = vmx_enable_nmi_window, + .nmi_allowed = vt_nmi_allowed, + .get_nmi_mask = vt_get_nmi_mask, + .set_nmi_mask = vt_set_nmi_mask, + .enable_nmi_window = vt_enable_nmi_window, .enable_irq_window = vt_enable_irq_window, .update_cr8_intercept = vmx_update_cr8_intercept, .set_virtual_apic_mode = vmx_set_virtual_apic_mode,
arch/x86/kvm/vmx/tdx.c +5 -0
@@ -802,6 +802,11 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)return EXIT_FASTPATH_NONE ; } +void tdx_inject_nmi (struct kvm_vcpu *vcpu) +{ +td_management_write8 (to_tdx (vcpu),TD_VCPU_PEND_NMI ,1 ); +} +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) {td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK );
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -153,6 +153,7 @@ bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu);void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector); +void tdx_inject_nmi (struct kvm_vcpu *vcpu);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -184,6 +185,7 @@ static inline bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu) { retstatic inline void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector) {} +static inline void tdx_inject_nmi (struct kvm_vcpu *vcpu) {}static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v8 80/103 KVM: VMX: Modify NMI and INTR handlers to take intr_info as function argument
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX uses different ABI to get information about VM exit. Pass intr_info to the NMI and INTR handlers instead of pulling it from vcpu_vmx in preparation for sharing the bulk of the handlers with TDX.
When the guest TD exits to VMM, RAX holds status and exit reason, RCX holds exit qualification etc rather than the VMCS fields because VMM doesn't have access to the VMCS. The eventual code will be
VMX:
- get exit reason, intr_info, exit_qualification, and etc from VMCS
- call NMI/INTR handlers (common code)
TDX:
- get exit reason, intr_info, exit_qualification, and etc from guest registers
- call NMI/INTR handlers (common code)
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/vmx.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
arch/x86/kvm/vmx/vmx.c +8 -9
@@ -6773,28 +6773,27 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu)rdmsrl (MSR_IA32_XFD_ERR , vcpu->arch.guest_fpu.xfd_err); } -static void handle_exception_nmi_irqoff (struct vcpu_vmx *vmx) +static void handle_exception_nmi_irqoff (struct kvm_vcpu *vcpu,u32 intr_info) {const unsigned long nmi_entry = (unsigned long )asm_exc_nmi_noist; -u32 intr_info =vmx_get_intr_info (&vmx->vcpu);/* if exit due to PF check for async PF */ if (is_page_fault (intr_info)) - vmx->vcpu.arch.apf.host_apf_flags =kvm_read_and_reset_apf_flags (); + vcpu->arch.apf.host_apf_flags =kvm_read_and_reset_apf_flags ();/* if exit due to NM, handle before interrupts are enabled */ else if (is_nm_fault (intr_info)) -handle_nm_fault_irqoff (&vmx->vcpu); +handle_nm_fault_irqoff (vcpu);/* Handle machine checks before interrupts are enabled */ else if (is_machine_check (intr_info))kvm_machine_check ();/* We need to handle NMIs before interrupts are enabled */ else if (is_nmi (intr_info)) -handle_interrupt_nmi_irqoff (&vmx->vcpu, nmi_entry); +handle_interrupt_nmi_irqoff (vcpu, nmi_entry); } -static void handle_external_interrupt_irqoff (struct kvm_vcpu *vcpu) +static void handle_external_interrupt_irqoff (struct kvm_vcpu *vcpu, +u32 intr_info) { -u32 intr_info =vmx_get_intr_info (vcpu);unsigned int vector = intr_info &INTR_INFO_VECTOR_MASK ;gate_desc *desc = (gate_desc *)host_idt_base + vector;
@@ -6814,9 +6813,9 @@ void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu)return ;if (vmx->exit_reason.basic ==EXIT_REASON_EXTERNAL_INTERRUPT ) -handle_external_interrupt_irqoff (vcpu); +handle_external_interrupt_irqoff (vcpu,vmx_get_intr_info (vcpu));else if (vmx->exit_reason.basic ==EXIT_REASON_EXCEPTION_NMI ) -handle_exception_nmi_irqoff (vmx); +handle_exception_nmi_irqoff (vcpu,vmx_get_intr_info (vcpu)); }/*
Signature · 1 line
2.25.1
PATCH v8 81/103 KVM: VMX: Move NMI/exception handler to common helper
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX mostly handles NMI/exception exit mostly the same to VMX case. The difference is how to retrieve exit qualification. To share the code with TDX, move NMI/exception to a common header, common.h.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/common.h | 70 ++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 79 ++++----------------------------------- 2 files changed, 78 insertions(+), 71 deletions(-)
arch/x86/kvm/vmx/common.h +70 -0
@@ -4,8 +4,78 @@#include <linux/kvm_host.h> +#include <asm/traps.h> +#include "posted_intr.h" #include "mmu.h" +#include "vmcs.h" +#include "x86.h" + +extern unsigned long vmx_host_idt_base; +void vmx_do_interrupt_nmi_irqoff (unsigned long entry); + +static inline void vmx_handle_interrupt_nmi_irqoff (struct kvm_vcpu *vcpu, +unsigned long entry) +{ +bool is_nmi = entry == (unsigned long )asm_exc_nmi_noist; + +kvm_before_interrupt (vcpu, is_nmi ?KVM_HANDLING_NMI :KVM_HANDLING_IRQ ); +vmx_do_interrupt_nmi_irqoff (entry); +kvm_after_interrupt (vcpu); +} + +static inline void vmx_handle_nm_fault_irqoff (struct kvm_vcpu *vcpu) +{ +/* +* Save xfd_err to guest_fpu before interrupt is enabled, so the +* MSR value is not clobbered by the host activity before the guest +* has chance to consume it. +* +* Do not blindly read xfd_err here, since this exception might +* be caused by L1 interception on a platform which doesn't +* support xfd at all. +* +* Do it conditionally upon guest_fpu::xfd. xfd_err matters +* only when xfd contains a non-zero value. +* +* Queuing exception is done in vmx_handle_exit. See comment there. +*/ +if (vcpu->arch.guest_fpu.fpstate->xfd) +rdmsrl (MSR_IA32_XFD_ERR , vcpu->arch.guest_fpu.xfd_err); +} + +static inline void vmx_handle_exception_nmi_irqoff (struct kvm_vcpu *vcpu, +u32 intr_info) +{ +const unsigned long nmi_entry = (unsigned long )asm_exc_nmi_noist; + +/* if exit due to PF check for async PF */ +if (is_page_fault (intr_info)) + vcpu->arch.apf.host_apf_flags =kvm_read_and_reset_apf_flags (); +/* if exit due to NM, handle before interrupts are enabled */ +else if (is_nm_fault (intr_info)) +vmx_handle_nm_fault_irqoff (vcpu); +/* Handle machine checks before interrupts are enabled */ +else if (is_machine_check (intr_info)) +kvm_machine_check (); +/* We need to handle NMIs before interrupts are enabled */ +else if (is_nmi (intr_info)) +vmx_handle_interrupt_nmi_irqoff (vcpu, nmi_entry); +} + +static inline void vmx_handle_external_interrupt_irqoff (struct kvm_vcpu *vcpu, +u32 intr_info) +{ +unsigned int vector = intr_info &INTR_INFO_VECTOR_MASK ; +gate_desc *desc = (gate_desc *)vmx_host_idt_base + vector; + +if (KVM_BUG (!is_external_intr (intr_info), vcpu->kvm, +"KVM: unexpected VM-Exit interrupt info: 0x%x" , intr_info)) +return ; + +vmx_handle_interrupt_nmi_irqoff (vcpu,gate_offset (desc)); + vcpu->arch.at_instruction_boundary = true; +}static inline int __vmx_handle_ept_violation (struct kvm_vcpu *vcpu,gpa_t gpa,unsigned long exit_qualification)
arch/x86/kvm/vmx/vmx.c +8 -71
118 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v8 82/103 KVM: x86: Split core of hypercall emulation to helper function
In reply to [email protected]From: Sean Christopherson <[email protected]>
By necessity, TDX will use a different register ABI for hypercalls. Break out the core functionality so that it may be reused for TDX.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/include/asm/kvm_host.h | 4 +++ arch/x86/kvm/x86.c | 54 ++++++++++++++++++++------------- 2 files changed, 37 insertions(+), 21 deletions(-)
arch/x86/include/asm/kvm_host.h +4 -0
@@ -1995,6 +1995,10 @@ static inline void kvm_clear_apicv_inhibit(struct kvm *kvm,kvm_set_or_clear_apicv_inhibit (kvm, reason, false); } +unsigned long __kvm_emulate_hypercall (struct kvm_vcpu *vcpu,unsigned long nr, +unsigned long a0,unsigned long a1, +unsigned long a2,unsigned long a3, +int op_64_bit);int kvm_emulate_hypercall (struct kvm_vcpu *vcpu); int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,
arch/x86/kvm/x86.c +33 -21
76 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v8 83/103 KVM: TDX: Add a place holder to handle TDX VM exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up handle_exit and handle_exit_irqoff methods and add a place holder to handle VM exit. Add helper functions to get exit info, exit qualification, etc.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/main.c | 33 ++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 81 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 11 ++++++ 3 files changed, 122 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/main.c +30 -3
@@ -188,6 +188,23 @@ static bool vt_protected_apic_has_interrupt(struct kvm_vcpu *vcpu)return tdx_protected_apic_has_interrupt (vcpu); } +static int vt_handle_exit (struct kvm_vcpu *vcpu, +enum exit_fastpath_completion fastpath) +{ +if (is_td_vcpu (vcpu)) +return tdx_handle_exit (vcpu, fastpath); + +return vmx_handle_exit (vcpu, fastpath); +} + +static void vt_handle_exit_irqoff (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_handle_exit_irqoff (vcpu); + +vmx_handle_exit_irqoff (vcpu); +} +static void vt_apicv_post_state_restore (struct kvm_vcpu *vcpu) {struct pi_desc *pi =vcpu_to_pi_desc (vcpu);
@@ -388,6 +405,16 @@ static void vt_request_immediate_exit(struct kvm_vcpu *vcpu)vmx_request_immediate_exit (vcpu); } +static void vt_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason, +u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_exit_info (vcpu, reason, info1, info2, intr_info, + error_code); + +return vmx_get_exit_info (vcpu, reason, info1, info2, intr_info, error_code); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -463,7 +490,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_pre_run = vt_vcpu_pre_run, .vcpu_run = vt_vcpu_run, - .handle_exit = vmx_handle_exit, + .handle_exit = vt_handle_exit, .skip_emulated_instruction = vmx_skip_emulated_instruction, .update_emulated_instruction = vmx_update_emulated_instruction, .set_interrupt_shadow = vt_set_interrupt_shadow,
@@ -498,7 +525,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_identity_map_addr = vmx_set_identity_map_addr, .get_mt_mask = vmx_get_mt_mask, - .get_exit_info = vmx_get_exit_info, + .get_exit_info = vt_get_exit_info, .vcpu_after_set_cpuid = vmx_vcpu_after_set_cpuid,
@@ -512,7 +539,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .load_mmu_pgd = vt_load_mmu_pgd, .check_intercept = vmx_check_intercept, - .handle_exit_irqoff = vmx_handle_exit_irqoff, + .handle_exit_irqoff = vt_handle_exit_irqoff, .request_immediate_exit = vt_request_immediate_exit,
arch/x86/kvm/vmx/tdx.c +81 -0
99 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +11 -0
@@ -150,10 +150,15 @@ void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu);void tdx_vcpu_put (struct kvm_vcpu *vcpu);void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu);bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu); +void tdx_handle_exit_irqoff (struct kvm_vcpu *vcpu); +int tdx_handle_exit (struct kvm_vcpu *vcpu, +enum exit_fastpath_completion fastpath);void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector);void tdx_inject_nmi (struct kvm_vcpu *vcpu); +void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason, +u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -182,10 +187,16 @@ static inline void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {}static inline bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) {return false; } +static inline void tdx_handle_exit_irqoff (struct kvm_vcpu *vcpu) {} +static inline int tdx_handle_exit (struct kvm_vcpu *vcpu, +enum exit_fastpath_completion fastpath) {return 0 ; }static inline void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector) {}static inline void tdx_inject_nmi (struct kvm_vcpu *vcpu) {} +static inline void tdx_get_exit_info ( +struct kvm_vcpu *vcpu,u32 *reason,u64 *info1,u64 *info2, +u32 *intr_info,u32 *error_code) {}static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v8 84/103 KVM: TDX: Retry seamcall when TDX_OPERAND_BUSY with operand SEPT
In reply to [email protected]From: Yuan Yao <[email protected]>
TDX module internally uses locks to protect internal resources. It tries to acquire the locks. If it fails to obtain the lock, it returns TDX_OPERAND_BUSY error without spin because its execution time limitation.
TDX SEAMCALL API reference describes what resources are used. It's known which TDX SEAMCALL can cause contention with which resources. VMM can avoid contention inside the TDX module by avoiding contentious TDX SEAMCALL with, for example, spinlock. Because OS knows better its process scheduling and its scalability, a lock at OS/VMM layer would work better than simply retrying TDX SEAMCALLs.
TDH.MEM.* API except for TDH.MEM.TRACK operates on a secure EPT tree and the TDX module internally tries to acquire the lock of the secure EPT tree. They return TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT in case of failure to get the lock. TDX KVM uses kvm_tdx::seamcall_lock spinlock at OS/VMM layer to avoid contention inside the TDX module.
TDH.VP.ENTER is an exception with zero-step attack mitigation. Normally TDH.VP.ENTER uses only TD vcpu resources and it doesn't cause contention. When a zero-step attack is suspected, it obtains a secure EPT tree lock and tracks the GPAs causing a secure EPT fault. Thus TDG.VP.ENTER may result in TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT. Also TDH.MEM.* SEAMCALLs may result in TDX_OPERAN_BUSY | TDX_OPERAND_ID_SEPT because TDH.VP.ENTER is not protected with seamcall_lock.
Retry TDX TDH.MEM.* API and TDH.VP.ENTER on the error because the error is a rare event caused by zero-step attack mitigation and spinlock can not be used for TDH.VP.ENTER due to indefinite time execution.
Signed-off-by Yuan Yao <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 4 ++++ arch/x86/kvm/vmx/tdx_ops.h | 36 ++++++++++++++++++++++++++++++------ 2 files changed, 34 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/tdx.c +4 -0
@@ -1188,6 +1188,10 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) {union tdx_exit_reason exit_reason =to_tdx (vcpu)->exit_reason; +/* See the comment of tdh_sept_seamcall(). */ +if (unlikely (exit_reason.full == (TDX_OPERAND_BUSY |TDX_OPERAND_ID_SEPT ))) +return 1 ; +if (unlikely (exit_reason.non_recoverable || exit_reason.error)) {if (exit_reason.basic ==EXIT_REASON_TRIPLE_FAULT )return tdx_handle_triple_fault (vcpu);
arch/x86/kvm/vmx/tdx_ops.h +30 -6
@@ -18,6 +18,26 @@void pr_tdx_error (u64 op,u64 error_code,const struct tdx_module_output *out); +/* +* Although seamcal_lock protects seamcall to avoid contention inside the TDX +* module, it doesn't protect TDH.VP.ENTER. With zero-step attack mitigation, +* TDH.VP.ENTER may rarely acquire SEPT lock and release it when zero-step +* attack is suspected. It results in TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT +* with TDH.MEM.* operation. (TDH.MEM.TRACK is an exception.) Because such +* error is rare event, just retry on those TDH.MEM operations and TDH.VP.ENTER. +*/ +static inline u64 seamcall_sept_retry (u64 op,u64 rcx,u64 rdx,u64 r8,u64 r9, +struct tdx_module_output *out) +{ +u64 ret; + +do { + ret =__seamcall (op, rcx, rdx, r8, r9, out); + }while (unlikely (ret == (TDX_OPERAND_BUSY |TDX_OPERAND_ID_SEPT ))); + +return ret; +} +static inline u64 tdh_mng_addcx (hpa_t tdr,hpa_t addr) {clflush_cache_range (__va (addr),PAGE_SIZE );
@@ -28,14 +48,15 @@ static inline u64 tdh_mem_page_add(hpa_t tdr, gpa_t gpa, hpa_t hpa, hpa_t source struct tdx_module_output *out) {clflush_cache_range (__va (hpa),PAGE_SIZE ); -return __seamcall (TDH_MEM_PAGE_ADD , gpa, tdr, hpa, source, out); +return seamcall_sept_retry (TDH_MEM_PAGE_ADD , gpa, tdr, hpa, source, out); }static inline u64 tdh_mem_sept_add (hpa_t tdr,gpa_t gpa,int level,hpa_t page,struct tdx_module_output *out) {clflush_cache_range (__va (page),PAGE_SIZE ); -return __seamcall (TDH_MEM_SEPT_ADD , gpa | level, tdr, page,0 , out); +return seamcall_sept_retry (TDH_MEM_SEPT_ADD , gpa | level, tdr, page,0 , + out); } static inline u64 tdh_mem_sept_remove(hpa_t tdr, gpa_t gpa, int level,
@@ -61,13 +82,14 @@ static inline u64 tdh_mem_page_aug(hpa_t tdr, gpa_t gpa, hpa_t hpa, struct tdx_module_output *out) {clflush_cache_range (__va (hpa),PAGE_SIZE ); -return __seamcall (TDH_MEM_PAGE_AUG , gpa, tdr, hpa,0 , out); +return seamcall_sept_retry (TDH_MEM_PAGE_AUG , gpa, tdr, hpa,0 , out); }static inline u64 tdh_mem_range_block (hpa_t tdr,gpa_t gpa,int level,struct tdx_module_output *out) { -return __seamcall (TDH_MEM_RANGE_BLOCK , gpa | level, tdr,0 ,0 , out); +return seamcall_sept_retry (TDH_MEM_RANGE_BLOCK , gpa | level, tdr,0 ,0 , + out); }static inline u64 tdh_mng_key_config (hpa_t tdr)
@@ -149,7 +171,8 @@ static inline u64 tdh_phymem_page_reclaim(hpa_t page,static inline u64 tdh_mem_page_remove (hpa_t tdr,gpa_t gpa,int level,struct tdx_module_output *out) { -return __seamcall (TDH_MEM_PAGE_REMOVE , gpa | level, tdr,0 ,0 , out); +return seamcall_sept_retry (TDH_MEM_PAGE_REMOVE , gpa | level, tdr,0 ,0 , + out); }static inline u64 tdh_sys_lp_shutdown (void )
@@ -165,7 +188,8 @@ static inline u64 tdh_mem_track(hpa_t tdr)static inline u64 tdh_mem_range_unblock (hpa_t tdr,gpa_t gpa,int level,struct tdx_module_output *out) { -return __seamcall (TDH_MEM_RANGE_UNBLOCK , gpa | level, tdr,0 ,0 , out); +return seamcall_sept_retry (TDH_MEM_RANGE_UNBLOCK , gpa | level, tdr,0 ,0 , + out); }static inline u64 tdh_phymem_cache_wb (bool resume)
Signature · 1 line
2.25.1
PATCH v8 85/103 KVM: TDX: handle EXIT_REASON_OTHER_SMI
In reply to [email protected]From: Isaku Yamahata <[email protected]>
If the control reaches EXIT_REASON_OTHER_SMI, #SMI is delivered and handled right after returning from the TDX module to KVM nothing needs to be done in KVM. Continue TDX vcpu execution.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/include/uapi/asm/vmx.h | 1 + arch/x86/kvm/vmx/tdx.c | 7 +++++++ 2 files changed, 8 insertions(+)
arch/x86/include/uapi/asm/vmx.h +1 -0
@@ -34,6 +34,7 @@#define EXIT_REASON_TRIPLE_FAULT 2#define EXIT_REASON_INIT_SIGNAL 3#define EXIT_REASON_SIPI_SIGNAL 4 +#define EXIT_REASON_OTHER_SMI 6#define EXIT_REASON_INTERRUPT_WINDOW 7#define EXIT_REASON_NMI_WINDOW 8
arch/x86/kvm/vmx/tdx.c +7 -0
@@ -1206,6 +1206,13 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath)WARN_ON_ONCE (fastpath !=EXIT_FASTPATH_NONE );switch (exit_reason.basic) { +case EXIT_REASON_OTHER_SMI : +/* +* If reach here, it's not a Machine Check System Management +* Interrupt(MSMI). #SMI is delivered and handled right after +* SEAMRET, nothing needs to be done in KVM. +*/ +return 1 ;default :break ; }
Signature · 1 line
2.25.1
PATCH v8 86/103 KVM: TDX: handle ept violation/misconfig exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
On EPT violation, call a common function, __vmx_handle_ept_violation() to trigger x86 MMU code. On EPT misconfiguration, exit to ring 3 with KVM_EXIT_UNKNOWN. because EPT misconfiguration can't happen as MMIO is trigged by TDG.VP.VMCALL. No point to set a misconfiguration value for the fast path.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 46 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+)
arch/x86/kvm/vmx/tdx.c +46 -0
@@ -1184,6 +1184,48 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,__vmx_deliver_posted_interrupt (vcpu, &tdx->pi_desc, vector); } +static int tdx_handle_ept_violation (struct kvm_vcpu *vcpu) +{ +unsigned long exit_qual; + +if (kvm_is_private_gpa (vcpu->kvm,tdexit_gpa (vcpu))) { +/* +* Always treat SEPT violations as write faults. Ignore the +* EXIT_QUALIFICATION reported by TDX-SEAM for SEPT violations. +* TD private pages are always RWX in the SEPT tables, +* i.e. they're always mapped writable. Just as importantly, +* treating SEPT violations as write faults is necessary to +* avoid COW allocations, which will cause TDAUGPAGE failures +* due to aliasing a single HPA to multiple GPAs. +*/ +#define TDX_SEPT_VIOLATION_EXIT_QUAL EPT_VIOLATION_ACC_WRITE + exit_qual =TDX_SEPT_VIOLATION_EXIT_QUAL ; + }else { + exit_qual =tdexit_exit_qual (vcpu);; +if (exit_qual &EPT_VIOLATION_ACC_INSTR ) { +pr_warn ("kvm: TDX instr fetch to shared GPA = 0x%lx @ RIP = 0x%lx\n" , +tdexit_gpa (vcpu),kvm_rip_read (vcpu)); + vcpu->run->exit_reason =KVM_EXIT_EXCEPTION ; + vcpu->run->ex.exception =PF_VECTOR ; + vcpu->run->ex.error_code = exit_qual; +return 0 ; + } + } + +trace_kvm_page_fault (tdexit_gpa (vcpu), exit_qual); +return __vmx_handle_ept_violation (vcpu,tdexit_gpa (vcpu), exit_qual); +} + +static int tdx_handle_ept_misconfig (struct kvm_vcpu *vcpu) +{ +WARN_ON (1 ); + + vcpu->run->exit_reason =KVM_EXIT_UNKNOWN ; + vcpu->run->hw.hardware_exit_reason =EXIT_REASON_EPT_MISCONFIG ; + +return 0 ; +} +int tdx_handle_exit (struct kvm_vcpu *vcpu,fastpath_t fastpath) {union tdx_exit_reason exit_reason =to_tdx (vcpu)->exit_reason;
@@ -1206,6 +1248,10 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath)WARN_ON_ONCE (fastpath !=EXIT_FASTPATH_NONE );switch (exit_reason.basic) { +case EXIT_REASON_EPT_VIOLATION : +return tdx_handle_ept_violation (vcpu); +case EXIT_REASON_EPT_MISCONFIG : +return tdx_handle_ept_misconfig (vcpu);case EXIT_REASON_OTHER_SMI :/* * If reach here, it's not a Machine Check System Management
Signature · 1 line
2.25.1
PATCH v8 87/103 KVM: TDX: handle EXCEPTION_NMI and EXTERNAL_INTERRUPT
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because guest TD state is protected, exceptions in guest TDs can't be intercepted. TDX VMM doesn't need to handle exceptions. tdx_handle_exit_irqoff() handles NMI and machine check. Ignore NMI and machine check and continue guest TD execution.
For external interrupt, increment stats same to the VMX case.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
arch/x86/kvm/vmx/tdx.c +23 -0
@@ -839,6 +839,25 @@ void tdx_handle_exit_irqoff(struct kvm_vcpu *vcpu) tdexit_intr_info(vcpu)); } +static int tdx_handle_exception (struct kvm_vcpu *vcpu) +{ +u32 intr_info =tdexit_intr_info (vcpu); + +if (is_nmi (intr_info) ||is_machine_check (intr_info)) +return 1 ; + +kvm_pr_unimpl ("unexpected exception 0x%x(exit_reason 0x%llx qual 0x%lx)\n" , + intr_info, +to_tdx (vcpu)->exit_reason.full,tdexit_exit_qual (vcpu)); +return -EFAULT ; +} + +static int tdx_handle_external_interrupt (struct kvm_vcpu *vcpu) +{ + ++vcpu->stat.irq_exits; +return 1 ; +} +static int tdx_handle_triple_fault (struct kvm_vcpu *vcpu) { vcpu->run->exit_reason =KVM_EXIT_SHUTDOWN ;
@@ -1248,6 +1267,10 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath)WARN_ON_ONCE (fastpath !=EXIT_FASTPATH_NONE );switch (exit_reason.basic) { +case EXIT_REASON_EXCEPTION_NMI : +return tdx_handle_exception (vcpu); +case EXIT_REASON_EXTERNAL_INTERRUPT : +return tdx_handle_external_interrupt (vcpu);case EXIT_REASON_EPT_VIOLATION :return tdx_handle_ept_violation (vcpu);case EXIT_REASON_EPT_MISCONFIG :
Signature · 1 line
2.25.1
PATCH v8 88/103 KVM: TDX: Add a place holder for handler of TDX hypercalls (TDG.VP.VMCALL)
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The TDX module specification defines TDG.VP.VMCALL API (TDVMCALL for short) for the guest TD to call hypercall to VMM. When the guest TD issues TDG.VP.VMCALL, the guest TD exits to VMM with a new exit reason of TDVMCALL. The arguments from the guest TD and returned values from the VMM are passed in the guest registers. The guest RCX registers indicates which registers are used. Define helper functions to access those registers as ABI.
Define the TDVMCALL exit reason, which is carved out from the VMX exit reason namespace as the TDVMCALL exit from TDX guest to TDX-SEAM is really just a VM-Exit. Add a place holder to handle TDVMCALL exit.
Co-developed-by Xiaoyao Li <[email protected]>
Signed-off-by Xiaoyao Li <[email protected]>
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/include/uapi/asm/vmx.h | 4 ++- arch/x86/kvm/vmx/tdx.c | 56 ++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 13 ++++++++ 3 files changed, 71 insertions(+), 2 deletions(-)
arch/x86/include/uapi/asm/vmx.h +3 -1
@@ -93,6 +93,7 @@#define EXIT_REASON_TPAUSE 68#define EXIT_REASON_BUS_LOCK 74#define EXIT_REASON_NOTIFY 75 +#define EXIT_REASON_TDCALL 77#define VMX_EXIT_REASONS \ { EXIT_REASON_EXCEPTION_NMI, "EXCEPTION_NMI" }, \
@@ -156,7 +157,8 @@ { EXIT_REASON_UMWAIT, "UMWAIT" }, \ { EXIT_REASON_TPAUSE, "TPAUSE" }, \ { EXIT_REASON_BUS_LOCK, "BUS_LOCK" }, \ - { EXIT_REASON_NOTIFY, "NOTIFY" } + { EXIT_REASON_NOTIFY, "NOTIFY" }, \ + { EXIT_REASON_TDCALL, "TDCALL" } #define VMX_EXIT_REASON_FLAGS \ { VMX_EXIT_REASONS_FAILED_VMENTRY, "FAILED_VMENTRY" }
arch/x86/kvm/vmx/tdx.c +55 -1
86 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +13 -0
@@ -94,6 +94,19 @@ struct vcpu_tdx {struct list_head cpu_list; +union { +struct { +union { +struct { +u16 gpr_mask; +u16 xmm_mask; + }; +u32 regs_mask; + }; +u32 reserved; + }; +u64 rcx; + } tdvmcall;union tdx_exit_reason exit_reason;bool vcpu_initialized;
Signature · 1 line
2.25.1
PATCH v8 89/103 KVM: TDX: handle KVM hypercall with TDG.VP.VMCALL
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The TDX Guest-Host communication interface (GHCI) specification defines
the ABI for the guest TD to issue hypercall. It reserves vendor specific
arguments for VMM specific use. Use it as KVM hypercall and handle it.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
arch/x86/kvm/vmx/tdx.c +31 -0
@@ -906,8 +906,39 @@ static int tdx_handle_triple_fault(struct kvm_vcpu *vcpu)return 0 ; } +static int tdx_emulate_vmcall (struct kvm_vcpu *vcpu) +{ +unsigned long nr, a0, a1, a2, a3, ret; + +/* +* ABI for KVM tdvmcall argument: +* In Guest-Hypervisor Communication Interface(GHCI) specification, +* Non-zero leaf number (R10 != 0) is defined to indicate +* vendor-specific. KVM uses this for KVM hypercall. NOTE: KVM +* hypercall number starts from one. Zero isn't used for KVM hypercall +* number. +* +* R10: KVM hypercall number +* arguments: R11, R12, R13, R14. +*/ + nr =kvm_r10_read (vcpu); + a0 =kvm_r11_read (vcpu); + a1 =kvm_r12_read (vcpu); + a2 =kvm_r13_read (vcpu); + a3 =kvm_r14_read (vcpu); + + ret =__kvm_emulate_hypercall (vcpu, nr, a0, a1, a2, a3, true); + +tdvmcall_set_return_code (vcpu, ret); + +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) { +if (tdvmcall_exit_type (vcpu)) +return tdx_emulate_vmcall (vcpu); +switch (tdvmcall_leaf (vcpu)) {default :break ;
Signature · 1 line
2.25.1
PATCH v8 90/103 KVM: TDX: Handle TDX PV CPUID hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV CPUID hypercall to the KVM backend function.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/x86/kvm/vmx/tdx.c +22 -0
@@ -934,12 +934,34 @@ static int tdx_emulate_vmcall(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_emulate_cpuid (struct kvm_vcpu *vcpu) +{ +u32 eax, ebx, ecx, edx; + +/* EAX and ECX for cpuid is stored in R12 and R13. */ + eax =tdvmcall_a0_read (vcpu); + ecx =tdvmcall_a1_read (vcpu); + +kvm_cpuid (vcpu, &eax, &ebx, &ecx, &edx, false); + +tdvmcall_a0_write (vcpu, eax); +tdvmcall_a1_write (vcpu, ebx); +tdvmcall_a2_write (vcpu, ecx); +tdvmcall_a3_write (vcpu, edx); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); + +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))return tdx_emulate_vmcall (vcpu);switch (tdvmcall_leaf (vcpu)) { +case EXIT_REASON_CPUID : +return tdx_emulate_cpuid (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v8 91/103 KVM: TDX: Handle TDX PV HLT hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV HLT hypercall to the KVM backend function.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 42 +++++++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 3 +++ 2 files changed, 44 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/tdx.c +41 -1
@@ -641,7 +641,32 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu)bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) { -return pi_has_pending_interrupt (vcpu); +bool ret =pi_has_pending_interrupt (vcpu); +struct vcpu_tdx *tdx =to_tdx (vcpu); + +if (ret || vcpu->arch.mp_state !=KVM_MP_STATE_HALTED ) +return true; + +if (tdx->interrupt_disabled_hlt) +return false; + +/* +* This is for the case where the virtual interrupt is recognized, +* i.e. set in vmcs.RVI, between the STI and "HLT". KVM doesn't have +* access to RVI and the interrupt is no longer in the PID (because it +* was "recognized". It doesn't get delivered in the guest because the +* TDCALL completes before interrupts are enabled. +* +* TDX modules sets RVI while in an STI interrupt shadow. +* - TDExit(typically TDG.VP.VMCALL<HLT>) from the guest to TDX module. +* The interrupt shadow at this point is gone. +* - It knows that there is an interrupt that can be delivered +* (RVI > PPR && EFLAGS.IF=1, the other conditions of 29.2.2 don't +* matter) +* - It forwards the TDExit nevertheless, to a clueless hypervisor that +* has no way to glean either RVI or PPR. +*/ +return !!xchg (&tdx->buggy_hlt_workaround,0 ); }void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu)
@@ -954,6 +979,17 @@ static int tdx_emulate_cpuid(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_emulate_hlt (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); + +/* See tdx_protected_apic_has_interrupt() to avoid heavy seamcall */ + tdx->interrupt_disabled_hlt =tdvmcall_a0_read (vcpu);; + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return kvm_emulate_halt_noskip (vcpu); +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -962,6 +998,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)switch (tdvmcall_leaf (vcpu)) {case EXIT_REASON_CPUID :return tdx_emulate_cpuid (vcpu); +case EXIT_REASON_HLT : +return tdx_emulate_hlt (vcpu);default :break ; }
@@ -1304,6 +1342,8 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,struct kvm_vcpu *vcpu = apic->vcpu;struct vcpu_tdx *tdx =to_tdx (vcpu); +/* See comment in tdx_protected_apic_has_interrupt(). */ + tdx->buggy_hlt_workaround =1 ;/* TDX supports only posted interrupt. No lapic emulation. */ __vmx_deliver_posted_interrupt (vcpu, &tdx->pi_desc, vector); }
arch/x86/kvm/vmx/tdx.h +3 -0
@@ -115,6 +115,9 @@ struct vcpu_tdx {bool host_state_need_restore;u64 msr_host_kernel_gs_base; +bool interrupt_disabled_hlt; +unsigned int buggy_hlt_workaround; +/* * Dummy to make pmu_intel not corrupt memory. * TODO: Support PMU for TDX. Future work.
Signature · 1 line
2.25.1
PATCH v8 92/103 KVM: TDX: Handle TDX PV port io hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV port IO hypercall to the KVM backend function.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 57 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)
arch/x86/kvm/vmx/tdx.c +57 -0
@@ -990,6 +990,61 @@ static int tdx_emulate_hlt(struct kvm_vcpu *vcpu)return kvm_emulate_halt_noskip (vcpu); } +static int tdx_complete_pio_in (struct kvm_vcpu *vcpu) +{ +struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt; +unsigned long val =0 ; +int ret; + +WARN_ON (vcpu->arch.pio.count !=1 ); + + ret = ctxt->ops->pio_in_emulated (ctxt, vcpu->arch.pio.size, + vcpu->arch.pio.port, &val,1 ); +WARN_ON (!ret); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +tdvmcall_set_return_val (vcpu, val); + +return 1 ; +} + +static int tdx_emulate_io (struct kvm_vcpu *vcpu) +{ +struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt; +unsigned long val =0 ; +unsigned int port; +int size, ret; +bool write; + + ++vcpu->stat.io_exits; + + size =tdvmcall_a0_read (vcpu); + write =tdvmcall_a1_read (vcpu); + port =tdvmcall_a2_read (vcpu); + +if (size !=1 && size !=2 && size !=4 ) { +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } + +if (write) { + val =tdvmcall_a3_read (vcpu); + ret = ctxt->ops->pio_out_emulated (ctxt, size, port, &val,1 ); + +/* No need for a complete_userspace_io callback. */ + vcpu->arch.pio.count =0 ; + }else { + ret = ctxt->ops->pio_in_emulated (ctxt, size, port, &val,1 ); +if (!ret) + vcpu->arch.complete_userspace_io = tdx_complete_pio_in; +else +tdvmcall_set_return_val (vcpu, val); + } +if (ret) +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return ret; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1000,6 +1055,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_cpuid (vcpu);case EXIT_REASON_HLT :return tdx_emulate_hlt (vcpu); +case EXIT_REASON_IO_INSTRUCTION : +return tdx_emulate_io (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v8 93/103 KVM: TDX: Handle TDX PV MMIO hypercall
In reply to [email protected]From: Sean Christopherson <[email protected]>
Export kvm_io_bus_read and kvm_mmio tracepoint and wire up TDX PV MMIO hypercall to the KVM backend functions.
kvm_io_bus_read/write() searches KVM device emulated in kernel of the given MMIO address and emulates the MMIO. As TDX PV MMIO also needs it, export kvm_io_bus_read(). kvm_io_bus_write() is already exported. TDX PV MMIO emulates some of MMIO itself. To add trace point consistently with x86 kvm, export kvm_mmio tracepoint.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 114 +++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 1 + virt/kvm/kvm_main.c | 2 + 3 files changed, 117 insertions(+)
arch/x86/kvm/vmx/tdx.c +114 -0
@@ -1045,6 +1045,118 @@ static int tdx_emulate_io(struct kvm_vcpu *vcpu)return ret; } +static int tdx_complete_mmio (struct kvm_vcpu *vcpu) +{ +unsigned long val =0 ; +gpa_t gpa; +int size; + +WARN_ON (vcpu->mmio_needed !=1 ); + vcpu->mmio_needed =0 ; + +if (!vcpu->mmio_is_write) { + gpa = vcpu->mmio_fragments[0 ].gpa; + size = vcpu->mmio_fragments[0 ].len; + +memcpy (&val, vcpu->run->mmio.data, size); +tdvmcall_set_return_val (vcpu, val); +trace_kvm_mmio (KVM_TRACE_MMIO_READ , size, gpa, &val); + } +return 1 ; +} + +static inline int tdx_mmio_write (struct kvm_vcpu *vcpu,gpa_t gpa,int size, +unsigned long val) +{ +if (kvm_iodevice_write (vcpu, &vcpu->arch.apic->dev, gpa, size, &val) && +kvm_io_bus_write (vcpu,KVM_MMIO_BUS , gpa, size, &val)) +return -EOPNOTSUPP ; + +trace_kvm_mmio (KVM_TRACE_MMIO_WRITE , size, gpa, &val); +return 0 ; +} + +static inline int tdx_mmio_read (struct kvm_vcpu *vcpu,gpa_t gpa,int size) +{ +unsigned long val; + +if (kvm_iodevice_read (vcpu, &vcpu->arch.apic->dev, gpa, size, &val) && +kvm_io_bus_read (vcpu,KVM_MMIO_BUS , gpa, size, &val)) +return -EOPNOTSUPP ; + +tdvmcall_set_return_val (vcpu, val); +trace_kvm_mmio (KVM_TRACE_MMIO_READ , size, gpa, &val); +return 0 ; +} + +static int tdx_emulate_mmio (struct kvm_vcpu *vcpu) +{ +struct kvm_memory_slot *slot; +int size, write, r; +unsigned long val; +gpa_t gpa; + +WARN_ON (vcpu->mmio_needed); + + size =tdvmcall_a0_read (vcpu); + write =tdvmcall_a1_read (vcpu); + gpa =tdvmcall_a2_read (vcpu); + val = write ?tdvmcall_a3_read (vcpu) :0 ; + +if (size !=1 && size !=2 && size !=4 && size !=8 ) + goto error; +if (write !=0 && write !=1 ) + goto error; + +/* Strip the shared bit, allow MMIO with and without it set. */ + gpa = gpa & ~gfn_to_gpa (kvm_gfn_shared_mask (vcpu->kvm)); + +if (size >8u || ((gpa + size -1 ) ^ gpa) &PAGE_MASK ) + goto error; + + slot =kvm_vcpu_gfn_to_memslot (vcpu,gpa_to_gfn (gpa)); +if (slot && !(slot->flags &KVM_MEMSLOT_INVALID )) + goto error; + +if (!kvm_io_bus_write (vcpu,KVM_FAST_MMIO_BUS , gpa,0 ,NULL )) { +trace_kvm_fast_mmio (gpa); +return 1 ; + } + +if (write) + r =tdx_mmio_write (vcpu, gpa, size, val); +else + r =tdx_mmio_read (vcpu, gpa, size); +if (!r) { +/* Kernel completed device emulation. */ +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return 1 ; + } + +/* Request the device emulation to userspace device model. */ + vcpu->mmio_needed =1 ; + vcpu->mmio_is_write = write; + vcpu->arch.complete_userspace_io = tdx_complete_mmio; + + vcpu->run->mmio.phys_addr = gpa; + vcpu->run->mmio.len = size; + vcpu->run->mmio.is_write = write; + vcpu->run->exit_reason =KVM_EXIT_MMIO ; + +if (write) { +memcpy (vcpu->run->mmio.data, &val, size); + }else { + vcpu->mmio_fragments[0 ].gpa = gpa; + vcpu->mmio_fragments[0 ].len = size; +trace_kvm_mmio (KVM_TRACE_MMIO_READ_UNSATISFIED , size, gpa,NULL ); + } +return 0 ; + +error: +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1057,6 +1169,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_hlt (vcpu);case EXIT_REASON_IO_INSTRUCTION :return tdx_emulate_io (vcpu); +case EXIT_REASON_EPT_VIOLATION : +return tdx_emulate_mmio (vcpu);default :break ; }
arch/x86/kvm/x86.c +1 -0
@@ -13401,6 +13401,7 @@ bool kvm_arch_dirty_log_supported(struct kvm *kvm)EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_entry);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_exit); +EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_mmio);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_fast_mmio);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_inj_virq);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_page_fault);
virt/kvm/kvm_main.c +2 -0
@@ -2402,6 +2402,7 @@ struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn_t gfnreturn NULL ; } +EXPORT_SYMBOL_GPL (kvm_vcpu_gfn_to_memslot);bool kvm_is_visible_gfn (struct kvm *kvm,gfn_t gfn) {
@@ -5336,6 +5337,7 @@ int kvm_io_bus_read(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, gpa_t addr, r =__kvm_io_bus_read (vcpu, bus, &range, val);return r <0 ? r :0 ; } +EXPORT_SYMBOL_GPL (kvm_io_bus_read);/* Caller must hold slots_lock. */ int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,
Signature · 1 line
2.25.1
PATCH v8 94/103 KVM: TDX: Implement callbacks for MSR operations for TDX
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Implements set_msr/get_msr/has_emulated_msr methods for TDX to handle hypercall from guest TD for paravirtualized rdmsr and wrmsr. The TDX module virtualizes MSRs. For some MSRs, it injects #VE to the guest TD upon RDMSR or WRMSR. The exact list of such MSRs are defined in the spec.
Upon #VE, the guest TD may execute hypercalls, TDG.VP.VMCALL<INSTRUCTION.RDMSR> and TDG.VP.VMCALL<INSTRUCTION.WRMSR>, which are defined in GHCI (Guest-Host Communication Interface) so that the host VMM (e.g. KVM) can virtualizes the MSRs.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/main.c | 34 +++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 68 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 6 ++++ 3 files changed, 105 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/main.c +31 -3
@@ -205,6 +205,34 @@ static void vt_handle_exit_irqoff(struct kvm_vcpu *vcpu)vmx_handle_exit_irqoff (vcpu); } +static int vt_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr_info) +{ +if (unlikely (is_td_vcpu (vcpu))) +return tdx_set_msr (vcpu, msr_info); + +return vmx_set_msr (vcpu, msr_info); +} + +/* +* The kvm parameter can be NULL (module initialization, or invocation before +* VM creation). Be sure to check the kvm parameter before using it. +*/ +static bool vt_has_emulated_msr (struct kvm *kvm,u32 index) +{ +if (kvm &&is_td (kvm)) +return tdx_is_emulated_msr (index, true); + +return vmx_has_emulated_msr (kvm, index); +} + +static int vt_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr_info) +{ +if (unlikely (is_td_vcpu (vcpu))) +return tdx_get_msr (vcpu, msr_info); + +return vmx_get_msr (vcpu, msr_info); +} +static void vt_apicv_post_state_restore (struct kvm_vcpu *vcpu) {struct pi_desc *pi =vcpu_to_pi_desc (vcpu);
@@ -439,7 +467,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .hardware_enable = vt_hardware_enable, .hardware_disable = vt_hardware_disable, - .has_emulated_msr = vmx_has_emulated_msr, + .has_emulated_msr = vt_has_emulated_msr, .is_vm_type_supported = vt_is_vm_type_supported, .vm_size = sizeof(struct kvm_vmx),
@@ -459,8 +487,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .update_exception_bitmap = vmx_update_exception_bitmap, .get_msr_feature = vmx_get_msr_feature, - .get_msr = vmx_get_msr, - .set_msr = vmx_set_msr, + .get_msr = vt_get_msr, + .set_msr = vt_set_msr, .get_segment_base = vmx_get_segment_base, .get_segment = vmx_get_segment, .set_segment = vmx_set_segment,
arch/x86/kvm/vmx/tdx.c +68 -0
74 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +6 -0
@@ -159,6 +159,9 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,void tdx_inject_nmi (struct kvm_vcpu *vcpu);void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason,u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code); +bool tdx_is_emulated_msr (u32 index,bool write); +int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr); +int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -197,6 +200,9 @@ static inline void tdx_inject_nmi(struct kvm_vcpu *vcpu) {}static inline void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason,u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code) {} +static inline bool tdx_is_emulated_msr (u32 index,bool write) {return false; } +static inline int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; } +static inline int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; }static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v8 95/103 KVM: TDX: Handle TDX PV rdmsr/wrmsr hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV rdmsr/wrmsr hypercall to the KVM backend function.
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
arch/x86/kvm/vmx/tdx.c +39 -0
@@ -1157,6 +1157,41 @@ static int tdx_emulate_mmio(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_emulate_rdmsr (struct kvm_vcpu *vcpu) +{ +u32 index =tdvmcall_a0_read (vcpu); +u64 data; + +if (!kvm_msr_allowed (vcpu, index,KVM_MSR_FILTER_READ ) || +kvm_get_msr (vcpu, index, &data)) { +trace_kvm_msr_read_ex (index); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } +trace_kvm_msr_read (index, data); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +tdvmcall_set_return_val (vcpu, data); +return 1 ; +} + +static int tdx_emulate_wrmsr (struct kvm_vcpu *vcpu) +{ +u32 index =tdvmcall_a0_read (vcpu); +u64 data =tdvmcall_a1_read (vcpu); + +if (!kvm_msr_allowed (vcpu, index,KVM_MSR_FILTER_READ ) || +kvm_set_msr (vcpu, index, data)) { +trace_kvm_msr_write_ex (index, data); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } + +trace_kvm_msr_write (index, data); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1171,6 +1206,10 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_io (vcpu);case EXIT_REASON_EPT_VIOLATION :return tdx_emulate_mmio (vcpu); +case EXIT_REASON_MSR_READ : +return tdx_emulate_rdmsr (vcpu); +case EXIT_REASON_MSR_WRITE : +return tdx_emulate_wrmsr (vcpu);default :break ; }
Signature · 1 line
2.25.1
Re: [PATCH v8 095/103] KVM: TDX: Handle TDX PV rdmsr/wrmsr hypercall
Quoted 33 lines · On Sun, Aug 7, 2022 at 3:03 PM <[email protected]> wrote:
From: Isaku Yamahata <[email protected]>
Wire up TDX PV rdmsr/wrmsr hypercall to the KVM backend function.
Signed-off-by Isaku Yamahata <[email protected]>Reviewed-by Paolo Bonzini <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)arch/x86/kvm/vmx/tdx.c +14
-0@@ -1157,6 +1157,41 @@ static int tdx_emulate_mmio(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_emulate_rdmsr (struct kvm_vcpu *vcpu) +{ +u32 index =tdvmcall_a0_read (vcpu); +u64 data; + +if (!kvm_msr_allowed (vcpu, index,KVM_MSR_FILTER_READ ) || +kvm_get_msr (vcpu, index, &data)) { +trace_kvm_msr_read_ex (index); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } +trace_kvm_msr_read (index, data); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS );
arch/x86/kvm/vmx/tdx.c static int tdx_emulate_mmio(struct kvm_vcpu *vcpu) · line 1183
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+tdvmcall_set_return_val (vcpu, data); +return 1 ; +} + +static int tdx_emulate_wrmsr (struct kvm_vcpu *vcpu) +{ +u32 index =tdvmcall_a0_read (vcpu); +u64 data =tdvmcall_a1_read (vcpu); + + if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_READ) ||
^
This should be KVM_MSR_FILTER_WRITE
Quoted 28 lines
arch/x86/kvm/vmx/tdx.c +15
-1+ kvm_set_msr(vcpu, index, data)) { +trace_kvm_msr_write_ex (index, data); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } + +trace_kvm_msr_write (index, data); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))@@ -1171,6 +1206,10 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_io (vcpu);case EXIT_REASON_EPT_VIOLATION :return tdx_emulate_mmio (vcpu); +case EXIT_REASON_MSR_READ : +return tdx_emulate_rdmsr (vcpu); +case EXIT_REASON_MSR_WRITE : +return tdx_emulate_wrmsr (vcpu);default :break ; } --2.25.1
Re: [PATCH v8 095/103] KVM: TDX: Handle TDX PV rdmsr/wrmsr hypercall
Quoted 9 lines · Sagi Shahar
arch/x86/kvm/vmx/tdx.c static int tdx_emulate_mmio(struct kvm_vcpu *vcpu) · line 1183
Quoted 6 lines · On Sun, Aug 7, 2022 at 3:03 PM <[email protected]> wrote:
arch/x86/kvm/vmx/tdx.c +6
-0+static int tdx_emulate_wrmsr (struct kvm_vcpu *vcpu) +{ +u32 index =tdvmcall_a0_read (vcpu); +u64 data =tdvmcall_a1_read (vcpu); + + if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_READ) ||^This should be
KVM_MSR_FILTER_WRITE
Oops. Thanks. I'll fix it.
Signature · 1 line
Isaku Yamahata <[email protected]>
PATCH v8 96/103 KVM: TDX: Handle TDX PV report fatal error hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV report fatal error hypercall to KVM_SYSTEM_EVENT_CRASH KVM exit event.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 20 ++++++++++++++++++++ include/uapi/linux/kvm.h | 1 + 2 files changed, 21 insertions(+)
arch/x86/kvm/vmx/tdx.c +20 -0
@@ -1192,6 +1192,24 @@ static int tdx_emulate_wrmsr(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_report_fatal_error (struct kvm_vcpu *vcpu) +{ +/* +* Exit to userspace device model for teardown. +* Because guest TD is already panicing, returning an error to guerst TD +* doesn't make sense. No argument check is done. +*/ + + vcpu->run->exit_reason =KVM_EXIT_SYSTEM_EVENT ; + vcpu->run->system_event.type =KVM_SYSTEM_EVENT_TDX ; + vcpu->run->system_event.ndata =3 ; + vcpu->run->system_event.data[0 ] =TDG_VP_VMCALL_REPORT_FATAL_ERROR ; + vcpu->run->system_event.data[1 ] =tdvmcall_a0_read (vcpu); + vcpu->run->system_event.data[2 ] =tdvmcall_a1_read (vcpu); + +return 0 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1210,6 +1228,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_rdmsr (vcpu);case EXIT_REASON_MSR_WRITE :return tdx_emulate_wrmsr (vcpu); +case TDG_VP_VMCALL_REPORT_FATAL_ERROR : +return tdx_report_fatal_error (vcpu);default :break ; }
include/uapi/linux/kvm.h +1 -0
@@ -448,6 +448,7 @@ struct kvm_run {#define KVM_SYSTEM_EVENT_WAKEUP 4#define KVM_SYSTEM_EVENT_SUSPEND 5#define KVM_SYSTEM_EVENT_SEV_TERM 6 +#define KVM_SYSTEM_EVENT_TDX 7__u32 type;__u32 ndata;union {
Signature · 1 line
2.25.1
PATCH v8 97/103 KVM: TDX: Handle TDX PV map_gpa hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV map_gpa hypercall to the kvm/mmu backend.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
arch/x86/kvm/vmx/tdx.c +33 -0
@@ -1210,6 +1210,37 @@ static int tdx_report_fatal_error(struct kvm_vcpu *vcpu)return 0 ; } +static int tdx_map_gpa (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm; +gpa_t gpa =tdvmcall_a0_read (vcpu); +gpa_t size =tdvmcall_a1_read (vcpu); +gpa_t end = gpa + size; +gfn_t s =gpa_to_gfn (gpa) & ~kvm_gfn_shared_mask (kvm); +gfn_t e =gpa_to_gfn (end) & ~kvm_gfn_shared_mask (kvm); +bool map_private =kvm_is_private_gpa (kvm, gpa); +int ret; + +if (!IS_ALIGNED (gpa,4096 ) || !IS_ALIGNED (size,4096 ) || + end < gpa || + end >kvm_gfn_shared_mask (kvm) << (PAGE_SHIFT +1 ) || +kvm_is_private_gpa (kvm, gpa) !=kvm_is_private_gpa (kvm, end)) { +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } + + ret =kvm_mmu_map_gpa (vcpu, &s, e, map_private); +if (ret == -EAGAIN ) { +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_RETRY ); +tdvmcall_set_return_val (vcpu,gfn_to_gpa (s)); + }else if (ret) +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +else +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); + +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1230,6 +1261,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_wrmsr (vcpu);case TDG_VP_VMCALL_REPORT_FATAL_ERROR :return tdx_report_fatal_error (vcpu); +case TDG_VP_VMCALL_MAP_GPA : +return tdx_map_gpa (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v8 98/103 KVM: TDX: Handle TDG.VP.VMCALL<GetTdVmCallInfo> hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Implement TDG.VP.VMCALL<GetTdVmCallInfo> hypercall. If the input value is zero, return success code and zero in output registers.
TDG.VP.VMCALL<GetTdVmCallInfo> hypercall is a subleaf of TDG.VP.VMCALL to enumerate which TDG.VP.VMCALL sub leaves are supported. This hypercall is for future enhancement of the Guest-Host-Communication Interface (GHCI) specification. The GHCI version of 344426-001US defines it to require input R12 to be zero and to return zero in output registers, R11, R12, R13, and R14 so that guest TD enumerates no enhancement.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/tdx.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
arch/x86/kvm/vmx/tdx.c +16 -0
@@ -1192,6 +1192,20 @@ static int tdx_emulate_wrmsr(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_get_td_vm_call_info (struct kvm_vcpu *vcpu) +{ +if (tdvmcall_a0_read (vcpu)) +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +else { +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +kvm_r11_write (vcpu,0 ); +tdvmcall_a0_write (vcpu,0 ); +tdvmcall_a1_write (vcpu,0 ); +tdvmcall_a2_write (vcpu,0 ); + } +return 1 ; +} +static int tdx_report_fatal_error (struct kvm_vcpu *vcpu) {/*
@@ -1259,6 +1273,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_rdmsr (vcpu);case EXIT_REASON_MSR_WRITE :return tdx_emulate_wrmsr (vcpu); +case TDG_VP_VMCALL_GET_TD_VM_CALL_INFO : +return tdx_get_td_vm_call_info (vcpu);case TDG_VP_VMCALL_REPORT_FATAL_ERROR :return tdx_report_fatal_error (vcpu);case TDG_VP_VMCALL_MAP_GPA :
Signature · 1 line
2.25.1
PATCH v8 99/103 KVM: TDX: Silently discard SMI request
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX doesn't support system-management mode (SMM) and system-management interrupt (SMI) in guest TDs. Because guest state (vcpu state, memory state) is protected, it must go through the TDX module APIs to change guest state, injecting SMI and changing vcpu mode into SMM. The TDX module doesn't provide a way for VMM to inject SMI into guest TD and a way for VMM to switch guest vcpu mode into SMM.
We have two options in KVM when handling SMM or SMI in the guest TD or the device model (e.g. QEMU): 1) silently ignore the request or 2) return a meaningful error.
For simplicity, we implemented the option 1).
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/lapic.c | 7 +++++-- arch/x86/kvm/vmx/main.c | 43 ++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 27 ++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 8 +++++++ arch/x86/kvm/x86.c | 3 ++- 5 files changed, 81 insertions(+), 7 deletions(-)
arch/x86/kvm/lapic.c +5 -2
@@ -1171,8 +1171,11 @@ static int __apic_accept_irq(struct kvm_lapic *apic, int delivery_mode,case APIC_DM_SMI : result =1 ; -kvm_make_request (KVM_REQ_SMI , vcpu); -kvm_vcpu_kick (vcpu); +if (static_call (kvm_x86_has_emulated_msr)(vcpu->kvm, +MSR_IA32_SMBASE )) { +kvm_make_request (KVM_REQ_SMI , vcpu); +kvm_vcpu_kick (vcpu); + }break ;case APIC_DM_NMI :
arch/x86/kvm/vmx/main.c +39 -4
55 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.c +27 -0
@@ -1800,6 +1800,33 @@ int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)return 1 ; } +int tdx_smi_allowed (struct kvm_vcpu *vcpu,bool for_injection) +{ +/* SMI isn't supported for TDX. */ +WARN_ON_ONCE (1 ); +return false; +} + +int tdx_enter_smm (struct kvm_vcpu *vcpu,char *smstate) +{ +/* smi_allowed() is always false for TDX as above. */ +WARN_ON_ONCE (1 ); +return 0 ; +} + +int tdx_leave_smm (struct kvm_vcpu *vcpu,const char *smstate) +{ +WARN_ON_ONCE (1 ); +return 0 ; +} + +void tdx_enable_smi_window (struct kvm_vcpu *vcpu) +{ +/* SMI isn't supported for TDX. Silently discard SMI request. */ +WARN_ON_ONCE (1 ); + vcpu->arch.smi_pending = false; +} +int tdx_dev_ioctl (void __user *argp) {struct kvm_tdx_capabilities __user *user_caps;
arch/x86/kvm/vmx/x86_ops.h +8 -0
@@ -162,6 +162,10 @@ void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason,bool tdx_is_emulated_msr (u32 index,bool write);int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr);int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr); +int tdx_smi_allowed (struct kvm_vcpu *vcpu,bool for_injection); +int tdx_enter_smm (struct kvm_vcpu *vcpu,char *smstate); +int tdx_leave_smm (struct kvm_vcpu *vcpu,const char *smstate); +void tdx_enable_smi_window (struct kvm_vcpu *vcpu);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -203,6 +207,10 @@ static inline void tdx_get_exit_info(static inline bool tdx_is_emulated_msr (u32 index,bool write) {return false; }static inline int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; }static inline int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; } +static inline int tdx_smi_allowed (struct kvm_vcpu *vcpu,bool for_injection) {return false; } +static inline int tdx_enter_smm (struct kvm_vcpu *vcpu,char *smstate) {return 0 ; } +static inline int tdx_leave_smm (struct kvm_vcpu *vcpu,const char *smstate) {return 0 ; } +static inline void tdx_enable_smi_window (struct kvm_vcpu *vcpu) {}static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
arch/x86/kvm/x86.c +2 -1
@@ -4893,7 +4893,8 @@ static int kvm_vcpu_ioctl_nmi(struct kvm_vcpu *vcpu)static int kvm_vcpu_ioctl_smi (struct kvm_vcpu *vcpu) { -kvm_make_request (KVM_REQ_SMI , vcpu); +if (static_call (kvm_x86_has_emulated_msr)(vcpu->kvm,MSR_IA32_SMBASE )) +kvm_make_request (KVM_REQ_SMI , vcpu);return 0 ; }
Signature · 1 line
2.25.1
PATCH v8 100/103 KVM: TDX: Silently ignore INIT/SIPI
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The TDX module API doesn't provide API for VMM to inject INIT IPI and SIPI. Instead it defines the different protocols to boot application processors. Ignore INIT and SIPI events for the TDX guest.
There are two options. 1) (silently) ignore INIT/SIPI request or 2) return error to guest TDs somehow. Given that TDX guest is paravirtualized to boot AP, the option 1 is chosen for simplicity.
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/lapic.c | 16 +++++++++++----- arch/x86/kvm/svm/svm.c | 1 + arch/x86/kvm/vmx/main.c | 22 +++++++++++++++++++++- 5 files changed, 36 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -136,6 +136,7 @@ KVM_X86_OP_OPTIONAL(migrate_timers)KVM_X86_OP (msr_filter_changed) KVM_X86_OP(complete_emulated_msr) KVM_X86_OP(vcpu_deliver_sipi_vector) +KVM_X86_OP(vcpu_deliver_init) KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);KVM_X86_OP (check_processor_compatibility)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1696,6 +1696,7 @@ struct kvm_x86_ops {int (*complete_emulated_msr)(struct kvm_vcpu *vcpu,int err);void (*vcpu_deliver_sipi_vector)(struct kvm_vcpu *vcpu,u8 vector); +void (*vcpu_deliver_init)(struct kvm_vcpu *vcpu);/* * Returns vCPU specific APICv inhibit reasons
@@ -1905,6 +1906,7 @@ int kvm_emulate_wbinvd(struct kvm_vcpu *vcpu);void kvm_get_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var,int seg);int kvm_load_segment_descriptor (struct kvm_vcpu *vcpu,u16 selector,int seg);void kvm_vcpu_deliver_sipi_vector (struct kvm_vcpu *vcpu,u8 vector); +void kvm_vcpu_deliver_init (struct kvm_vcpu *vcpu);int kvm_task_switch (struct kvm_vcpu *vcpu,u16 tss_selector,int idt_index,int reason,bool has_error_code,u32 error_code);
arch/x86/kvm/lapic.c +11 -5
@@ -3033,6 +3033,16 @@ int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len)return 0 ; } +void kvm_vcpu_deliver_init (struct kvm_vcpu *vcpu) +{ +kvm_vcpu_reset (vcpu, true); +if (kvm_vcpu_is_bsp (vcpu)) + vcpu->arch.mp_state =KVM_MP_STATE_RUNNABLE ; +else + vcpu->arch.mp_state =KVM_MP_STATE_INIT_RECEIVED ; +} +EXPORT_SYMBOL_GPL (kvm_vcpu_deliver_init); +int kvm_apic_accept_events (struct kvm_vcpu *vcpu) {struct kvm_lapic *apic = vcpu->arch.apic;
@@ -3080,11 +3090,7 @@ int kvm_apic_accept_events(struct kvm_vcpu *vcpu)if (test_bit (KVM_APIC_INIT , &pe)) {clear_bit (KVM_APIC_INIT , &apic->pending_events); -kvm_vcpu_reset (vcpu, true); -if (kvm_vcpu_is_bsp (apic->vcpu)) - vcpu->arch.mp_state =KVM_MP_STATE_RUNNABLE ; -else - vcpu->arch.mp_state =KVM_MP_STATE_INIT_RECEIVED ; +static_call (kvm_x86_vcpu_deliver_init)(vcpu); }if (test_bit (KVM_APIC_SIPI , &pe)) {clear_bit (KVM_APIC_SIPI , &apic->pending_events);
arch/x86/kvm/svm/svm.c +1 -0
@@ -4870,6 +4870,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .complete_emulated_msr = svm_complete_emulated_msr, .vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector, + .vcpu_deliver_init = kvm_vcpu_deliver_init, .vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons, };
arch/x86/kvm/vmx/main.c +21 -1
@@ -295,6 +295,25 @@ static void vt_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,vmx_deliver_interrupt (apic, delivery_mode, trig_mode, vector); } +static void vt_vcpu_deliver_sipi_vector (struct kvm_vcpu *vcpu,u8 vector) +{ +if (is_td_vcpu (vcpu)) +return ; + +kvm_vcpu_deliver_sipi_vector (vcpu, vector); +} + +static void vt_vcpu_deliver_init (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +/* TDX doesn't support INIT. Ignore INIT event */ + vcpu->arch.mp_state =KVM_MP_STATE_RUNNABLE ; +return ; + } + +kvm_vcpu_deliver_init (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -635,7 +654,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .msr_filter_changed = vmx_msr_filter_changed, .complete_emulated_msr = kvm_complete_insn_gp, - .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector, + .vcpu_deliver_sipi_vector = vt_vcpu_deliver_sipi_vector, + .vcpu_deliver_init = vt_vcpu_deliver_init, .dev_mem_enc_ioctl = tdx_dev_ioctl, .mem_enc_ioctl = vt_mem_enc_ioctl,
Signature · 1 line
2.25.1
PATCH v8 101/103 KVM: TDX: Add methods to ignore accesses to CPU state
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX protects TDX guest state from VMM. Implements to access methods for TDX guest state to ignore them or return zero.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
--- arch/x86/kvm/vmx/main.c | 463 +++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 55 ++++- arch/x86/kvm/vmx/x86_ops.h | 17 ++ 3 files changed, 490 insertions(+), 45 deletions(-)
arch/x86/kvm/vmx/main.c +420 -43
@@ -268,6 +268,46 @@ static void vt_enable_smi_window(struct kvm_vcpu *vcpu)vmx_enable_smi_window (vcpu); } +static bool vt_can_emulate_instruction (struct kvm_vcpu *vcpu,int emul_type, +void *insn,int insn_len) +{ +if (is_td_vcpu (vcpu)) +return false; + +return vmx_can_emulate_instruction (vcpu, emul_type, insn, insn_len); +} + +static int vt_check_intercept (struct kvm_vcpu *vcpu, +struct x86_instruction_info *info, +enum x86_intercept_stage stage, +struct x86_exception *exception) +{ +/* +* This call back is triggered by the x86 instruction emulator. TDX +* doesn't allow guest memory inspection. +*/ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return X86EMUL_UNHANDLEABLE ; + +return vmx_check_intercept (vcpu, info, stage, exception); +} + +static bool vt_apic_init_signal_blocked (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return true; + +return vmx_apic_init_signal_blocked (vcpu); +} + +static void vt_set_virtual_apic_mode (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_set_virtual_apic_mode (vcpu); + +return vmx_set_virtual_apic_mode (vcpu); +} +static void vt_apicv_post_state_restore (struct kvm_vcpu *vcpu) {struct pi_desc *pi =vcpu_to_pi_desc (vcpu);
@@ -275,6 +315,31 @@ static void vt_apicv_post_state_restore(struct kvm_vcpu *vcpu)memset (pi->pir,0 ,sizeof (pi->pir)); } +static void vt_hwapic_irr_update (struct kvm_vcpu *vcpu,int max_irr) +{ +if (is_td_vcpu (vcpu)) +return ; + +return vmx_hwapic_irr_update (vcpu, max_irr); +} + +static void vt_hwapic_isr_update (int max_isr) +{ +if (is_td_vcpu (kvm_get_running_vcpu ())) +return ; + +return vmx_hwapic_isr_update (max_isr); +} + +static bool vt_guest_apic_has_interrupt (struct kvm_vcpu *vcpu) +{ +/* TDX doesn't support L2 at the moment. */ +if (WARN_ON_ONCE (is_td_vcpu (vcpu))) +return false; + +return vmx_guest_apic_has_interrupt (vcpu); +} +static int vt_sync_pir_to_irr (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -314,6 +379,177 @@ static void vt_vcpu_deliver_init(struct kvm_vcpu *vcpu)kvm_vcpu_deliver_init (vcpu); } +static void vt_vcpu_after_set_cpuid (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +return vmx_vcpu_after_set_cpuid (vcpu); +} + +static void vt_update_exception_bitmap (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_update_exception_bitmap (vcpu); +} + +static u64 vt_get_segment_base (struct kvm_vcpu *vcpu,int seg) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return tdx_get_segment_base (vcpu, seg); + +return vmx_get_segment_base (vcpu, seg); +} + +static void vt_get_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var, +int seg) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return tdx_get_segment (vcpu, var, seg); + +vmx_get_segment (vcpu, var, seg); +} + +static void vt_set_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var, +int seg) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_set_segment (vcpu, var, seg); +} + +static int vt_get_cpl (struct kvm_vcpu *vcpu) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return tdx_get_cpl (vcpu); + +return vmx_get_cpl (vcpu); +} + +static void vt_get_cs_db_l_bits (struct kvm_vcpu *vcpu,int *db,int *l) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_get_cs_db_l_bits (vcpu, db, l); +} + +static void vt_set_cr0 (struct kvm_vcpu *vcpu,unsigned long cr0) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_cr0 (vcpu, cr0); +} + +static void vt_set_cr4 (struct kvm_vcpu *vcpu,unsigned long cr4) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_cr4 (vcpu, cr4); +} + +static int vt_set_efer (struct kvm_vcpu *vcpu,u64 efer) +{ +if (is_td_vcpu (vcpu)) +return 0 ; + +return vmx_set_efer (vcpu, efer); +} + +static void vt_get_idt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) { +memset (dt,0 ,sizeof (*dt)); +return ; + } + +vmx_get_idt (vcpu, dt); +} + +static void vt_set_idt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_set_idt (vcpu, dt); +} + +static void vt_get_gdt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) { +memset (dt,0 ,sizeof (*dt)); +return ; + } + +vmx_get_gdt (vcpu, dt); +} + +static void vt_set_gdt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_set_gdt (vcpu, dt); +} + +static void vt_set_dr7 (struct kvm_vcpu *vcpu,unsigned long val) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_dr7 (vcpu, val); +} + +static void vt_sync_dirty_debug_regs (struct kvm_vcpu *vcpu) +{ +/* +* MOV-DR exiting is always cleared for TD guest, even in debug mode. +* Thus KVM_DEBUGREG_WONT_EXIT can never be set and it should never +* reach here for TD vcpu. +*/ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_sync_dirty_debug_regs (vcpu); +} + +static void vt_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg) +{ +if (is_td_vcpu (vcpu)) +return tdx_cache_reg (vcpu, reg); + +return vmx_cache_reg (vcpu, reg); +} + +static unsigned long vt_get_rflags (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_rflags (vcpu); + +return vmx_get_rflags (vcpu); +} + +static void vt_set_rflags (struct kvm_vcpu *vcpu,unsigned long rflags) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_rflags (vcpu, rflags); +} + +static bool vt_get_if_flag (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return false; + +return vmx_get_if_flag (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -447,6 +683,15 @@ static u32 vt_get_interrupt_shadow(struct kvm_vcpu *vcpu)return vmx_get_interrupt_shadow (vcpu); } +static void vt_patch_hypercall (struct kvm_vcpu *vcpu, +unsigned char *hypercall) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_patch_hypercall (vcpu, hypercall); +} +static void vt_inject_irq (struct kvm_vcpu *vcpu,bool reinjected) {if (is_td_vcpu (vcpu))
@@ -455,6 +700,14 @@ static void vt_inject_irq(struct kvm_vcpu *vcpu, bool reinjected)vmx_inject_irq (vcpu, reinjected); } +static void vt_queue_exception (struct kvm_vcpu *vcpu) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_queue_exception (vcpu); +} +static void vt_cancel_injection (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -487,6 +740,130 @@ static void vt_request_immediate_exit(struct kvm_vcpu *vcpu)vmx_request_immediate_exit (vcpu); } +static void vt_update_cr8_intercept (struct kvm_vcpu *vcpu,int tpr,int irr) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_update_cr8_intercept (vcpu, tpr, irr); +} + +static void vt_set_apic_access_page_addr (struct kvm_vcpu *vcpu) +{ +if (WARN_ON_ONCE (is_td_vcpu (vcpu))) +return ; + +vmx_set_apic_access_page_addr (vcpu); +} + +static void vt_refresh_apicv_exec_ctrl (struct kvm_vcpu *vcpu) +{ +if (WARN_ON_ONCE (is_td_vcpu (vcpu))) +return ; + +vmx_refresh_apicv_exec_ctrl (vcpu); +} + +static void vt_load_eoi_exitmap (struct kvm_vcpu *vcpu,u64 *eoi_exit_bitmap) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_load_eoi_exitmap (vcpu, eoi_exit_bitmap); +} + +static int vt_set_tss_addr (struct kvm *kvm,unsigned int addr) +{ +if (is_td (kvm)) +return 0 ; + +return vmx_set_tss_addr (kvm, addr); +} + +static int vt_set_identity_map_addr (struct kvm *kvm,u64 ident_addr) +{ +if (is_td (kvm)) +return 0 ; + +return vmx_set_identity_map_addr (kvm, ident_addr); +} + +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +{ +if (is_td_vcpu (vcpu)) { +if (is_mmio) +return MTRR_TYPE_UNCACHABLE <<VMX_EPT_MT_EPTE_SHIFT ; +return MTRR_TYPE_WRBACK <<VMX_EPT_MT_EPTE_SHIFT ; + } + +return vmx_get_mt_mask (vcpu, gfn, is_mmio); +} + +static u64 vt_get_l2_tsc_offset (struct kvm_vcpu *vcpu) +{ +/* TDX doesn't support L2 guest at the moment. */ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return 0 ; + +return vmx_get_l2_tsc_offset (vcpu); +} + +static u64 vt_get_l2_tsc_multiplier (struct kvm_vcpu *vcpu) +{ +/* TDX doesn't support L2 guest at the moment. */ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return 0 ; + +return vmx_get_l2_tsc_multiplier (vcpu); +} + +static void vt_write_tsc_offset (struct kvm_vcpu *vcpu,u64 offset) +{ +/* In TDX, tsc offset can't be changed. */ +if (is_td_vcpu (vcpu)) +return ; + +vmx_write_tsc_offset (vcpu, offset); +} + +static void vt_write_tsc_multiplier (struct kvm_vcpu *vcpu,u64 multiplier) +{ +/* In TDX, tsc multiplier can't be changed. */ +if (is_td_vcpu (vcpu)) +return ; + +vmx_write_tsc_multiplier (vcpu, multiplier); +} + +static void vt_update_cpu_dirty_logging (struct kvm_vcpu *vcpu) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_update_cpu_dirty_logging (vcpu); +} + +#ifdef CONFIG_X86_64 +static int vt_set_hv_timer (struct kvm_vcpu *vcpu,u64 guest_deadline_tsc, +bool *expired) +{ +/* VMX-preemption timer isn't available for TDX. */ +if (is_td_vcpu (vcpu)) +return -EINVAL ; + +return vmx_set_hv_timer (vcpu, guest_deadline_tsc, expired); +} + +static void vt_cancel_hv_timer (struct kvm_vcpu *vcpu) +{ +/* VMX-preemption timer can't be set. Set vt_set_hv_timer(). */ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_cancel_hv_timer (vcpu); +} +#endif +static void vt_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason,u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code) {
@@ -539,29 +916,29 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_load = vt_vcpu_load, .vcpu_put = vt_vcpu_put, - .update_exception_bitmap = vmx_update_exception_bitmap, + .update_exception_bitmap = vt_update_exception_bitmap, .get_msr_feature = vmx_get_msr_feature, .get_msr = vt_get_msr, .set_msr = vt_set_msr, - .get_segment_base = vmx_get_segment_base, - .get_segment = vmx_get_segment, - .set_segment = vmx_set_segment, - .get_cpl = vmx_get_cpl, - .get_cs_db_l_bits = vmx_get_cs_db_l_bits, - .set_cr0 = vmx_set_cr0, + .get_segment_base = vt_get_segment_base, + .get_segment = vt_get_segment, + .set_segment = vt_set_segment, + .get_cpl = vt_get_cpl, + .get_cs_db_l_bits = vt_get_cs_db_l_bits, + .set_cr0 = vt_set_cr0, .is_valid_cr4 = vmx_is_valid_cr4, - .set_cr4 = vmx_set_cr4, - .set_efer = vmx_set_efer, - .get_idt = vmx_get_idt, - .set_idt = vmx_set_idt, - .get_gdt = vmx_get_gdt, - .set_gdt = vmx_set_gdt, - .set_dr7 = vmx_set_dr7, - .sync_dirty_debug_regs = vmx_sync_dirty_debug_regs, - .cache_reg = vmx_cache_reg, - .get_rflags = vmx_get_rflags, - .set_rflags = vmx_set_rflags, - .get_if_flag = vmx_get_if_flag, + .set_cr4 = vt_set_cr4, + .set_efer = vt_set_efer, + .get_idt = vt_get_idt, + .set_idt = vt_set_idt, + .get_gdt = vt_get_gdt, + .set_gdt = vt_set_gdt, + .set_dr7 = vt_set_dr7, + .sync_dirty_debug_regs = vt_sync_dirty_debug_regs, + .cache_reg = vt_cache_reg, + .get_rflags = vt_get_rflags, + .set_rflags = vt_set_rflags, + .get_if_flag = vt_get_if_flag, .flush_tlb_all = vt_flush_tlb_all, .flush_tlb_current = vt_flush_tlb_current,
@@ -577,10 +954,10 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .update_emulated_instruction = vmx_update_emulated_instruction, .set_interrupt_shadow = vt_set_interrupt_shadow, .get_interrupt_shadow = vt_get_interrupt_shadow, - .patch_hypercall = vmx_patch_hypercall, + .patch_hypercall = vt_patch_hypercall, .inject_irq = vt_inject_irq, .inject_nmi = vt_inject_nmi, - .queue_exception = vmx_queue_exception, + .queue_exception = vt_queue_exception, .cancel_injection = vt_cancel_injection, .interrupt_allowed = vt_interrupt_allowed, .nmi_allowed = vt_nmi_allowed,
@@ -588,39 +965,39 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_nmi_mask = vt_set_nmi_mask, .enable_nmi_window = vt_enable_nmi_window, .enable_irq_window = vt_enable_irq_window, - .update_cr8_intercept = vmx_update_cr8_intercept, - .set_virtual_apic_mode = vmx_set_virtual_apic_mode, - .set_apic_access_page_addr = vmx_set_apic_access_page_addr, - .refresh_apicv_exec_ctrl = vmx_refresh_apicv_exec_ctrl, - .load_eoi_exitmap = vmx_load_eoi_exitmap, + .update_cr8_intercept = vt_update_cr8_intercept, + .set_virtual_apic_mode = vt_set_virtual_apic_mode, + .set_apic_access_page_addr = vt_set_apic_access_page_addr, + .refresh_apicv_exec_ctrl = vt_refresh_apicv_exec_ctrl, + .load_eoi_exitmap = vt_load_eoi_exitmap, .apicv_post_state_restore = vt_apicv_post_state_restore, .check_apicv_inhibit_reasons = vmx_check_apicv_inhibit_reasons, - .hwapic_irr_update = vmx_hwapic_irr_update, - .hwapic_isr_update = vmx_hwapic_isr_update, - .guest_apic_has_interrupt = vmx_guest_apic_has_interrupt, + .hwapic_irr_update = vt_hwapic_irr_update, + .hwapic_isr_update = vt_hwapic_isr_update, + .guest_apic_has_interrupt = vt_guest_apic_has_interrupt, .sync_pir_to_irr = vt_sync_pir_to_irr, .deliver_interrupt = vt_deliver_interrupt, .dy_apicv_has_pending_interrupt = pi_has_pending_interrupt, .protected_apic_has_interrupt = vt_protected_apic_has_interrupt, - .set_tss_addr = vmx_set_tss_addr, - .set_identity_map_addr = vmx_set_identity_map_addr, - .get_mt_mask = vmx_get_mt_mask, + .set_tss_addr = vt_set_tss_addr, + .set_identity_map_addr = vt_set_identity_map_addr, + .get_mt_mask = vt_get_mt_mask, .get_exit_info = vt_get_exit_info, - .vcpu_after_set_cpuid = vmx_vcpu_after_set_cpuid, + .vcpu_after_set_cpuid = vt_vcpu_after_set_cpuid, .has_wbinvd_exit = cpu_has_vmx_wbinvd_exit, - .get_l2_tsc_offset = vmx_get_l2_tsc_offset, - .get_l2_tsc_multiplier = vmx_get_l2_tsc_multiplier, - .write_tsc_offset = vmx_write_tsc_offset, - .write_tsc_multiplier = vmx_write_tsc_multiplier, + .get_l2_tsc_offset = vt_get_l2_tsc_offset, + .get_l2_tsc_multiplier = vt_get_l2_tsc_multiplier, + .write_tsc_offset = vt_write_tsc_offset, + .write_tsc_multiplier = vt_write_tsc_multiplier, .load_mmu_pgd = vt_load_mmu_pgd, - .check_intercept = vmx_check_intercept, + .check_intercept = vt_check_intercept, .handle_exit_irqoff = vt_handle_exit_irqoff, .request_immediate_exit = vt_request_immediate_exit,
@@ -628,7 +1005,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .sched_in = vt_sched_in, .cpu_dirty_log_size = PML_ENTITY_NUM, - .update_cpu_dirty_logging = vmx_update_cpu_dirty_logging, + .update_cpu_dirty_logging = vt_update_cpu_dirty_logging, .nested_ops = &vmx_nested_ops,
@@ -636,8 +1013,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .pi_start_assignment = vmx_pi_start_assignment, #ifdef CONFIG_X86_64 - .set_hv_timer = vmx_set_hv_timer, - .cancel_hv_timer = vmx_cancel_hv_timer, + .set_hv_timer = vt_set_hv_timer, + .cancel_hv_timer = vt_cancel_hv_timer, #endif .setup_mce = vmx_setup_mce,
@@ -647,8 +1024,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .leave_smm = vt_leave_smm, .enable_smi_window = vt_enable_smi_window, - .can_emulate_instruction = vmx_can_emulate_instruction, - .apic_init_signal_blocked = vmx_apic_init_signal_blocked, + .can_emulate_instruction = vt_can_emulate_instruction, + .apic_init_signal_blocked = vt_apic_init_signal_blocked, .migrate_timers = vmx_migrate_timers, .msr_filter_changed = vmx_msr_filter_changed,
arch/x86/kvm/vmx/tdx.c +53 -2
73 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +17 -0
@@ -166,6 +166,14 @@ int tdx_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection);int tdx_enter_smm (struct kvm_vcpu *vcpu,char *smstate);int tdx_leave_smm (struct kvm_vcpu *vcpu,const char *smstate);void tdx_enable_smi_window (struct kvm_vcpu *vcpu); +void tdx_set_virtual_apic_mode (struct kvm_vcpu *vcpu); + +int tdx_get_cpl (struct kvm_vcpu *vcpu); +void tdx_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg); +unsigned long tdx_get_rflags (struct kvm_vcpu *vcpu); +bool tdx_is_emulated_msr (u32 index,bool write); +u64 tdx_get_segment_base (struct kvm_vcpu *vcpu,int seg); +void tdx_get_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var,int seg);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -207,10 +215,19 @@ static inline void tdx_get_exit_info(static inline bool tdx_is_emulated_msr (u32 index,bool write) {return false; }static inline int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; }static inline int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; } +static inline int tdx_smi_allowed (struct kvm_vcpu *vcpu,bool for_injection) {return false; }static inline int tdx_enter_smm (struct kvm_vcpu *vcpu,char *smstate) {return 0 ; }static inline int tdx_leave_smm (struct kvm_vcpu *vcpu,const char *smstate) {return 0 ; }static inline void tdx_enable_smi_window (struct kvm_vcpu *vcpu) {} +static inline void tdx_set_virtual_apic_mode (struct kvm_vcpu *vcpu) {} + +static inline int tdx_get_cpl (struct kvm_vcpu *vcpu) {return 0 ; } +static inline void tdx_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg) {} +static inline unsigned long tdx_get_rflags (struct kvm_vcpu *vcpu) {return 0 ; } +static inline u64 tdx_get_segment_base (struct kvm_vcpu *vcpu,int seg) {return 0 ;} +static inline void tdx_get_segment ( +struct kvm_vcpu *vcpu,struct kvm_segment *var,int seg) {}static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v8 102/103 Documentation/virt/kvm: Document on Trust Domain Extensions(TDX)
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add documentation to Intel Trusted Domain Extensions(TDX) support.
Signed-off-by Isaku Yamahata <[email protected]>
--- Documentation/virt/kvm/api.rst | 9 +- Documentation/virt/kvm/index.rst | 2 + Documentation/virt/kvm/intel-tdx.rst | 345 +++++++++++++++++++++++++++ 3 files changed, 355 insertions(+), 1 deletion(-) create mode 100644 Documentation/virt/kvm/intel-tdx.rst
Documentation/virt/kvm/api.rst +8 -1
@@ -1402,6 +1402,9 @@ It is recommended to use this API instead of the KVM_SET_MEMORY_REGION ioctl. The KVM_SET_MEMORY_REGION does not allow fine grained control over memory allocation and is deprecated. +For TDX guest, deleting/moving memory region loses guest memory contents. +Read only region isn't supported. Only as-id 0 is supported. + 4.36 KVM_SET_TSS_ADDR ---------------------
@@ -4688,7 +4691,7 @@ H_GET_CPU_CHARACTERISTICS hypercall. :Capability: basic :Architectures: x86 -:Type: vm +:Type: vm ioctl, vcpu ioctl :Parameters: an opaque platform specific structure (in/out) :Returns: 0 on success; -1 on error
@@ -4700,6 +4703,10 @@ Currently, this ioctl is used for issuing Secure Encrypted Virtualization (SEV) commands on AMD Processors. The SEV commands are defined in Documentation/virt/kvm/amd-memory-encryption.rst. +Currently, this ioctl is used for issuing Trusted Domain Extensions +(TDX) commands on Intel Processors. The TDX commands are defined in +Documentation/virt/kvm/intel-tdx.rst. + 4.111 KVM_MEMORY_ENCRYPT_REG_REGION -----------------------------------
Documentation/virt/kvm/index.rst +2 -0
@@ -18,3 +18,5 @@ KVM locking vcpu-requests review-checklist + + intel-tdx
Documentation/virt/kvm/intel-tdx.rst new file +345 -0
345 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v8 103/103 KVM: x86: design documentation on TDX support of x86 KVM TDP MMU
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add a high level design document on TDX changes to TDP MMU.
Signed-off-by Isaku Yamahata <[email protected]>
Co-developed-by Bagas Sanjaya <[email protected]>
Signed-off-by Bagas Sanjaya <[email protected]>
--- Documentation/virt/kvm/tdx-tdp-mmu.rst | 417 +++++++++++++++++++++++++ 1 file changed, 417 insertions(+) create mode 100644 Documentation/virt/kvm/tdx-tdp-mmu.rst
Documentation/virt/kvm/tdx-tdp-mmu.rst new file +417 -0
@@ -0,0 +1,417 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Design of TDP MMU for TDX support +================================= +This document describes a (high level) design for TDX support of KVM TDP MMU of +x86 KVM. + +In this document, we use "TD" or "guest TD" to differentiate it from the current +"VM" (Virtual Machine), which is supported by KVM today. + + +Background of TDX +================= +TD private memory is designed to hold TD private content, encrypted by the CPU +using the TD ephemeral key. An encryption engine holds a table of encryption +keys, and an encryption key is selected for each memory transaction based on a +Host Key Identifier (HKID). By design, the host VMM does not have access to the +encryption keys. + +In the first generation of MKTME, HKID is "stolen" from the physical address by +allocating a configurable number of bits from the top of the physical address. +The HKID space is partitioned into shared HKIDs for legacy MKTME accesses and +private HKIDs for SEAM-mode-only accesses. We use 0 for the shared HKID on the +host so that MKTME can be opaque or bypassed on the host. + +During TDX non-root operation (i.e. guest TD), memory accesses can be qualified +as either shared or private, based on the value of a new SHARED bit in the Guest +Physical Address (GPA). The CPU translates shared GPAs using the usual VMX EPT +(Extended Page Table) or "Shared EPT" (in this document), which resides in the +host VMM memory. The Shared EPT is directly managed by the host VMM - the same +as with the current VMX. Since guest TDs usually require I/O, and the data +exchange needs to be done via shared memory, thus KVM needs to use the current +EPT functionality even for TDs. + +The CPU translates private GPAs using a separate Secure EPT. The Secure EPT +pages are encrypted and integrity-protected with the TD's ephemeral private key. +Secure EPT can be managed _indirectly_ by the host VMM, using the TDX interface +functions (SEAMCALLs), and thus conceptually Secure EPT is a subset of EPT +because not all functionalities are available. + +Since the execution of such interface functions takes much longer time than +accessing memory directly, in KVM we use the existing TDP code to mirror the +Secure EPT for the TD. And we think there are at least two options today in +terms of the timing for executing such SEAMCALLs: + +1. synchronous, i.e. while walking the TDP page tables, or +2. post-walk, i.e. record what needs to be done to the real Secure EPT during + the walk, and execute SEAMCALLs later. + +The option 1 seems to be more intuitive and simpler, but the Secure EPT +concurrency rules are different from the ones of the TDP or EPT. For example, +MEM.SEPT.RD acquire shared access to the whole Secure EPT tree of the target + +Secure EPT(SEPT) operations +--------------------------- +Secure EPT is an Extended Page Table for GPA-to-HPA translation of TD private +HPA. A Secure EPT is designed to be encrypted with the TD's ephemeral private +key. SEPT pages are allocated by the host VMM via Intel TDX functions, but their +content is intended to be hidden and is not architectural. + +Unlike the conventional EPT, the CPU can't directly read/write its entry. +Instead, TDX SEAMCALL API is used. Several SEAMCALLs correspond to operation on +the EPT entry. + +* TDH.MEM.SEPT.ADD(): + + Add a secure EPT page from the secure EPT tree. This corresponds to updating + the non-leaf EPT entry with present bit set + +* TDH.MEM.SEPT.REMOVE(): + + Remove the secure page from the secure EPT tree. There is no corresponding + to the EPT operation. + +* TDH.MEM.SEPT.RD(): + + Read the secure EPT entry. This corresponds to reading the EPT entry as + memory. Please note that this is much slower than direct memory reading. + +* TDH.MEM.PAGE.ADD() and TDH.MEM.PAGE.AUG(): + + Add a private page to the secure EPT tree. This corresponds to updating the + leaf EPT entry with present bit set. + +* THD.MEM.PAGE.REMOVE(): + + Remove a private page from the secure EPT tree. There is no corresponding + to the EPT operation. + +* TDH.MEM.RANGE.BLOCK(): + + This (mostly) corresponds to clearing the present bit of the leaf EPT entry. + Note that the private page is still linked in the secure EPT. To remove it + from the secure EPT, TDH.MEM.SEPT.REMOVE() and TDH.MEM.PAGE.REMOVE() needs to + be called. + +* TDH.MEM.TRACK(): + + Increment the TLB epoch counter. This (mostly) corresponds to EPT TLB flush. + Note that the private page is still linked in the secure EPT. To remove it + from the secure EPT, tdh_mem_page_remove() needs to be called. + + +Adding private page +------------------- +The procedure of populating the private page looks as follows. + +1. TDH.MEM.SEPT.ADD(512G level) +2. TDH.MEM.SEPT.ADD(1G level) +3. TDH.MEM.SEPT.ADD(2M level) +4. TDH.MEM.PAGE.AUG(4K level) + +Those operations correspond to updating the EPT entries. + +Dropping private page and TLB shootdown +--------------------------------------- +The procedure of dropping the private page looks as follows. + +1. TDH.MEM.RANGE.BLOCK(4K level) + + This mostly corresponds to clear the present bit in the EPT entry. This + prevents (or blocks) TLB entry from creating in the future. Note that the + private page is still linked in the secure EPT tree and the existing cache + entry in the TLB isn't flushed. + +2. TDH.MEM.TRACK(range) and TLB shootdown + + This mostly corresponds to the EPT TLB shootdown. Because all vcpus share + the same Secure EPT, all vcpus need to flush TLB. + + * TDH.MEM.TRACK(range) by one vcpu. It increments the global internal TLB + epoch counter. + + * send IPI to remote vcpus + * Other vcpu exits to VMM from guest TD and then re-enter. TDH.VP.ENTER(). + * TDH.VP.ENTER() checks the TLB epoch counter and If its TLB is old, flush + TLB. + + Note that only single vcpu issues tdh_mem_track(). + + Note that the private page is still linked in the secure EPT tree, unlike the + conventional EPT. + +3. TDH.MEM.PAGE.PROMOTE, TDH.MEM.PAGEDEMOTE(), TDH.MEM.PAGE.RELOCATE(), or + TDH.MEM.PAGE.REMOVE() + + There is no corresponding operation to the conventional EPT. + + * When changing page size (e.g. 4K <-> 2M) TDH.MEM.PAGE.PROMOTE() or + TDH.MEM.PAGE.DEMOTE() is used. During those operation, the guest page is + kept referenced in the Secure EPT. + + * When migrating page, TDH.MEM.PAGE.RELOCATE(). This requires both source + page and destination page. + * when destroying TD, TDH.MEM.PAGE.REMOVE() removes the private page from the + secure EPT tree. In this case TLB shootdown is not needed because vcpus + don't run any more. + +The basic idea for TDX support +============================== +Because shared EPT is the same as the existing EPT, use the existing logic for +shared EPT. On the other hand, secure EPT requires additional operations +instead of directly reading/writing of the EPT entry. + +On EPT violation, The KVM mmu walks down the EPT tree from the root, determines +the EPT entry to operate, and updates the entry. If necessary, a TLB shootdown +is done. Because it's very slow to directly walk secure EPT by TDX SEAMCALL, +TDH.MEM.SEPT.RD(), the mirror of secure EPT is created and maintained. Add +hooks to KVM MMU to reuse the existing code. + +EPT violation on shared GPA +--------------------------- +(1) EPT violation on shared GPA or zapping shared GPA + :: + + walk down shared EPT tree (the existing code) + | + | + V + shared EPT tree (CPU refers.) + +(2) update the EPT entry. (the existing code) + + TLB shootdown in the case of zapping. + + +EPT violation on private GPA +---------------------------- +(1) EPT violation on private GPA or zapping private GPA + :: + + walk down the mirror of secure EPT tree (mostly same as the existing code) + | + | + V + mirror of secure EPT tree (KVM MMU software only. reuse of the existing code) + +(2) update the (mirrored) EPT entry. (mostly same as the existing code) + +(3) call the hooks with what EPT entry is changed + :: + + | + NEW: hooks in KVM MMU + | + V + secure EPT root(CPU refers) + +(4) the TDX backend calls necessary TDX SEAMCALLs to update real secure EPT. + +The major modification is to add hooks for the TDX backend for additional +operations and to pass down which EPT, shared EPT, or private EPT is used, and +twist the behavior if we're operating on private EPT. + +The following depicts the relationship. +:: + + KVM | TDX module + | | | + -------------+---------- | | + | | | | + V V | | + shared GPA private GPA | | + CPU shared EPT pointer KVM private EPT pointer | CPU secure EPT pointer + | | | | + | | | | + V V | V + shared EPT private EPT<-------mirror----->Secure EPT + | | | | + | \--------------------+------\ | + | | | | + V | V V + shared guest page | private guest page + | + | + non-encrypted memory | encrypted memory + | + +shared EPT: CPU and KVM walk with shared GPA + Maintained by the existing code +private EPT: KVM walks with private GPA + Maintained by the twisted existing code +secure EPT: CPU walks with private GPA. + Maintained by TDX module with TDX SEAMCALLs via hooks + + +Tracking private EPT page +========================= +Shared EPT pages are managed by struct kvm_mmu_page. They are linked in a list +structure. When necessary, the list is traversed to operate on. Private EPT +pages have different characteristics. For example, private pages can't be +swapped out. When shrinking memory, we'd like to traverse only shared EPT pages +and skip private EPT pages. Likewise, page migration isn't supported for +private pages (yet). Introduce an additional list to track shared EPT pages and +track private EPT pages independently. + +At the beginning of EPT violation, the fault handler knows fault GPA, thus it +knows which EPT to operate on, private or shared. If it's private EPT, +an additional task is done. Something like "if (private) { callback a hook }". +Since the fault handler has deep function calls, it's cumbersome to hold the +information of which EPT is operating. Options to mitigate it are + +1. Pass the information as an argument for the function call. +2. Record the information in struct kvm_mmu_page somehow. +3. Record the information in vcpu structure. + +Option 2 was chosen. Because option 1 requires modifying all the functions. It +would affect badly to the normal case. Option 3 doesn't work well because in +some cases, we need to walk both private and shared EPT. + +The role of the EPT page can be utilized and one bit can be curved out from +unused bits in struct kvm_mmu_page_role. When allocating the EPT page, +initialize the information. Mostly struct kvm_mmu_page is available because +we're operating on EPT pages. + + +The conversion of private GPA and shared GPA +============================================ +A page of a given GPA can be assigned to only private GPA xor shared GPA at one +time. The GPA can't be accessed simultaneously via both private GPA and shared +GPA. On guest startup, all the GPAs are assigned as private. Guest converts +the range of GPA to shared (or private) from private (or shared) by MapGPA +hypercall. MapGPA hypercall takes the start GPA and the size of the region. If +the given start GPA is shared, VMM converts the region into shared (if it's +already shared, nop). If the start GPA is private, VMM converts the region into +private. It implies the guest won't access the unmapped region. private(or +shared) region after converting to shared(or private). + +If the guest TD triggers an EPT violation on the already converted region, the +access won't be allowed (loop in EPT violation) until other vcpu converts back +the region. + +KVM MMU records which GPA is allowed to access, private or shared by xarray. + + +The original TDP MMU and race condition +======================================= +Because vcpus share the EPT, once the EPT entry is zapped, we need to shootdown +TLB. Send IPI to remote vcpus. Remote vcpus flush their down TLBs. Until TLB +shootdown is done, vcpus may reference the zapped guest page. + +TDP MMU uses read lock of mmu_lock to mitigate vcpu contention. When read lock +is obtained, it depends on the atomic update of the EPT entry. (On the other +hand legacy MMU uses write lock.) When vcpu is populating/zapping the EPT entry +with a read lock held, other vcpu may be populating or zapping the same EPT +entry at the same time. + +To avoid the race condition, the entry is frozen. It means the EPT entry is set +to the special value, REMOVED_SPTE which clears the present bit. And then after +TLB shootdown, update the EPT entry to the final value. + +Concurrent zapping +------------------ +1. read lock +2. freeze the EPT entry (atomically set the value to REMOVED_SPTE) + If other vcpu froze the entry, restart page fault. +3. TLB shootdown + + * send IPI to remote vcpus + * TLB flush (local and remote) + + For each entry update, TLB shootdown is needed because of the + concurrency. +4. atomically set the EPT entry to the final value +5. read unlock + +Concurrent populating +--------------------- +In the case of populating the non-present EPT entry, atomically update the EPT +entry. + +1. read lock + +2. atomically update the EPT entry + If other vcpu frozen the entry or updated the entry, restart page fault. + +3. read unlock + +In the case of updating the present EPT entry (e.g. page migration), the +operation is split into two. Zapping the entry and populating the entry. + +1. read lock +2. zap the EPT entry. follow the concurrent zapping case. +3. populate the non-present EPT entry. +4. read unlock + +Non-concurrent batched zapping +------------------------------ +In some cases, zapping the ranges is done exclusively with a write lock held. +In this case, the TLB shootdown is batched into one. + +1. write lock +2. zap the EPT entries by traversing them +3. TLB shootdown +4. write unlock + +For Secure EPT, TDX SEAMCALLs are needed in addition to updating the mirrored +EPT entry. + +TDX concurrent zapping +---------------------- +Add a hook for TDX SEAMCALLs at the step of the TLB shootdown. + +1. read lock +2. freeze the EPT entry(set the value to REMOVED_SPTE) +3. TLB shootdown via a hook + + * TLB.MEM.RANGE.BLOCK() + * TLB.MEM.TRACK() + * send IPI to remote vcpus + +4. set the EPT entry to the final value +5. read unlock + +TDX concurrent populating +------------------------- +TDX SEAMCALLs are required in addition to operating the mirrored EPT entry. The +frozen entry is utilized by following the zapping case to avoid the race +condition. A hook can be added. + +1. read lock +2. freeze the EPT entry +3. hook + + * TDH_MEM_SEPT_ADD() for non-leaf or TDH_MEM_PAGE_AUG() for leaf. + +4. set the EPT entry to the final value +5. read unlock + +Without freezing the entry, the following race can happen. Suppose two vcpus +are faulting on the same GPA and the 2M and 4K level entries aren't populated +yet. + +* vcpu 1: update 2M level EPT entry +* vcpu 2: update 4K level EPT entry +* vcpu 2: TDX SEAMCALL to update 4K secure EPT entry => error +* vcpu 1: TDX SEAMCALL to update 2M secure EPT entry + + +TDX non-concurrent batched zapping +---------------------------------- +For simplicity, the procedure of concurrent populating is utilized. The +procedure can be optimized later. + + +Co-existing with unmapping guest private memory +=============================================== +TODO. This needs to be addressed. + + +Restrictions or future work +=========================== +The following features aren't supported yet at the moment. + +* optimizing non-concurrent zap +* Large page +* Page migration
Signature · 1 line
2.25.1
Bagas Sanjaya
AppliedQuoted 16 lines · On Sun, Aug 07, 2022 at 03:00:45PM -0700, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
KVM TDX basic feature support
Hello. This is v8 the patch series vof KVM TDX support.
This is based on v5.19-rc8 + kvm/queue branch + TDX HOST patch series. The tree can be found at https://github.com/intel/tdx/tree/kvm-upstream
How to run/test: It's describe at https://github.com/intel/tdx/wiki/TDX-KVMMajor changes from v7:
- Use xarray to track whether GFN is private or shared. DropSPTE_SHARED_MASK. The complex state machine withSPTE_SHARED_MASKwas ditched.
- Large page support is implemented. But will be posted as independent RFC patch.
- fd-based private page v7 is integrated. This is mostly same to Chao's patches. It's in github.
Quoted 3 lines
Thanks,
Isaku Yamahata
Hi, thanks for the series.
When building htmldocs, I found new warnings:
Documentation/x86/tdx.rst:69: WARNING: Unexpected indentation. Documentation/x86/tdx.rst:70: WARNING: Block quote ends without a blank line; unexpected unindent. Documentation/virt/kvm/tdx-tdp-mmu.rst: WARNING: document isn't included in any toctree
I have applied the fixup (also with line blocks to code blocks conversion):
Documentation/virt/kvm/index.rst +1 -0
@@ -20,3 +20,4 @@ KVM review-checklist intel-tdx + tdx-tdp-mmu
Documentation/x86/tdx.rst +12 -12
@@ -62,7 +62,7 @@ use it as 'metadata' for the TDX memory. It also takes additional CPU time to initialize those metadata along with the TDX module itself. Both are not trivial. Current kernel doesn't choose to always initialize the TDX module during kernel boot, but provides a function tdx_init() to -allow the caller to initialize TDX when it truly wants to use TDX: +allow the caller to initialize TDX when it truly wants to use TDX:: ret = tdx_init(); if (ret)
@@ -79,20 +79,20 @@ caller. User can consult dmesg to see the presence of the TDX module, and whether it has been initialized. -If the TDX module is not loaded, dmesg shows below: +If the TDX module is not loaded, dmesg shows below:: -| [..] tdx: TDX module is not loaded. + [..] tdx: TDX module is not loaded. If the TDX module is initialized successfully, dmesg shows something -like below: +like below:: -| [..] tdx: TDX module: vendor_id 0x8086, major_version 1, minor_version 0, build_date 20211209, build_num 160 -| [..] tdx: 65667 pages allocated for PAMT. -| [..] tdx: TDX module initialized. + [..] tdx: TDX module: vendor_id 0x8086, major_version 1, minor_version 0, build_date 20211209, build_num 160 + [..] tdx: 65667 pages allocated for PAMT. + [..] tdx: TDX module initialized. -If the TDX module failed to initialize, dmesg shows below: +If the TDX module failed to initialize, dmesg shows below:: -| [..] tdx: Failed to initialize TDX module. Shut it down. + [..] tdx: Failed to initialize TDX module. Shut it down. TDX Interaction to Other Kernel Components ------------------------------------------
@@ -143,10 +143,10 @@ There are basically two memory hot-add cases that need to be prevented: ACPI memory hot-add and driver managed memory hot-add. The kernel rejectes the driver managed memory hot-add too when TDX is enabled by BIOS. For instance, dmesg shows below error when using kmem driver to -add a legacy PMEM as system RAM: +add a legacy PMEM as system RAM:: -| [..] tdx: Unable to add memory [0x580000000, 0x600000000) on TDX enabled platform. -| [..] kmem dax0.0: mapping0: 0x580000000-0x5ffffffff memory add failed + [..] tdx: Unable to add memory [0x580000000, 0x600000000) on TDX enabled platform. + [..] kmem dax0.0: mapping0: 0x580000000-0x5ffffffff memory add failed However, adding new memory to ZONE_DEVICE should not be prevented as those pages are not managed by the page allocator. Therefore,
Thanks.
Signature · 1 line
An old man doll... just what I always wanted! - Clara
Quoted 23 lines · Bagas Sanjaya
Quoted 16 lines · On Sun, Aug 07, 2022 at 03:00:45PM -0700, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
KVM TDX basic feature support
Hello. This is v8 the patch series vof KVM TDX support.
This is based on v5.19-rc8 + kvm/queue branch + TDX HOST patch series. The tree can be found at https://github.com/intel/tdx/tree/kvm-upstream
How to run/test: It's describe at https://github.com/intel/tdx/wiki/TDX-KVMMajor changes from v7:
- Use xarray to track whether GFN is private or shared. DropSPTE_SHARED_MASK. The complex state machine withSPTE_SHARED_MASKwas ditched.
- Large page support is implemented. But will be posted as independent RFC patch.
- fd-based private page v7 is integrated. This is mostly same to Chao's patches. It's in github.Quoted 3 lines
Thanks,
Isaku YamahataHi, thanks for the series.
Quoted 7 lines
When building htmldocs, I found new warnings:
Documentation/x86/tdx.rst:69: WARNING: Unexpected indentation.Documentation/x86/tdx.rst:70: WARNING: Block quote ends without a blank line; unexpected unindent.Documentation/virt/kvm/tdx-tdp-mmu.rst: WARNING: document isn't included in any toctreeI have applied the fixup (also with line blocks to code blocks conversion):
Thanks. I'll apply them.
Signature · 1 line
Isaku Yamahata <[email protected]>