KVM TDX basic feature support
Subsystems (6)
PATCH v18 47/121 KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page
In reply to [email protected]From: Isaku Yamahata <[email protected]>
For private GPA, CPU refers a private page table whose contents are encrypted. The dedicated APIs to operate on it (e.g. updating/reading its PTE entry) are used and their cost is expensive.
When KVM resolves KVM page fault, it walks the page tables. To reuse the existing KVM MMU code and mitigate the heavy cost to directly walk private page table, allocate one more page to copy the dummy page table for KVM MMU code to directly walk. Resolve KVM page fault with the existing code, and do additional operations necessary for the private page table. To distinguish such cases, the existing KVM page table is called a shared page table (i.e. not associated with private page table), and the page table with private page table is called a private page table. The relationship is depicted below.
Add a private pointer to struct kvm_mmu_page for private page table and add helper functions to allocate/initialize/free a private page table page.
KVM page fault |
| |
V |
-------------+---------- |
| | |
V V |
shared GPA private GPA |
| | |
V V |
shared PT root dummy PT root | private PT root
| | | |
V V | V
shared PT dummy PT ----propagate----> private PT
| | | |
| \-----------------+------\ |
| | | |
V | V V
shared guest page | private guest page
|
non-encrypted memory | encrypted memory
|PT: page table
- Shared PT is visible to KVM and it is used by CPU.
- Private PT is used by CPU but it is invisible to KVM.
- Dummy PT is visible to KVM but not used by CPU. It is used to propagate PT change to the actual private PT which is used by CPU.
--- arch/x86/include/asm/kvm_host.h | 5 ++ arch/x86/kvm/mmu/mmu.c | 7 +++ arch/x86/kvm/mmu/mmu_internal.h | 83 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 1 + 4 files changed, 92 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +5 -0
@@ -841,6 +841,11 @@ struct kvm_vcpu_arch {struct kvm_mmu_memory_cache mmu_shadow_page_cache;struct kvm_mmu_memory_cache mmu_shadowed_info_cache;struct kvm_mmu_memory_cache mmu_page_header_cache; +/* +* This cache is to allocate private page table. E.g. Secure-EPT used +* by the TDX module. +*/ +struct kvm_mmu_memory_cache mmu_private_spt_cache;/* * QEMU userspace and the guest each have their own FPU state.
arch/x86/kvm/mmu/mmu.c +7 -0
@@ -685,6 +685,12 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect) 1 + PT64_ROOT_MAX_LEVEL + PTE_PREFETCH_NUM);if (r)return r; +if (kvm_gfn_shared_mask (vcpu->kvm)) { + r =kvm_mmu_topup_memory_cache (&vcpu->arch.mmu_private_spt_cache, +PT64_ROOT_MAX_LEVEL ); +if (r) +return r; + } r =kvm_mmu_topup_memory_cache (&vcpu->arch.mmu_shadow_page_cache,PT64_ROOT_MAX_LEVEL );if (r)
@@ -704,6 +710,7 @@ static void mmu_free_memory_caches(struct kvm_vcpu *vcpu)kvm_mmu_free_memory_cache (&vcpu->arch.mmu_pte_list_desc_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_shadow_page_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_shadowed_info_cache); +kvm_mmu_free_memory_cache (&vcpu->arch.mmu_private_spt_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_page_header_cache); }
arch/x86/kvm/mmu/mmu_internal.h +79 -4
101 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.c +1 -0
@@ -53,6 +53,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)static void tdp_mmu_free_sp (struct kvm_mmu_page *sp) { +kvm_mmu_free_private_spt (sp);free_page ((unsigned long )sp->spt);kmem_cache_free (mmu_page_header_cache, sp); }
Signature · 1 line
2.25.1
Re: [PATCH v18 047/121] KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page
Quoted 41 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
For private GPA, CPU refers a private page table whose contents are encrypted. The dedicated APIs to operate on it (e.g. updating/reading its PTE entry) are used and their cost is expensive.
When KVM resolves KVM page fault, it walks the page tables. To reuse the existing KVM MMU code and mitigate the heavy cost to directly walk private page table, allocate one more page to copy the dummy page table for KVM MMU code to directly walk. Resolve KVM page fault with the existing code, and do additional operations necessary for the private page table. To distinguish such cases, the existing KVM page table is called a shared page table (i.e. not associated with private page table), and the page table with private page table is called a private page table. The relationship is depicted below.
Add a private pointer to struct
kvm_mmu_pagefor private page table and add helper functions to allocate/initialize/free a private page table page.KVM page fault | | | V | -------------+---------- | | | | V V | shared GPA private GPA | | | | V V | shared PT root dummy PT root | private PT root | | | | V V | V shared PT dummy PT ----propagate----> private PT | | | | | \-----------------+------\ | | | | | V | V V shared guest page | private guest page | non-encrypted memory | encrypted memory |
Quoted 5 lines
PT: page table
- Shared PT is visible to KVM and it is used by CPU.
- Private PT is used by CPU but it is invisible to KVM.
- Dummy PT is visible to KVM but not used by CPU. It is used to propagate PT change to the actual private PT which is used by CPU.
Nit: one typo below.
Quoted 115 lines
Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/include/asm/kvm_host.h | 5 ++ arch/x86/kvm/mmu/mmu.c | 7 +++ arch/x86/kvm/mmu/mmu_internal.h | 83 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 1 + 4 files changed, 92 insertions(+), 4 deletions(-)arch/x86/include/asm/kvm_host.h +5
-0@@ -841,6 +841,11 @@ struct kvm_vcpu_arch {struct kvm_mmu_memory_cache mmu_shadow_page_cache;struct kvm_mmu_memory_cache mmu_shadowed_info_cache;struct kvm_mmu_memory_cache mmu_page_header_cache; +/* +* This cache is to allocate private page table. E.g. Secure-EPT used +* by the TDX module. +*/ +struct kvm_mmu_memory_cache mmu_private_spt_cache;/* * QEMU userspace and the guest each have their own FPU state. arch/x86/kvm/mmu/mmu.c +7
-0@@ -685,6 +685,12 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect) 1 + PT64_ROOT_MAX_LEVEL + PTE_PREFETCH_NUM);if (r)return r; +if (kvm_gfn_shared_mask (vcpu->kvm)) { + r =kvm_mmu_topup_memory_cache (&vcpu->arch.mmu_private_spt_cache, +PT64_ROOT_MAX_LEVEL ); +if (r) +return r; + } r =kvm_mmu_topup_memory_cache (&vcpu->arch.mmu_shadow_page_cache,PT64_ROOT_MAX_LEVEL );if (r)@@ -704,6 +710,7 @@ static void mmu_free_memory_caches(struct kvm_vcpu *vcpu)kvm_mmu_free_memory_cache (&vcpu->arch.mmu_pte_list_desc_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_shadow_page_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_shadowed_info_cache); +kvm_mmu_free_memory_cache (&vcpu->arch.mmu_private_spt_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_page_header_cache); }arch/x86/kvm/mmu/mmu_internal.h +39
-4@@ -101,7 +101,23 @@ struct kvm_mmu_page {int root_count;refcount_t tdp_mmu_root_count; }; -unsigned int unsync_children; +union { +struct { +unsigned int unsync_children; +/* +* Number of writes since the last time traversal +* visited this page. +*/ +atomic_t write_flooding_count; + }; +#ifdef CONFIG_KVM_MMU_PRIVATE +/* +* Associated private shadow page table, e.g. Secure-EPT page +* passed to the TDX module. +*/ +void *private_spt; +#endif + };union {struct kvm_rmap_head parent_ptes;/* rmap pointers to parent sptes */ tdp_ptep_t ptep;@@ -124,9 +140,6 @@ struct kvm_mmu_page {int clear_spte_count;#endif -/* Number of writes since the last time traversal visited this page. */ -atomic_t write_flooding_count; -#ifdef CONFIG_X86_64 /* Used for freeing the page asynchronously if it is a TDP MMU page. */ struct rcu_head rcu_head;@@ -150,6 +163,68 @@ static inline bool is_private_sp(const struct kvm_mmu_page *sp)return kvm_mmu_page_role_is_private (sp->role); } +#ifdef CONFIG_KVM_MMU_PRIVATE +static inline void *kvm_mmu_private_spt (struct kvm_mmu_page *sp) +{ +return sp->private_spt; +} + +static inline void kvm_mmu_init_private_spt (struct kvm_mmu_page *sp,void *private_spt) +{ + sp->private_spt = private_spt; +} + +static inline void kvm_mmu_alloc_private_spt (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) +{ +bool is_root = vcpu->arch.root_mmu.root_role.level == sp->role.level; + +KVM_BUG_ON (!kvm_mmu_page_role_is_private (sp->role), vcpu->kvm); +if (is_root) +/* +* Because TDX module assigns root Secure-EPT page and set it to +* Secure-EPTP when TD vcpu is created, secure page table for +* root isn't needed. +*/
arch/x86/kvm/mmu/mmu_internal.h static inline bool is_private_sp(const struct kvm_mmu_page *sp) · line 197
Quoted 10 lines
arch/x86/kvm/mmu/mmu_internal.h +10
-0+ sp->private_spt =NULL ; + else { +/* +* Because the TDX module doesn't trust VMM and initializes +* the pages itself, KVM doesn't initialize them. Allocate +* pages with garbage and give them to the TDX module. +*/ + sp->private_spt =kvm_mmu_memory_cache_alloc (&vcpu->arch.mmu_private_spt_cache); +/* +* Because mmu_private_spt_cache is topped up before staring kvm
s/staring/starting
Quoted 45 lines
arch/x86/kvm/mmu/mmu_internal.h +30
-0+* page fault resolving, the allocation above shouldn't fail. +*/ + WARN_ON_ONCE(!sp->private_spt); + } +} + +static inline void kvm_mmu_free_private_spt (struct kvm_mmu_page *sp) +{ +if (sp->private_spt) +free_page ((unsigned long )sp->private_spt); +} +#else +static inline void *kvm_mmu_private_spt (struct kvm_mmu_page *sp) +{ +return NULL ; +} + +static inline void kvm_mmu_init_private_spt (struct kvm_mmu_page *sp,void *private_spt) +{ +} + +static inline void kvm_mmu_alloc_private_spt (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) +{ +} + +static inline void kvm_mmu_free_private_spt (struct kvm_mmu_page *sp) +{ +} +#endif +static inline bool kvm_mmu_page_ad_need_write_protect (struct kvm_mmu_page *sp) {/* arch/x86/kvm/mmu/tdp_mmu.c +1
-0@@ -53,6 +53,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)static void tdp_mmu_free_sp (struct kvm_mmu_page *sp) { +kvm_mmu_free_private_spt (sp);free_page ((unsigned long )sp->spt);kmem_cache_free (mmu_page_header_cache, sp); }
PATCH v18 48/121 KVM: x86/tdp_mmu: Don't zap private pages for unsupported cases
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX supports only write-back(WB) memory type for private memory architecturally so that (virtualized) memory type change doesn't make sense for private memory. Also currently, page migration isn't supported for TDX yet. (TDX architecturally supports page migration. it's KVM and kernel implementation issue.)
Regarding memory type change (mtrr virtualization and lapic page mapping change), pages are zapped by kvm_zap_gfn_range(). On the next KVM page fault, the SPTE entry with a new memory type for the page is populated. Regarding page migration, pages are zapped by the mmu notifier. On the next KVM page fault, the new migrated page is populated. Don't zap private pages on unmapping for those two cases.
When deleting/moving a KVM memory slot, zap private pages. Typically tearing down VM. Don't invalidate private page tables. i.e. zap only leaf SPTEs for KVM mmu that has a shared bit mask. The existing kvm_tdp_mmu_invalidate_all_roots() depends on role.invalid with read-lock of mmu_lock so that other vcpu can operate on KVM mmu concurrently. It marks the root page table invalid and zaps SPTEs of the root page tables. The TDX module doesn't allow to unlink a protected root page table from the hardware and then allocate a new one for it. i.e. replacing a protected root page table. Instead, zap only leaf SPTEs for KVM mmu with a shared bit mask set.
--- arch/x86/kvm/mmu/mmu.c | 61 ++++++++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 37 +++++++++++++++++++---- arch/x86/kvm/mmu/tdp_mmu.h | 5 ++-- 3 files changed, 92 insertions(+), 11 deletions(-)
arch/x86/kvm/mmu/mmu.c +58 -3
@@ -6267,7 +6267,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)* e.g. before kvm_zap_obsolete_pages() could drop mmu_lock and yield. */ if (tdp_mmu_enabled) -kvm_tdp_mmu_invalidate_all_roots (kvm); +kvm_tdp_mmu_invalidate_all_roots (kvm, true);/* * Notify all vcpus to reload its shadow page table and flush TLB.
@@ -6389,7 +6389,16 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) flush =kvm_rmap_zap_gfn_range (kvm, gfn_start, gfn_end);if (tdp_mmu_enabled) - flush =kvm_tdp_mmu_zap_leafs (kvm, gfn_start, gfn_end, flush); +/* +* zap_private = false. Zap only shared pages. +* +* kvm_zap_gfn_range() is used when MTRR or PAT memory +* type was changed. Later on the next kvm page fault, +* populate it with updated spte entry. +* Because only WB is supported for private pages, don't +* care of private pages. +*/ + flush =kvm_tdp_mmu_zap_leafs (kvm, gfn_start, gfn_end, flush, false);if (flush)kvm_flush_remote_tlbs_range (kvm, gfn_start, gfn_end - gfn_start);
@@ -6835,10 +6844,56 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)kvm_mmu_zap_all (kvm); } +static void kvm_mmu_zap_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) +{ +bool flush = false; + +write_lock (&kvm->mmu_lock); + +/* +* Zapping non-leaf SPTEs, a.k.a. not-last SPTEs, isn't required, worst +* case scenario we'll have unused shadow pages lying around until they +* are recycled due to age or when the VM is destroyed. +*/ +if (tdp_mmu_enabled) { +struct kvm_gfn_range range = { + .slot = slot, + .start = slot->base_gfn, + .end = slot->base_gfn + slot->npages, + .may_block = true, + +/* +* This handles both private gfn and shared gfn. +* All private page should be zapped on memslot deletion. +*/ + .only_private = true, + .only_shared = true, + }; + + flush =kvm_tdp_mmu_unmap_gfn_range (kvm, &range, flush); + }else { +/* TDX supports only TDP-MMU case. */ +WARN_ON_ONCE (1 ); + flush = true; + } +if (flush) +kvm_flush_remote_tlbs (kvm); + +write_unlock (&kvm->mmu_lock); +} +void kvm_arch_flush_shadow_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) { -kvm_mmu_zap_all_fast (kvm); +if (kvm_gfn_shared_mask (kvm)) +/* +* Secure-EPT requires to release PTs from the leaf. The +* optimization to zap root PT first with child PT doesn't +* work. +*/ +kvm_mmu_zap_memslot (kvm, slot); +else +kvm_mmu_zap_all_fast (kvm); }void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen)
arch/x86/kvm/mmu/tdp_mmu.c +31 -6
@@ -37,7 +37,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)* for zapping and thus puts the TDP MMU's reference to each root, i.e. * ultimately frees all roots. */ -kvm_tdp_mmu_invalidate_all_roots (kvm); +kvm_tdp_mmu_invalidate_all_roots (kvm, false);kvm_tdp_mmu_zap_invalidated_roots (kvm);WARN_ON (atomic64_read (&kvm->arch.tdp_mmu_pages));
@@ -771,7 +771,8 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)* operation can cause a soft lockup. */ static bool tdp_mmu_zap_leafs (struct kvm *kvm,struct kvm_mmu_page *root, -gfn_t start,gfn_t end,bool can_yield,bool flush) +gfn_t start,gfn_t end,bool can_yield,bool flush, +bool zap_private) {struct tdp_iter iter;
@@ -779,6 +780,10 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,lockdep_assert_held_write (&kvm->mmu_lock); +WARN_ON_ONCE (zap_private && !is_private_sp (root)); +if (!zap_private &&is_private_sp (root)) +return false; +rcu_read_lock ();for_each_tdp_pte_min_level (iter, root,PG_LEVEL_4K , start, end) {
@@ -810,13 +815,15 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,* true if a TLB flush is needed before releasing the MMU lock, i.e. if one or * more SPTEs were zapped since the MMU lock was last acquired. */ -bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,gfn_t start,gfn_t end,bool flush) +bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,gfn_t start,gfn_t end,bool flush, +bool zap_private) {struct kvm_mmu_page *root;lockdep_assert_held_write (&kvm->mmu_lock);for_each_tdp_mmu_root_yield_safe (kvm, root) - flush =tdp_mmu_zap_leafs (kvm, root, start, end, true, flush); + flush =tdp_mmu_zap_leafs (kvm, root, start, end, true, flush, + zap_private &&is_private_sp (root));return flush; }
@@ -891,7 +898,7 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm)* Note, kvm_tdp_mmu_zap_invalidated_roots() is gifted the TDP MMU's reference. * See kvm_tdp_mmu_get_vcpu_root_hpa(). */ -void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm) +void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm,bool skip_private) {struct kvm_mmu_page *root;
@@ -916,6 +923,12 @@ void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm)* or get/put references to roots. */ list_for_each_entry (root, &kvm->arch.tdp_mmu_roots, link) { +/* +* Skip private root since private page table +* is only torn down when VM is destroyed. +*/ +if (skip_private &&is_private_sp (root)) +continue ;/* * Note, invalid roots can outlive a memslot update! Invalid * roots must be *zapped* before the memslot update completes,
@@ -1104,14 +1117,26 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return ret; } +/* Used by mmu notifier via kvm_unmap_gfn_range() */ bool kvm_tdp_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range,bool flush) {struct kvm_mmu_page *root; +bool zap_private = false; + +if (kvm_gfn_shared_mask (kvm)) { +if (!range->only_private && !range->only_shared) +/* attributes change */ + zap_private = !(range->arg.attributes & +KVM_MEMORY_ATTRIBUTE_PRIVATE ); +else + zap_private = range->only_private; + }__for_each_tdp_mmu_root_yield_safe (kvm, root, range->slot->as_id, false) flush =tdp_mmu_zap_leafs (kvm, root, range->start, range->end, - range->may_block, flush); + range->may_block, flush, + zap_private &&is_private_sp (root));return flush; }
arch/x86/kvm/mmu/tdp_mmu.h +3 -2
@@ -19,10 +19,11 @@ __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *root)void kvm_tdp_mmu_put_root (struct kvm *kvm,struct kvm_mmu_page *root); -bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,gfn_t start,gfn_t end,bool flush); +bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,gfn_t start,gfn_t end,bool flush, +bool zap_private);bool kvm_tdp_mmu_zap_sp (struct kvm *kvm,struct kvm_mmu_page *sp);void kvm_tdp_mmu_zap_all (struct kvm *kvm); -void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm); +void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm,bool skip_private);void kvm_tdp_mmu_zap_invalidated_roots (struct kvm *kvm);int kvm_tdp_mmu_map (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);
Signature · 1 line
2.25.1
Re: [PATCH v18 048/121] KVM: x86/tdp_mmu: Don't zap private pages for unsupported cases
Quoted 83 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Sean Christopherson <[email protected]>
TDX supports only write-back(WB) memory type for private memory architecturally so that (virtualized) memory type change doesn't make sense for private memory. Also currently, page migration isn't supported for TDX yet. (TDX architecturally supports page migration. it's KVM and kernel implementation issue.)
Regarding memory type change (mtrr virtualization and lapic page mapping change), pages are zapped by
kvm_zap_gfn_range(). On the next KVM page fault, the SPTE entry with a new memory type for the page is populated. Regarding page migration, pages are zapped by the mmu notifier. On the next KVM page fault, the new migrated page is populated. Don't zap private pages on unmapping for those two cases.When deleting/moving a KVM memory slot, zap private pages. Typically tearing down VM. Don't invalidate private page tables. i.e. zap only leaf SPTEs for KVM mmu that has a shared bit mask. The existing
kvm_tdp_mmu_invalidate_all_roots()depends on role.invalid with read-lock ofmmu_lockso that other vcpu can operate on KVM mmu concurrently. It marks the root page table invalid and zaps SPTEs of the root page tables. The TDX module doesn't allow to unlink a protected root page table from the hardware and then allocate a new one for it. i.e. replacing a protected root page table. Instead, zap only leaf SPTEs for KVM mmu with a shared bit mask set.Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 61 ++++++++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 37 +++++++++++++++++++---- arch/x86/kvm/mmu/tdp_mmu.h | 5 ++-- 3 files changed, 92 insertions(+), 11 deletions(-)arch/x86/kvm/mmu/mmu.c +25
-2@@ -6267,7 +6267,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)* e.g. before kvm_zap_obsolete_pages() could drop mmu_lock and yield. */ if (tdp_mmu_enabled) -kvm_tdp_mmu_invalidate_all_roots (kvm); +kvm_tdp_mmu_invalidate_all_roots (kvm, true);/* * Notify all vcpus to reload its shadow page table and flush TLB. @@ -6389,7 +6389,16 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) flush =kvm_rmap_zap_gfn_range (kvm, gfn_start, gfn_end);if (tdp_mmu_enabled) - flush =kvm_tdp_mmu_zap_leafs (kvm, gfn_start, gfn_end, flush); +/* +* zap_private = false. Zap only shared pages. +* +* kvm_zap_gfn_range() is used when MTRR or PAT memory +* type was changed. Later on the next kvm page fault, +* populate it with updated spte entry. +* Because only WB is supported for private pages, don't +* care of private pages. +*/ + flush =kvm_tdp_mmu_zap_leafs (kvm, gfn_start, gfn_end, flush, false);if (flush)kvm_flush_remote_tlbs_range (kvm, gfn_start, gfn_end - gfn_start);@@ -6835,10 +6844,56 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)kvm_mmu_zap_all (kvm); } +static void kvm_mmu_zap_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) +{ +bool flush = false; + +write_lock (&kvm->mmu_lock); + +/* +* Zapping non-leaf SPTEs, a.k.a. not-last SPTEs, isn't required, worst +* case scenario we'll have unused shadow pages lying around until they +* are recycled due to age or when the VM is destroyed. +*/ +if (tdp_mmu_enabled) { +struct kvm_gfn_range range = { + .slot = slot,
arch/x86/kvm/mmu/mmu.c void kvm_arch_flush_shadow_all(struct kvm *kvm) · line 6870
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10
-0+ .start = slot->base_gfn, + .end = slot->base_gfn + slot->npages, + .may_block = true, + +/* +* This handles both private gfn and shared gfn. +* All private page should be zapped on memslot deletion. +*/ + .only_private = true, + .only_shared = true,
only_{private,shared} were added in Sean's gmem patch series, but removed in v14.
https://lore.kernel.org/kvm/[email protected]/
I didn't find the two members are added in this patch series, is there other dependent patches?
Also, what does it mean by setting both only_private and only_shared as true? Per your comment, it seems both shared & private should be handled. Then, they both should be false?
It is a bit confusing, maybe you can consider to use enum mentioned in the discussion below?
https://lore.kernel.org/kvm/[email protected]/#t
Quoted 150 lines
arch/x86/kvm/mmu/mmu.c +23
-1+ }; + + flush =kvm_tdp_mmu_unmap_gfn_range (kvm, &range, flush); + }else { +/* TDX supports only TDP-MMU case. */ +WARN_ON_ONCE (1 ); + flush = true; + } +if (flush) +kvm_flush_remote_tlbs (kvm); + +write_unlock (&kvm->mmu_lock); +} +void kvm_arch_flush_shadow_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) { -kvm_mmu_zap_all_fast (kvm); +if (kvm_gfn_shared_mask (kvm)) +/* +* Secure-EPT requires to release PTs from the leaf. The +* optimization to zap root PT first with child PT doesn't +* work. +*/ +kvm_mmu_zap_memslot (kvm, slot); +else +kvm_mmu_zap_all_fast (kvm); }void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen)arch/x86/kvm/mmu/tdp_mmu.c +31
-6@@ -37,7 +37,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)* for zapping and thus puts the TDP MMU's reference to each root, i.e. * ultimately frees all roots. */ -kvm_tdp_mmu_invalidate_all_roots (kvm); +kvm_tdp_mmu_invalidate_all_roots (kvm, false);kvm_tdp_mmu_zap_invalidated_roots (kvm);WARN_ON (atomic64_read (&kvm->arch.tdp_mmu_pages));@@ -771,7 +771,8 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)* operation can cause a soft lockup. */ static bool tdp_mmu_zap_leafs (struct kvm *kvm,struct kvm_mmu_page *root, -gfn_t start,gfn_t end,bool can_yield,bool flush) +gfn_t start,gfn_t end,bool can_yield,bool flush, +bool zap_private) {struct tdp_iter iter;@@ -779,6 +780,10 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,lockdep_assert_held_write (&kvm->mmu_lock); +WARN_ON_ONCE (zap_private && !is_private_sp (root)); +if (!zap_private &&is_private_sp (root)) +return false; +rcu_read_lock ();for_each_tdp_pte_min_level (iter, root,PG_LEVEL_4K , start, end) {@@ -810,13 +815,15 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,* true if a TLB flush is needed before releasing the MMU lock, i.e. if one or * more SPTEs were zapped since the MMU lock was last acquired. */ -bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,gfn_t start,gfn_t end,bool flush) +bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,gfn_t start,gfn_t end,bool flush, +bool zap_private) {struct kvm_mmu_page *root;lockdep_assert_held_write (&kvm->mmu_lock);for_each_tdp_mmu_root_yield_safe (kvm, root) - flush =tdp_mmu_zap_leafs (kvm, root, start, end, true, flush); + flush =tdp_mmu_zap_leafs (kvm, root, start, end, true, flush, + zap_private &&is_private_sp (root));return flush; }@@ -891,7 +898,7 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm)* Note, kvm_tdp_mmu_zap_invalidated_roots() is gifted the TDP MMU's reference. * See kvm_tdp_mmu_get_vcpu_root_hpa(). */ -void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm) +void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm,bool skip_private) {struct kvm_mmu_page *root;@@ -916,6 +923,12 @@ void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm)* or get/put references to roots. */ list_for_each_entry (root, &kvm->arch.tdp_mmu_roots, link) { +/* +* Skip private root since private page table +* is only torn down when VM is destroyed. +*/ +if (skip_private &&is_private_sp (root)) +continue ;/* * Note, invalid roots can outlive a memslot update! Invalid * roots must be *zapped* before the memslot update completes, @@ -1104,14 +1117,26 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return ret; } +/* Used by mmu notifier via kvm_unmap_gfn_range() */ bool kvm_tdp_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range,bool flush) {struct kvm_mmu_page *root; +bool zap_private = false; + +if (kvm_gfn_shared_mask (kvm)) { +if (!range->only_private && !range->only_shared) +/* attributes change */ + zap_private = !(range->arg.attributes & +KVM_MEMORY_ATTRIBUTE_PRIVATE ); +else + zap_private = range->only_private; + }__for_each_tdp_mmu_root_yield_safe (kvm, root, range->slot->as_id, false) flush =tdp_mmu_zap_leafs (kvm, root, range->start, range->end, - range->may_block, flush); + range->may_block, flush, + zap_private &&is_private_sp (root));return flush; }arch/x86/kvm/mmu/tdp_mmu.h +3
-2@@ -19,10 +19,11 @@ __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *root)void kvm_tdp_mmu_put_root (struct kvm *kvm,struct kvm_mmu_page *root); -bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,gfn_t start,gfn_t end,bool flush); +bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,gfn_t start,gfn_t end,bool flush, +bool zap_private);bool kvm_tdp_mmu_zap_sp (struct kvm *kvm,struct kvm_mmu_page *sp);void kvm_tdp_mmu_zap_all (struct kvm *kvm); -void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm); +void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm,bool skip_private);void kvm_tdp_mmu_zap_invalidated_roots (struct kvm *kvm);int kvm_tdp_mmu_map (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);
PATCH v18 49/121 KVM: x86/tdp_mmu: Apply mmu notifier callback to only shared GPA
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The private GPAs that typically guest memfd backs aren't subject to MMU notifier because it isn't mapped into virtual address of user process. kvm_tdp_mmu_handle_gfn() handles the callback of the MMU notifier, clear_flush_young(), clear_young(), test_gfn() and change_pte(). Make kvm_tdp_mmu_handle_gfn() aware of private mapping and skip private mapping.
Even with AS_UNMOVABLE set, those mmu notifier are called. For example, ksmd triggers change_pte().
---
v18:
- newly added
--- arch/x86/kvm/mmu/tdp_mmu.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/tdp_mmu.c +21 -1
@@ -1157,9 +1157,29 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm,* into this helper allow blocking; it'd be dead, wasteful code. */ for_each_tdp_mmu_root (kvm, root, range->slot->as_id) { +gfn_t start, end; + +/* +* This function is called on behalf of mmu_notifier of +* clear_flush_young(), clear_young(), test_gfn(), and +* change_pte(). They apply to only shared GPAs. +*/ +WARN_ON_ONCE (range->only_private); +WARN_ON_ONCE (!range->only_shared); +if (is_private_sp (root)) +continue ; + +/* +* For TDX shared mapping, set GFN shared bit to the range, +* so the handler() doesn't need to set it, to avoid duplicated +* code in multiple handler()s. +*/ + start =kvm_gfn_to_shared (kvm, range->start); + end =kvm_gfn_to_shared (kvm, range->end); +rcu_read_lock (); -tdp_root_for_each_leaf_pte (iter, root, range->start, range->end) +tdp_root_for_each_leaf_pte (iter, root, start, end) ret |=handler (kvm, &iter, range);rcu_read_unlock ();
Signature · 1 line
2.25.1
Re: [PATCH v18 049/121] KVM: x86/tdp_mmu: Apply mmu notifier callback to only shared GPA
Quoted 6 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
The private GPAs that typically guest memfd backs aren't subject to MMU notifier because it isn't mapped into virtual address of user process.
kvm_tdp_mmu_handle_gfn()handles the callback of the MMU notifier,clear_flush_young(),clear_young(),test_gfn()andchange_pte(). Make
Nit: test_gfn() -> test_young(), and a same typo in the comment below.
Quoted 18 lines
kvm_tdp_mmu_handle_gfn()aware of private mapping and skip private mapping.Even with
AS_UNMOVABLEset, those mmu notifier are called. For example, ksmd triggerschange_pte().Signed-off-by Isaku Yamahata <[email protected]>---
v18:
- newly added--- arch/x86/kvm/mmu/tdp_mmu.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)arch/x86/kvm/mmu/tdp_mmu.c +0
-0
arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, · line 1164
Quoted 8 lines
arch/x86/kvm/mmu/tdp_mmu.c +5
-0@@ -1157,9 +1157,29 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm,* into this helper allow blocking; it'd be dead, wasteful code. */ for_each_tdp_mmu_root (kvm, root, range->slot->as_id) { +gfn_t start, end; + +/* +* This function is called on behalf of mmu_notifier of +* clear_flush_young(), clear_young(), test_gfn(), and
test_gfn() should be test_young()
Quoted 22 lines
arch/x86/kvm/mmu/tdp_mmu.c +16
-1+* change_pte(). They apply to only shared GPAs. +*/ +WARN_ON_ONCE (range->only_private); +WARN_ON_ONCE (!range->only_shared); +if (is_private_sp (root)) +continue ; + +/* +* For TDX shared mapping, set GFN shared bit to the range, +* so the handler() doesn't need to set it, to avoid duplicated +* code in multiple handler()s. +*/ + start =kvm_gfn_to_shared (kvm, range->start); + end =kvm_gfn_to_shared (kvm, range->end); +rcu_read_lock (); -tdp_root_for_each_leaf_pte (iter, root, range->start, range->end) +tdp_root_for_each_leaf_pte (iter, root, start, end) ret |=handler (kvm, &iter, range);rcu_read_unlock ();
PATCH v18 50/121 KVM: x86/tdp_mmu: Sprinkle __must_check
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDP MMU allows tdp_mmu_set_spte_atomic() and tdp_mmu_zap_spte_atomic() to return -EBUSY or -EAGAIN error. The caller must check the return value and retry. Sprinkle __must_check to guarantee it.
--- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +5 -5
@@ -507,9 +507,9 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,* no side-effects other than setting iter->old_spte to the last * known value of the spte. */ -static inline int tdp_mmu_set_spte_atomic (struct kvm *kvm, -struct tdp_iter *iter, -u64 new_spte) +static inline int __must_check tdp_mmu_set_spte_atomic (struct kvm *kvm, +struct tdp_iter *iter, +u64 new_spte) {u64 *sptep =rcu_dereference (iter->sptep);
@@ -539,8 +539,8 @@ static inline int tdp_mmu_set_spte_atomic(struct kvm *kvm,return 0 ; } -static inline int tdp_mmu_zap_spte_atomic (struct kvm *kvm, -struct tdp_iter *iter) +static inline int __must_check tdp_mmu_zap_spte_atomic (struct kvm *kvm, +struct tdp_iter *iter) {int ret;
Signature · 1 line
2.25.1
Re: [PATCH v18 050/121] KVM: x86/tdp_mmu: Sprinkle __must_check
Quoted 5 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
TDP MMU allows
tdp_mmu_set_spte_atomic()andtdp_mmu_zap_spte_atomic()to return-EBUSYor-EAGAINerror. The caller must check the return value and retry. Sprinkle__must_checkto guarantee it.
Quoted 34 lines
Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)arch/x86/kvm/mmu/tdp_mmu.c +5
-5@@ -507,9 +507,9 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,* no side-effects other than setting iter->old_spte to the last * known value of the spte. */ -static inline int tdp_mmu_set_spte_atomic (struct kvm *kvm, -struct tdp_iter *iter, -u64 new_spte) +static inline int __must_check tdp_mmu_set_spte_atomic (struct kvm *kvm, +struct tdp_iter *iter, +u64 new_spte) {u64 *sptep =rcu_dereference (iter->sptep);@@ -539,8 +539,8 @@ static inline int tdp_mmu_set_spte_atomic(struct kvm *kvm,return 0 ; } -static inline int tdp_mmu_zap_spte_atomic (struct kvm *kvm, -struct tdp_iter *iter) +static inline int __must_check tdp_mmu_zap_spte_atomic (struct kvm *kvm, +struct tdp_iter *iter) {int ret;
PATCH v18 51/121 KVM: x86/tdp_mmu: Support TDX private mapping for TDP MMU
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Allocate protected page table for private page table, and add hooks to operate on protected page table. This patch adds allocation/free of protected page tables and hooks. When calling hooks to update SPTE entry, freeze the entry, call hooks and unfreeze the entry to allow concurrent updates on page tables. Which is the advantage of TDP MMU. As kvm_gfn_shared_mask() returns false always, those hooks aren't called yet with this patch.
When the faulting GPA is private, the KVM fault is called private. When resolving private KVM fault, allocate protected page table and call hooks to operate on protected page table. On the change of the private PTE entry, invoke kvm_x86_ops hook in __handle_changed_spte() to propagate the change to protected page table. The following depicts the relationship.
private KVM page fault |
| |
V |
private GPA | CPU protected EPTP
| | |
V | V
private PT root | protected PT root
| | |
V | V
private PT --hook to propagate-->protected PT
| | |
\--------------------+------\ |
| | |
| V V
| private guest page
|
|
non-encrypted memory | encrypted memory
|PT: page table
The existing KVM TDP MMU code uses atomic update of SPTE. On populating the EPT entry, atomically set the entry. However, it requires TLB shootdown to zap SPTE. To address it, the entry is frozen with the special SPTE value that clears the present bit. After the TLB shootdown, the entry is set to the eventual value (unfreeze).
For protected page table, hooks are called to update protected page table in addition to direct access to the private SPTE. For the zapping case, it works to freeze the SPTE. It can call hooks in addition to TLB shootdown. For populating the private SPTE entry, there can be a race condition without further protection
vcpu 1: populating 2M private SPTE
vcpu 2: populating 4K private SPTE
vcpu 2: TDX SEAMCALL to update 4K protected SPTE => error vcpu 1: TDX SEAMCALL to update 2M protected SPTE
To avoid the race, the frozen SPTE is utilized. Instead of atomic update of the private entry, freeze the entry, call the hook that update protected SPTE, set the entry to the final value.
Support 4K page only at this stage. 2M page support can be done in future patches.
---
v18:
- Rename freezed => frozen
v14 -> v15:
- Refined is_private condition check in kvm_tdp_mmu_map(). Add kvm_gfn_shared_mask() check.
- catch up for struct kvm_range change
--- arch/x86/include/asm/kvm-x86-ops.h | 5 + arch/x86/include/asm/kvm_host.h | 11 ++ arch/x86/kvm/mmu/mmu.c | 16 +- arch/x86/kvm/mmu/mmu_internal.h | 19 +- arch/x86/kvm/mmu/tdp_iter.h | 2 +- arch/x86/kvm/mmu/tdp_mmu.c | 308 +++++++++++++++++++++++++---- arch/x86/kvm/mmu/tdp_mmu.h | 2 +- virt/kvm/kvm_main.c | 1 + 8 files changed, 325 insertions(+), 39 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +5 -0
@@ -101,6 +101,11 @@ KVM_X86_OP_OPTIONAL_RET0(set_tss_addr) KVM_X86_OP_OPTIONAL_RET0(set_identity_map_addr) KVM_X86_OP_OPTIONAL_RET0(get_mt_mask) KVM_X86_OP(load_mmu_pgd) +KVM_X86_OP_OPTIONAL(link_private_spt) +KVM_X86_OP_OPTIONAL(free_private_spt) +KVM_X86_OP_OPTIONAL(set_private_spte) +KVM_X86_OP_OPTIONAL(remove_private_spte) +KVM_X86_OP_OPTIONAL(zap_private_spte) KVM_X86_OP(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier)
arch/x86/include/asm/kvm_host.h +11 -0
@@ -484,6 +484,7 @@ struct kvm_mmu {int (*sync_spte)(struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp,int i);struct kvm_mmu_root_info root; +hpa_t private_root_hpa;union kvm_cpu_role cpu_role;union kvm_mmu_page_role root_role;
@@ -1757,6 +1758,16 @@ struct kvm_x86_ops {void (*load_mmu_pgd)(struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level); +int (*link_private_spt)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +void *private_spt); +int (*free_private_spt)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +void *private_spt); +int (*set_private_spte)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +kvm_pfn_t pfn); +int (*remove_private_spte)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +kvm_pfn_t pfn); +int (*zap_private_spte)(struct kvm *kvm,gfn_t gfn,enum pg_level level); +bool (*has_wbinvd_exit)(void );u64 (*get_l2_tsc_offset)(struct kvm_vcpu *vcpu);
arch/x86/kvm/mmu/mmu.c +14 -2
@@ -3715,7 +3715,12 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) goto out_unlock;if (tdp_mmu_enabled) { - root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu); +if (kvm_gfn_shared_mask (vcpu->kvm) && + !VALID_PAGE (mmu->private_root_hpa)) { + root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu, true); + mmu->private_root_hpa = root; + } + root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu, false); mmu->root.hpa = root; }else if (shadow_root_level >=PT64_ROOT_4LEVEL ) { root =mmu_alloc_root (vcpu,0 ,0 , shadow_root_level);
@@ -4613,7 +4618,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)if (kvm_mmu_honors_guest_mtrrs (vcpu->kvm)) {for ( ; fault->max_level >PG_LEVEL_4K ; --fault->max_level) {int page_num =KVM_PAGES_PER_HPAGE (fault->max_level); -gfn_t base =gfn_round_for_level (fault->gfn, +gfn_t base =gfn_round_for_level (gpa_to_gfn (fault->addr), fault->max_level);if (kvm_mtrr_check_gfn_range_consistency (vcpu, base, page_num))
@@ -6094,6 +6099,7 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) mmu->root.hpa =INVALID_PAGE ; mmu->root.pgd =0 ; + mmu->private_root_hpa =INVALID_PAGE ;for (i =0 ; i <KVM_MMU_NUM_PREV_ROOTS ; i++) mmu->prev_roots[i] =KVM_MMU_ROOT_INFO_INVALID ;
@@ -7141,6 +7147,12 @@ int kvm_mmu_vendor_module_init(void)void kvm_mmu_destroy (struct kvm_vcpu *vcpu) {kvm_mmu_unload (vcpu); +if (tdp_mmu_enabled) { +write_lock (&vcpu->kvm->mmu_lock); +mmu_free_root_page (vcpu->kvm, &vcpu->arch.mmu->private_root_hpa, +NULL ); +write_unlock (&vcpu->kvm->mmu_lock); + }free_mmu_pages (&vcpu->arch.root_mmu);free_mmu_pages (&vcpu->arch.guest_mmu);mmu_free_memory_caches (vcpu);
arch/x86/kvm/mmu/mmu_internal.h +18 -1
@@ -6,6 +6,8 @@#include <linux/kvm_host.h> #include <asm/kvm_host.h> +#include "mmu.h" +#ifdef CONFIG_KVM_PROVE_MMU #define KVM_MMU_WARN_ON (x) WARN_ON_ONCE(x)#else
@@ -206,6 +208,15 @@ static inline void kvm_mmu_free_private_spt(struct kvm_mmu_page *sp)if (sp->private_spt) free_page((unsigned long)sp->private_spt); } + +static inline gfn_t kvm_gfn_for_root (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t gfn) +{ +if (is_private_sp (root)) +return kvm_gfn_to_private (kvm, gfn); +else +return kvm_gfn_to_shared (kvm, gfn); +}#else static inline void *kvm_mmu_private_spt (struct kvm_mmu_page *sp) {
@@ -223,6 +234,12 @@ static inline void kvm_mmu_alloc_private_spt(struct kvm_vcpu *vcpu, struct kvm_mstatic inline void kvm_mmu_free_private_spt (struct kvm_mmu_page *sp) { } + +static inline gfn_t kvm_gfn_for_root (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t gfn) +{ +return gfn; +}#endif static inline bool kvm_mmu_page_ad_need_write_protect (struct kvm_mmu_page *sp)
@@ -383,7 +400,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int r;if (vcpu->arch.mmu->root_role.direct) { - fault.gfn = fault.addr >>PAGE_SHIFT ; + fault.gfn =gpa_to_gfn (fault.addr) & ~kvm_gfn_shared_mask (vcpu->kvm); fault.slot =kvm_vcpu_gfn_to_memslot (vcpu, fault.gfn); }
arch/x86/kvm/mmu/tdp_iter.h +1 -1
@@ -91,7 +91,7 @@ struct tdp_iter {tdp_ptep_t pt_path[PT64_ROOT_MAX_LEVEL ];/* A pointer to the current SPTE */ tdp_ptep_t sptep; -/* The lowest GFN mapped by the current SPTE */ +/* The lowest GFN (shared bits included) mapped by the current SPTE */ gfn_t gfn;/* The level of the root page given to the iterator */ int root_level;
arch/x86/kvm/mmu/tdp_mmu.c +274 -34
470 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.h +1 -1
@@ -10,7 +10,7 @@void kvm_mmu_init_tdp_mmu (struct kvm *kvm);void kvm_mmu_uninit_tdp_mmu (struct kvm *kvm); -hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu); +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu,bool private);__must_check static inline bool kvm_tdp_mmu_get_root (struct kvm_mmu_page *root) {
virt/kvm/kvm_main.c +1 -0
@@ -201,6 +201,7 @@ struct page *kvm_pfn_to_refcounted_page(kvm_pfn_t pfn)return NULL ; } +EXPORT_SYMBOL_GPL (kvm_pfn_to_refcounted_page);/* * Switches to specified vcpu, until a matching vcpu_put()
Signature · 1 line
2.25.1
PATCH v18 52/121 The start of TDX KVM patch series: TDX EPT violation
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TDX EPT violation.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -2
@@ -20,11 +20,11 @@ Patch Layer status * TDX architectural definitions: Applied * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied -* TDX EPT violation: Not yet +* TDX EPT violation: Applying * TD finalization: Not yet * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet * KVM MMU GPA shared bits: Applied * KVM TDP refactoring for TDX: Applied -* KVM TDP MMU hooks: Applying +* KVM TDP MMU hooks: Applied
Signature · 1 line
2.25.1
PATCH v18 53/121 KVM: x86/mmu: TDX: Do not enable page track for TD guest
In reply to [email protected]From: Yan Zhao <[email protected]>
TDX does not support write protection and hence page track. Though !tdp_enabled and kvm_shadow_root_allocated(kvm) are always false for TD guest, should also return false when external write tracking is enabled.
--- arch/x86/kvm/mmu/page_track.c | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/kvm/mmu/page_track.c +3 -0
@@ -22,6 +22,9 @@bool kvm_page_track_write_tracking_enabled (struct kvm *kvm) { +if (kvm->arch.vm_type ==KVM_X86_TDX_VM ) +return false; +return IS_ENABLED (CONFIG_KVM_EXTERNAL_WRITE_TRACKING ) || !tdp_enabled ||kvm_shadow_root_allocated (kvm); }
Signature · 1 line
2.25.1
Re: [PATCH v18 053/121] KVM: x86/mmu: TDX: Do not enable page track for TD guest
Quoted 6 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Yan Zhao <[email protected]>
TDX does not support write protection and hence page track. Though !tdp_enabled and
kvm_shadow_root_allocated(kvm)are always false for TD guest, should also return false when external write tracking is enabled.
Nit:
The preferred shortlog prefix format is "KVM: <topic>:", remove "TDX" from the shortlog?
"KVM: x86/mmu: Do not enable page track for TD guest" should be OK.
Quoted 21 lines
Cc Yuan Yao <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/mmu/page_track.c | 3 +++ 1 file changed, 3 insertions(+)arch/x86/kvm/mmu/page_track.c +3
-0@@ -22,6 +22,9 @@bool kvm_page_track_write_tracking_enabled (struct kvm *kvm) { +if (kvm->arch.vm_type ==KVM_X86_TDX_VM ) +return false; +return IS_ENABLED (CONFIG_KVM_EXTERNAL_WRITE_TRACKING ) || !tdp_enabled ||kvm_shadow_root_allocated (kvm); }
PATCH v18 54/121 KVM: VMX: Split out guts of EPT violation to common/exposed function
In reply to [email protected]From: Sean Christopherson <[email protected]>
The difference of TDX EPT violation is how to retrieve information, GPA, and exit qualification. To share the code to handle EPT violation, split out the guts of EPT violation handler so that VMX/TDX exit handler can call it after retrieving GPA and exit qualification.
--- arch/x86/kvm/vmx/common.h | 33 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 25 +++---------------------- 2 files changed, 36 insertions(+), 22 deletions(-) create mode 100644 arch/x86/kvm/vmx/common.h
arch/x86/kvm/vmx/common.h new file +33 -0
@@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_X86_VMX_COMMON_H +#define __KVM_X86_VMX_COMMON_H + +#include <linux/kvm_host.h> + +#include "mmu.h" + +static inline int __vmx_handle_ept_violation (struct kvm_vcpu *vcpu,gpa_t gpa, +unsigned long exit_qualification) +{ +u64 error_code; + +/* Is it a read fault? */ + error_code = (exit_qualification &EPT_VIOLATION_ACC_READ ) + ?PFERR_USER_MASK :0 ; +/* Is it a write fault? */ + error_code |= (exit_qualification &EPT_VIOLATION_ACC_WRITE ) + ?PFERR_WRITE_MASK :0 ; +/* Is it a fetch fault? */ + error_code |= (exit_qualification &EPT_VIOLATION_ACC_INSTR ) + ?PFERR_FETCH_MASK :0 ; +/* ept page table entry is present? */ + error_code |= (exit_qualification &EPT_VIOLATION_RWX_MASK ) + ?PFERR_PRESENT_MASK :0 ; + + error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ? +PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; + +return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); +} + +#endif /* __KVM_X86_VMX_COMMON_H */
arch/x86/kvm/vmx/vmx.c +3 -22
@@ -50,6 +50,7 @@#include <asm/vmx.h> #include "capabilities.h" +#include "common.h" #include "cpuid.h" #include "hyperv.h" #include "kvm_onhyperv.h"
@@ -5779,11 +5780,8 @@ static int handle_task_switch(struct kvm_vcpu *vcpu)static int handle_ept_violation (struct kvm_vcpu *vcpu) { -unsigned long exit_qualification; +unsigned long exit_qualification =vmx_get_exit_qual (vcpu);gpa_t gpa; -u64 error_code; - - exit_qualification =vmx_get_exit_qual (vcpu);/* * EPT violation happened while executing iret from NMI,
@@ -5798,23 +5796,6 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu) gpa =vmcs_read64 (GUEST_PHYSICAL_ADDRESS );trace_kvm_page_fault (vcpu, gpa, exit_qualification); - -/* Is it a read fault? */ - error_code = (exit_qualification &EPT_VIOLATION_ACC_READ ) - ?PFERR_USER_MASK :0 ; -/* Is it a write fault? */ - error_code |= (exit_qualification &EPT_VIOLATION_ACC_WRITE ) - ?PFERR_WRITE_MASK :0 ; -/* Is it a fetch fault? */ - error_code |= (exit_qualification &EPT_VIOLATION_ACC_INSTR ) - ?PFERR_FETCH_MASK :0 ; -/* ept page table entry is present? */ - error_code |= (exit_qualification &EPT_VIOLATION_RWX_MASK ) - ?PFERR_PRESENT_MASK :0 ; - - error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ? -PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; - vcpu->arch.exit_qualification = exit_qualification;/*
@@ -5828,7 +5809,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)if (unlikely (allow_smaller_maxphyaddr && !kvm_vcpu_is_legal_gpa (vcpu, gpa)))return kvm_emulate_instruction (vcpu,0 ); -return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); +return __vmx_handle_ept_violation (vcpu, gpa, exit_qualification); }static int handle_ept_misconfig (struct kvm_vcpu *vcpu)
Signature · 1 line
2.25.1
Re: [PATCH v18 054/121] KVM: VMX: Split out guts of EPT violation to common/exposed function
Quoted 6 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Sean Christopherson <[email protected]>
The difference of TDX EPT violation is how to retrieve information, GPA, and exit qualification. To share the code to handle EPT violation, split out the guts of EPT violation handler so that VMX/TDX exit handler can call it after retrieving GPA and exit qualification.
Quoted 108 lines
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Reviewed-by Paolo Bonzini <[email protected]>Reviewed-by Kai Huang <[email protected]>--- arch/x86/kvm/vmx/common.h | 33 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 25 +++---------------------- 2 files changed, 36 insertions(+), 22 deletions(-)create mode 100644
arch/x86/kvm/vmx/common.harch/x86/kvm/vmx/common.h new file +33
-0@@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_X86_VMX_COMMON_H +#define __KVM_X86_VMX_COMMON_H + +#include <linux/kvm_host.h> + +#include "mmu.h" + +static inline int __vmx_handle_ept_violation (struct kvm_vcpu *vcpu,gpa_t gpa, +unsigned long exit_qualification) +{ +u64 error_code; + +/* Is it a read fault? */ + error_code = (exit_qualification &EPT_VIOLATION_ACC_READ ) + ?PFERR_USER_MASK :0 ; +/* Is it a write fault? */ + error_code |= (exit_qualification &EPT_VIOLATION_ACC_WRITE ) + ?PFERR_WRITE_MASK :0 ; +/* Is it a fetch fault? */ + error_code |= (exit_qualification &EPT_VIOLATION_ACC_INSTR ) + ?PFERR_FETCH_MASK :0 ; +/* ept page table entry is present? */ + error_code |= (exit_qualification &EPT_VIOLATION_RWX_MASK ) + ?PFERR_PRESENT_MASK :0 ; + + error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ? +PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; + +return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); +} + +#endif /* __KVM_X86_VMX_COMMON_H */ arch/x86/kvm/vmx/vmx.c +3
-22@@ -50,6 +50,7 @@#include <asm/vmx.h> #include "capabilities.h" +#include "common.h" #include "cpuid.h" #include "hyperv.h" #include "kvm_onhyperv.h" @@ -5779,11 +5780,8 @@ static int handle_task_switch(struct kvm_vcpu *vcpu)static int handle_ept_violation (struct kvm_vcpu *vcpu) { -unsigned long exit_qualification; +unsigned long exit_qualification =vmx_get_exit_qual (vcpu);gpa_t gpa; -u64 error_code; - - exit_qualification =vmx_get_exit_qual (vcpu);/* * EPT violation happened while executing iret from NMI, @@ -5798,23 +5796,6 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu) gpa =vmcs_read64 (GUEST_PHYSICAL_ADDRESS );trace_kvm_page_fault (vcpu, gpa, exit_qualification); - -/* Is it a read fault? */ - error_code = (exit_qualification &EPT_VIOLATION_ACC_READ ) - ?PFERR_USER_MASK :0 ; -/* Is it a write fault? */ - error_code |= (exit_qualification &EPT_VIOLATION_ACC_WRITE ) - ?PFERR_WRITE_MASK :0 ; -/* Is it a fetch fault? */ - error_code |= (exit_qualification &EPT_VIOLATION_ACC_INSTR ) - ?PFERR_FETCH_MASK :0 ; -/* ept page table entry is present? */ - error_code |= (exit_qualification &EPT_VIOLATION_RWX_MASK ) - ?PFERR_PRESENT_MASK :0 ; - - error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ? -PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; - vcpu->arch.exit_qualification = exit_qualification;/* @@ -5828,7 +5809,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)if (unlikely (allow_smaller_maxphyaddr && !kvm_vcpu_is_legal_gpa (vcpu, gpa)))return kvm_emulate_instruction (vcpu,0 ); -return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); +return __vmx_handle_ept_violation (vcpu, gpa, exit_qualification); }static int handle_ept_misconfig (struct kvm_vcpu *vcpu)
PATCH v18 55/121 KVM: VMX: Move setting of EPT MMU masks to common VT-x code
In reply to [email protected]From: Sean Christopherson <[email protected]>
EPT MMU masks are used commonly for VMX and TDX. The value needs to be initialized in common code before both VMX/TDX-specific initialization code.
--- arch/x86/kvm/vmx/main.c | 9 +++++++++ arch/x86/kvm/vmx/vmx.c | 4 ---- 2 files changed, 9 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/main.c +9 -0
@@ -4,6 +4,7 @@#include "x86_ops.h" #include "vmx.h" #include "nested.h" +#include "mmu.h" #include "pmu.h" #include "tdx.h" #include "tdx_arch.h"
@@ -54,6 +55,14 @@ static __init int vt_hardware_setup(void)if (ret)return ret; +/* +* As kvm_mmu_set_ept_masks() updates enable_mmio_caching, call it +* before checking enable_mmio_caching. +*/ +if (enable_ept) +kvm_mmu_set_ept_masks (enable_ept_ad_bits, +cpu_has_vmx_ept_execute_only ()); + enable_tdx = enable_tdx && !tdx_hardware_setup (&vt_x86_ops);return 0 ;
arch/x86/kvm/vmx/vmx.c +0 -4
@@ -8452,10 +8452,6 @@ __init int vmx_hardware_setup(void)set_bit (0 , vmx_vpid_bitmap);/* 0 is reserved for host */ -if (enable_ept) -kvm_mmu_set_ept_masks (enable_ept_ad_bits, -cpu_has_vmx_ept_execute_only ()); -/* * Setup shadow_me_value/shadow_me_mask to include MKTME KeyID * bits to shadow_zero_check.
Signature · 1 line
2.25.1
Re: [PATCH v18 055/121] KVM: VMX: Move setting of EPT MMU masks to common VT-x code
Quoted 45 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Sean Christopherson <[email protected]>
EPT MMU masks are used commonly for VMX and TDX. The value needs to be initialized in common code before both VMX/TDX-specific initialization code.
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/kvm/vmx/main.c | 9 +++++++++ arch/x86/kvm/vmx/vmx.c | 4 ---- 2 files changed, 9 insertions(+), 4 deletions(-)arch/x86/kvm/vmx/main.c +9
-0@@ -4,6 +4,7 @@#include "x86_ops.h" #include "vmx.h" #include "nested.h" +#include "mmu.h" #include "pmu.h" #include "tdx.h" #include "tdx_arch.h" @@ -54,6 +55,14 @@ static __init int vt_hardware_setup(void)if (ret)return ret; +/* +* As kvm_mmu_set_ept_masks() updates enable_mmio_caching, call it +* before checking enable_mmio_caching. +*/ +if (enable_ept) +kvm_mmu_set_ept_masks (enable_ept_ad_bits, +cpu_has_vmx_ept_execute_only ()); + enable_tdx = enable_tdx && !tdx_hardware_setup (&vt_x86_ops);return 0 ;arch/x86/kvm/vmx/vmx.c +0
-0
arch/x86/kvm/vmx/vmx.c __init int vmx_hardware_setup(void) · line 8458
Quoted 7 lines
arch/x86/kvm/vmx/vmx.c +0
-4@@ -8452,10 +8452,6 @@ __init int vmx_hardware_setup(void)set_bit (0 , vmx_vpid_bitmap);/* 0 is reserved for host */ -if (enable_ept) -kvm_mmu_set_ept_masks (enable_ept_ad_bits, -cpu_has_vmx_ept_execute_only ()); -
From hardware_setup aspect, vmx_hardware_setup() is the dependency of tdx_hardware_setup() and vmx_hardware_setup() is called earlier than tdx_hardware_setup(), it seems no need to move the code.
Quoted 3 lines
arch/x86/kvm/vmx/vmx.c +0
-0/* * Setup shadow_me_value/shadow_me_mask to include MKTME KeyID * bits to shadow_zero_check.
PATCH v18 56/121 KVM: TDX: Add accessors VMX VMCS helpers
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX defines SEAMCALL APIs to access TDX control structures corresponding to the VMX VMCS. Introduce helper accessors to hide its SEAMCALL ABI details.
--- arch/x86/kvm/vmx/tdx.h | 95 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+)
arch/x86/kvm/vmx/tdx.h +95 -0
@@ -58,6 +58,101 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu) return container_of(vcpu, struct vcpu_tdx, vcpu); } +static __always_inline void tdvps_vmcs_check(u32 field, u8 bits) +{ +#define VMCS_ENC_ACCESS_TYPE_MASK 0x1UL +#define VMCS_ENC_ACCESS_TYPE_FULL 0x0UL +#define VMCS_ENC_ACCESS_TYPE_HIGH 0x1UL +#define VMCS_ENC_ACCESS_TYPE (field) ((field) & VMCS_ENC_ACCESS_TYPE_MASK) + +/* TDX is 64bit only. HIGH field isn't supported. */ +BUILD_BUG_ON_MSG (__builtin_constant_p (field) && +VMCS_ENC_ACCESS_TYPE (field) ==VMCS_ENC_ACCESS_TYPE_HIGH , +"Read/Write to TD VMCS *_HIGH fields not supported" ); + +BUILD_BUG_ON (bits !=16 && bits !=32 && bits !=64 ); + +#define VMCS_ENC_WIDTH_MASK GENMASK(14, 13) +#define VMCS_ENC_WIDTH_16BIT (0UL << 13) +#define VMCS_ENC_WIDTH_64BIT (1UL << 13) +#define VMCS_ENC_WIDTH_32BIT (2UL << 13) +#define VMCS_ENC_WIDTH_NATURAL (3UL << 13) +#define VMCS_ENC_WIDTH (field) ((field) & VMCS_ENC_WIDTH_MASK) + +/* TDX is 64bit only. i.e. natural width = 64bit. */ +BUILD_BUG_ON_MSG (bits !=64 &&__builtin_constant_p (field) && + (VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_64BIT || +VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_NATURAL ), +"Invalid TD VMCS access for 64-bit field" ); +BUILD_BUG_ON_MSG (bits !=32 &&__builtin_constant_p (field) && +VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_32BIT , +"Invalid TD VMCS access for 32-bit field" ); +BUILD_BUG_ON_MSG (bits !=16 &&__builtin_constant_p (field) && +VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_16BIT , +"Invalid TD VMCS access for 16-bit field" ); +} + +static __always_inline void tdvps_state_non_arch_check(u64 field, u8 bits) {} +static __always_inline void tdvps_management_check(u64 field, u8 bits) {} + +#define TDX_BUILD_TDVPS_ACCESSORS (bits, uclass, lclass) \ +static __always_inline u##bits td_##lclass##_read##bits(struct vcpu_tdx *tdx, \ + u32 field) \ +{ \ + struct tdx_module_args out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_rd(tdx->tdvpr_pa, TDVPS_##uclass(field), &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) { \ + pr_err("TDH_VP_RD["#uclass".0x%x] failed: 0x%llx\n", \ + field, err); \ + return 0; \ + } \ + return (u##bits)out.r8; \ +} \ +static __always_inline void td_##lclass##_write##bits(struct vcpu_tdx *tdx, \ + u32 field, u##bits val) \ +{ \ + struct tdx_module_args out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), val, \ + GENMASK_ULL(bits - 1, 0), &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] = 0x%llx failed: 0x%llx\n", \ + field, (u64)val, err); \ +} \ +static __always_inline void td_##lclass##_setbit##bits(struct vcpu_tdx *tdx, \ + u32 field, u64 bit) \ +{ \ + struct tdx_module_args out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), bit, bit, &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] |= 0x%llx failed: 0x%llx\n", \ + field, bit, err); \ +} \ +static __always_inline void td_##lclass##_clearbit##bits(struct vcpu_tdx *tdx, \ + u32 field, u64 bit) \ +{ \ + struct tdx_module_args out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), 0, bit, &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] &= ~0x%llx failed: 0x%llx\n", \ + field, bit, err); \ +} + +TDX_BUILD_TDVPS_ACCESSORS (16 ,VMCS , vmcs); +TDX_BUILD_TDVPS_ACCESSORS (32 ,VMCS , vmcs); +TDX_BUILD_TDVPS_ACCESSORS (64 ,VMCS , vmcs); + static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field) {struct tdx_module_args out;
Signature · 1 line
2.25.1
Re: [PATCH v18 056/121] KVM: TDX: Add accessors VMX VMCS helpers
Quoted 44 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
TDX defines SEAMCALL APIs to access TDX control structures corresponding to the VMX VMCS. Introduce helper accessors to hide its SEAMCALL ABI details.
Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/kvm/vmx/tdx.h | 95 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+)arch/x86/kvm/vmx/tdx.h +26
-0@@ -58,6 +58,101 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu) return container_of(vcpu, struct vcpu_tdx, vcpu); } +static __always_inline void tdvps_vmcs_check(u32 field, u8 bits) +{ +#define VMCS_ENC_ACCESS_TYPE_MASK 0x1UL +#define VMCS_ENC_ACCESS_TYPE_FULL 0x0UL +#define VMCS_ENC_ACCESS_TYPE_HIGH 0x1UL +#define VMCS_ENC_ACCESS_TYPE (field) ((field) & VMCS_ENC_ACCESS_TYPE_MASK) + +/* TDX is 64bit only. HIGH field isn't supported. */ +BUILD_BUG_ON_MSG (__builtin_constant_p (field) && +VMCS_ENC_ACCESS_TYPE (field) ==VMCS_ENC_ACCESS_TYPE_HIGH , +"Read/Write to TD VMCS *_HIGH fields not supported" ); + +BUILD_BUG_ON (bits !=16 && bits !=32 && bits !=64 ); + +#define VMCS_ENC_WIDTH_MASK GENMASK(14, 13) +#define VMCS_ENC_WIDTH_16BIT (0UL << 13) +#define VMCS_ENC_WIDTH_64BIT (1UL << 13) +#define VMCS_ENC_WIDTH_32BIT (2UL << 13) +#define VMCS_ENC_WIDTH_NATURAL (3UL << 13) +#define VMCS_ENC_WIDTH (field) ((field) & VMCS_ENC_WIDTH_MASK) + +/* TDX is 64bit only. i.e. natural width = 64bit. */ +BUILD_BUG_ON_MSG (bits !=64 &&__builtin_constant_p (field) && + (VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_64BIT || +VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_NATURAL ), +"Invalid TD VMCS access for 64-bit field" );
arch/x86/kvm/vmx/tdx.h static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu) · line 96
Quoted 10 lines
arch/x86/kvm/vmx/tdx.h +10
-0+BUILD_BUG_ON_MSG (bits !=32 &&__builtin_constant_p (field) && +VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_32BIT , +"Invalid TD VMCS access for 32-bit field" ); +BUILD_BUG_ON_MSG (bits !=16 &&__builtin_constant_p (field) && +VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_16BIT , +"Invalid TD VMCS access for 16-bit field" ); +} + +static __always_inline void tdvps_state_non_arch_check(u64 field, u8 bits) {} +static __always_inline void tdvps_management_check(u64 field, u8 bits) {}
Should this two APIs be added along with for he accessors for MANAGEMENT / STATE_NON_ARCH?
Quoted 62 lines
arch/x86/kvm/vmx/tdx.h +59
-0+ +#define TDX_BUILD_TDVPS_ACCESSORS (bits, uclass, lclass) \ +static __always_inline u##bits td_##lclass##_read##bits(struct vcpu_tdx *tdx, \ + u32 field) \ +{ \ + struct tdx_module_args out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_rd(tdx->tdvpr_pa, TDVPS_##uclass(field), &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) { \ + pr_err("TDH_VP_RD["#uclass".0x%x] failed: 0x%llx\n", \ + field, err); \ + return 0; \ + } \ + return (u##bits)out.r8; \ +} \ +static __always_inline void td_##lclass##_write##bits(struct vcpu_tdx *tdx, \ + u32 field, u##bits val) \ +{ \ + struct tdx_module_args out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), val, \ + GENMASK_ULL(bits - 1, 0), &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] = 0x%llx failed: 0x%llx\n", \ + field, (u64)val, err); \ +} \ +static __always_inline void td_##lclass##_setbit##bits(struct vcpu_tdx *tdx, \ + u32 field, u64 bit) \ +{ \ + struct tdx_module_args out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), bit, bit, &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] |= 0x%llx failed: 0x%llx\n", \ + field, bit, err); \ +} \ +static __always_inline void td_##lclass##_clearbit##bits(struct vcpu_tdx *tdx, \ + u32 field, u64 bit) \ +{ \ + struct tdx_module_args out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), 0, bit, &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] &= ~0x%llx failed: 0x%llx\n", \ + field, bit, err); \ +} + +TDX_BUILD_TDVPS_ACCESSORS (16 ,VMCS , vmcs); +TDX_BUILD_TDVPS_ACCESSORS (32 ,VMCS , vmcs); +TDX_BUILD_TDVPS_ACCESSORS (64 ,VMCS , vmcs); + static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field) {struct tdx_module_args out;
PATCH v18 57/121 KVM: TDX: Add load_mmu_pgd method for TDX
In reply to [email protected]From: Sean Christopherson <[email protected]>
For virtual IO, the guest TD shares guest pages with VMM without encryption. Shared EPT is used to map guest pages in unprotected way.
Add the VMCS field encoding for the shared EPTP, which will be used by TDX to have separate EPT walks for private GPAs (existing EPTP) versus shared GPAs (new shared EPTP).
Set shared EPT pointer value for the TDX guest to initialize TDX MMU.
--- arch/x86/include/asm/vmx.h | 1 + arch/x86/kvm/vmx/main.c | 13 ++++++++++++- arch/x86/kvm/vmx/tdx.c | 5 +++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ 4 files changed, 22 insertions(+), 1 deletion(-)
arch/x86/include/asm/vmx.h +1 -0
@@ -236,6 +236,7 @@ enum vmcs_field {TSC_MULTIPLIER_HIGH =0x00002033 ,TERTIARY_VM_EXEC_CONTROL =0x00002034 ,TERTIARY_VM_EXEC_CONTROL_HIGH =0x00002035 , +SHARED_EPT_POINTER =0x0000203C ,PID_POINTER_TABLE =0x00002042 ,PID_POINTER_TABLE_HIGH =0x00002043 , GUEST_PHYSICAL_ADDRESS = 0x00002400,
arch/x86/kvm/vmx/main.c +12 -1
@@ -147,6 +147,17 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)vmx_vcpu_reset (vcpu, init_event); } +static void vt_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa, +int pgd_level) +{ +if (is_td_vcpu (vcpu)) { +tdx_load_mmu_pgd (vcpu, root_hpa, pgd_level); +return ; + } + +vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -279,7 +290,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .write_tsc_offset = vmx_write_tsc_offset, .write_tsc_multiplier = vmx_write_tsc_multiplier, - .load_mmu_pgd = vmx_load_mmu_pgd, + .load_mmu_pgd = vt_load_mmu_pgd, .check_intercept = vmx_check_intercept, .handle_exit_irqoff = vmx_handle_exit_irqoff,
arch/x86/kvm/vmx/tdx.c +5 -0
@@ -501,6 +501,11 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)*/ } +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) +{ +td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK ); +} +static int tdx_get_capabilities (struct kvm_tdx_cmd *cmd) {struct kvm_tdx_capabilities __user *user_caps;
arch/x86/kvm/vmx/x86_ops.h +4 -0
@@ -152,6 +152,8 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp); + +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -EOPNOTSUPP ; }static inline void tdx_hardware_unsetup (void ) {}
@@ -173,6 +175,8 @@ static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {}static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; } + +static inline void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level) {}#endif #endif /* __KVM_X86_VMX_X86_OPS_H */
Signature · 1 line
2.25.1
Re: [PATCH v18 057/121] KVM: TDX: Add load_mmu_pgd method for TDX
Quoted 69 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Sean Christopherson <[email protected]>
For virtual IO, the guest TD shares guest pages with VMM without encryption. Shared EPT is used to map guest pages in unprotected way.
Add the VMCS field encoding for the shared EPTP, which will be used by TDX to have separate EPT walks for private GPAs (existing EPTP) versus shared GPAs (new shared EPTP).
Set shared EPT pointer value for the TDX guest to initialize TDX MMU.
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Reviewed-by Paolo Bonzini <[email protected]>--- arch/x86/include/asm/vmx.h | 1 + arch/x86/kvm/vmx/main.c | 13 ++++++++++++- arch/x86/kvm/vmx/tdx.c | 5 +++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ 4 files changed, 22 insertions(+), 1 deletion(-)arch/x86/include/asm/vmx.h +1
-0@@ -236,6 +236,7 @@ enum vmcs_field {TSC_MULTIPLIER_HIGH =0x00002033 ,TERTIARY_VM_EXEC_CONTROL =0x00002034 ,TERTIARY_VM_EXEC_CONTROL_HIGH =0x00002035 , +SHARED_EPT_POINTER =0x0000203C ,PID_POINTER_TABLE =0x00002042 ,PID_POINTER_TABLE_HIGH =0x00002043 , GUEST_PHYSICAL_ADDRESS = 0x00002400,arch/x86/kvm/vmx/main.c +12
-1@@ -147,6 +147,17 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)vmx_vcpu_reset (vcpu, init_event); } +static void vt_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa, +int pgd_level) +{ +if (is_td_vcpu (vcpu)) { +tdx_load_mmu_pgd (vcpu, root_hpa, pgd_level); +return ; + } + +vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))@@ -279,7 +290,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .write_tsc_offset = vmx_write_tsc_offset, .write_tsc_multiplier = vmx_write_tsc_multiplier, - .load_mmu_pgd = vmx_load_mmu_pgd, + .load_mmu_pgd = vt_load_mmu_pgd, .check_intercept = vmx_check_intercept, .handle_exit_irqoff = vmx_handle_exit_irqoff,arch/x86/kvm/vmx/tdx.c +0
-0
arch/x86/kvm/vmx/tdx.c void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) · line 506
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +3
-0@@ -501,6 +501,11 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)*/ } +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) +{ +td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK );
If we have concern about the alignment of root_hpa, shouldn't we do some check instead of masking the address quietly?
Quoted 27 lines
arch/x86/kvm/vmx/tdx.c +2
-0+} +static int tdx_get_capabilities (struct kvm_tdx_cmd *cmd) {struct kvm_tdx_capabilities __user *user_caps;arch/x86/kvm/vmx/x86_ops.h +4
-0@@ -152,6 +152,8 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp); + +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -EOPNOTSUPP ; }static inline void tdx_hardware_unsetup (void ) {}@@ -173,6 +175,8 @@ static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {}static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; } + +static inline void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level) {}#endif #endif /* __KVM_X86_VMX_X86_OPS_H */
Re: [PATCH v18 057/121] KVM: TDX: Add load_mmu_pgd method for TDX
Quoted 5 lines · Binbin Wu
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +0
-0@@ -501,6 +501,11 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) · line 506
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +3
-0*/ } +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) +{ +td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK );If we have concern about the alignment of
root_hpa, shouldn't we do some check instead of masking the address quietly?
Makes sense. I'll add WARN_ON_ONCE(root_hpa & ~PAGE_MASK)
Signature · 1 line
Isaku Yamahata <[email protected]>
PATCH v18 58/121 KVM: TDX: Retry seamcall when TDX_OPERAND_BUSY with operand SEPT
In reply to [email protected]From: Yuan Yao <[email protected]>
TDX module internally uses locks to protect internal resources. It tries to acquire the locks. If it fails to obtain the lock, it returns TDX_OPERAND_BUSY error without spin because its execution time limitation.
TDX SEAMCALL API reference describes what resources are used. It's known which TDX SEAMCALL can cause contention with which resources. VMM can avoid contention inside the TDX module by avoiding contentious TDX SEAMCALL with, for example, spinlock. Because OS knows better its process scheduling and its scalability, a lock at OS/VMM layer would work better than simply retrying TDX SEAMCALLs.
TDH.MEM.* API except for TDH.MEM.TRACK operates on a secure EPT tree and the TDX module internally tries to acquire the lock of the secure EPT tree. They return TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT in case of failure to get the lock. TDX KVM allows sept callbacks to return error so that TDP MMU layer can retry.
TDH.VP.ENTER is an exception with zero-step attack mitigation. Normally TDH.VP.ENTER uses only TD vcpu resources and it doesn't cause contention. When a zero-step attack is suspected, it obtains a secure EPT tree lock and tracks the GPAs causing a secure EPT fault. Thus TDG.VP.ENTER may result in TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT. Also TDH.MEM.* SEAMCALLs may result in TDX_OPERAN_BUSY | TDX_OPERAND_ID_SEPT.
Retry TDX TDH.MEM.* API and TDH.VP.ENTER on the error because the error is a rare event caused by zero-step attack mitigation and spinlock can not be used for TDH.VP.ENTER due to indefinite time execution.
--- arch/x86/kvm/vmx/tdx_ops.h | 48 +++++++++++++++++++++++++++++++------- 1 file changed, 39 insertions(+), 9 deletions(-)
arch/x86/kvm/vmx/tdx_ops.h +39 -9
@@ -52,6 +52,36 @@ static inline u64 tdx_seamcall(u64 op, struct tdx_module_args *in,void pr_tdx_error (u64 op,u64 error_code,const struct tdx_module_args *out);#endif +/* +* TDX module acquires its internal lock for resources. It doesn't spin to get +* locks because of its restrictions of allowed execution time. Instead, it +* returns TDX_OPERAND_BUSY with an operand id. +* +* Multiple VCPUs can operate on SEPT. Also with zero-step attack mitigation, +* TDH.VP.ENTER may rarely acquire SEPT lock and release it when zero-step +* attack is suspected. It results in TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT +* with TDH.MEM.* operation. Note: TDH.MEM.TRACK is an exception. +* +* Because TDP MMU uses read lock for scalability, spin lock around SEAMCALL +* spoils TDP MMU effort. Retry several times with the assumption that SEPT +* lock contention is rare. But don't loop forever to avoid lockup. Let TDP +* MMU retry. +*/ +#define TDX_ERROR_SEPT_BUSY (TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT) + +static inline u64 tdx_seamcall_sept (u64 op,struct tdx_module_args *in, +struct tdx_module_args *out) +{ +#define SEAMCALL_RETRY_MAX 16 +int retry =SEAMCALL_RETRY_MAX ; +u64 ret; + +do { + ret =tdx_seamcall (op, in, out); + }while (ret ==TDX_ERROR_SEPT_BUSY && retry-- >0 ); +return ret; +} +static inline u64 tdh_mng_addcx (hpa_t tdr,hpa_t addr) {struct tdx_module_args in = {
@@ -74,7 +104,7 @@ static inline u64 tdh_mem_page_add(hpa_t tdr, gpa_t gpa, hpa_t hpa, hpa_t source };clflush_cache_range (__va (hpa),PAGE_SIZE ); -return tdx_seamcall (TDH_MEM_PAGE_ADD , &in, out); +return tdx_seamcall_sept (TDH_MEM_PAGE_ADD , &in, out); } static inline u64 tdh_mem_sept_add(hpa_t tdr, gpa_t gpa, int level, hpa_t page,
@@ -87,7 +117,7 @@ static inline u64 tdh_mem_sept_add(hpa_t tdr, gpa_t gpa, int level, hpa_t page, };clflush_cache_range (__va (page),PAGE_SIZE ); -return tdx_seamcall (TDH_MEM_SEPT_ADD , &in, out); +return tdx_seamcall_sept (TDH_MEM_SEPT_ADD , &in, out); } static inline u64 tdh_mem_sept_rd(hpa_t tdr, gpa_t gpa, int level,
@@ -98,7 +128,7 @@ static inline u64 tdh_mem_sept_rd(hpa_t tdr, gpa_t gpa, int level, .rdx = tdr, }; -return tdx_seamcall (TDH_MEM_SEPT_RD , &in, out); +return tdx_seamcall_sept (TDH_MEM_SEPT_RD , &in, out); } static inline u64 tdh_mem_sept_remove(hpa_t tdr, gpa_t gpa, int level,
@@ -109,7 +139,7 @@ static inline u64 tdh_mem_sept_remove(hpa_t tdr, gpa_t gpa, int level, .rdx = tdr, }; -return tdx_seamcall (TDH_MEM_SEPT_REMOVE , &in, out); +return tdx_seamcall_sept (TDH_MEM_SEPT_REMOVE , &in, out); }static inline u64 tdh_vp_addcx (hpa_t tdvpr,hpa_t addr)
@@ -133,7 +163,7 @@ static inline u64 tdh_mem_page_relocate(hpa_t tdr, gpa_t gpa, hpa_t hpa, };clflush_cache_range (__va (hpa),PAGE_SIZE ); -return tdx_seamcall (TDH_MEM_PAGE_RELOCATE , &in, out); +return tdx_seamcall_sept (TDH_MEM_PAGE_RELOCATE , &in, out); } static inline u64 tdh_mem_page_aug(hpa_t tdr, gpa_t gpa, hpa_t hpa,
@@ -146,7 +176,7 @@ static inline u64 tdh_mem_page_aug(hpa_t tdr, gpa_t gpa, hpa_t hpa, };clflush_cache_range (__va (hpa),PAGE_SIZE ); -return tdx_seamcall (TDH_MEM_PAGE_AUG , &in, out); +return tdx_seamcall_sept (TDH_MEM_PAGE_AUG , &in, out); } static inline u64 tdh_mem_range_block(hpa_t tdr, gpa_t gpa, int level,
@@ -157,7 +187,7 @@ static inline u64 tdh_mem_range_block(hpa_t tdr, gpa_t gpa, int level, .rdx = tdr, }; -return tdx_seamcall (TDH_MEM_RANGE_BLOCK , &in, out); +return tdx_seamcall_sept (TDH_MEM_RANGE_BLOCK , &in, out); }static inline u64 tdh_mng_key_config (hpa_t tdr)
@@ -307,7 +337,7 @@ static inline u64 tdh_mem_page_remove(hpa_t tdr, gpa_t gpa, int level, .rdx = tdr, }; -return tdx_seamcall (TDH_MEM_PAGE_REMOVE , &in, out); +return tdx_seamcall_sept (TDH_MEM_PAGE_REMOVE , &in, out); }static inline u64 tdh_sys_lp_shutdown (void )
@@ -335,7 +365,7 @@ static inline u64 tdh_mem_range_unblock(hpa_t tdr, gpa_t gpa, int level, .rdx = tdr, }; -return tdx_seamcall (TDH_MEM_RANGE_UNBLOCK , &in, out); +return tdx_seamcall_sept (TDH_MEM_RANGE_UNBLOCK , &in, out); }static inline u64 tdh_phymem_cache_wb (bool resume)
Signature · 1 line
2.25.1
Re: [PATCH v18 058/121] KVM: TDX: Retry seamcall when TDX_OPERAND_BUSY with operand SEPT
Quoted 19 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Yuan Yao <[email protected]>
TDX module internally uses locks to protect internal resources. It tries to acquire the locks. If it fails to obtain the lock, it returns
TDX_OPERAND_BUSYerror without spin because its execution time limitation.TDX SEAMCALL API reference describes what resources are used. It's known which TDX SEAMCALL can cause contention with which resources. VMM can avoid contention inside the TDX module by avoiding contentious TDX SEAMCALL with, for example, spinlock. Because OS knows better its process scheduling and its scalability, a lock at OS/VMM layer would work better than simply retrying TDX SEAMCALLs.
TDH.MEM.* API except for TDH.MEM.TRACK operates on a secure EPT tree and the TDX module internally tries to acquire the lock of the secure EPT tree. They return
TDX_OPERAND_BUSY|TDX_OPERAND_ID_SEPTin case of failure to get the lock. TDX KVM allows sept callbacks to return error so that TDP MMU layer can retry.
Quoted 4 lines
TDH.VP.ENTER is an exception with zero-step attack mitigation. Normally TDH.VP.ENTER uses only TD vcpu resources and it doesn't cause contention. When a zero-step attack is suspected, it obtains a secure EPT tree lock and tracks the GPAs causing a secure EPT fault. Thus TDG.VP.ENTER may result
Should be TDH.VP.ENTER.
Quoted 2 lines
in
TDX_OPERAND_BUSY|TDX_OPERAND_ID_SEPT. Also TDH.MEM.* SEAMCALLs may result inTDX_OPERAN_BUSY|TDX_OPERAND_ID_SEPT.
s/TDX_OPERAN_BUSY/TDX_OPERAND_BUSY
Quoted 4 lines
Retry TDX TDH.MEM.* API and TDH.VP.ENTER on the error because the error is a rare event caused by zero-step attack mitigation and spinlock can not be used for TDH.VP.ENTER due to indefinite time execution.
Does it retry TDH.VP.ENTER on SEPT busy?
I didn't see the related code in this patch.
Quoted 129 lines
Signed-off-by Yuan Yao <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/kvm/vmx/tdx_ops.h | 48 +++++++++++++++++++++++++++++++------- 1 file changed, 39 insertions(+), 9 deletions(-)arch/x86/kvm/vmx/tdx_ops.h +39
-9@@ -52,6 +52,36 @@ static inline u64 tdx_seamcall(u64 op, struct tdx_module_args *in,void pr_tdx_error (u64 op,u64 error_code,const struct tdx_module_args *out);#endif +/* +* TDX module acquires its internal lock for resources. It doesn't spin to get +* locks because of its restrictions of allowed execution time. Instead, it +* returns TDX_OPERAND_BUSY with an operand id. +* +* Multiple VCPUs can operate on SEPT. Also with zero-step attack mitigation, +* TDH.VP.ENTER may rarely acquire SEPT lock and release it when zero-step +* attack is suspected. It results in TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT +* with TDH.MEM.* operation. Note: TDH.MEM.TRACK is an exception. +* +* Because TDP MMU uses read lock for scalability, spin lock around SEAMCALL +* spoils TDP MMU effort. Retry several times with the assumption that SEPT +* lock contention is rare. But don't loop forever to avoid lockup. Let TDP +* MMU retry. +*/ +#define TDX_ERROR_SEPT_BUSY (TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT) + +static inline u64 tdx_seamcall_sept (u64 op,struct tdx_module_args *in, +struct tdx_module_args *out) +{ +#define SEAMCALL_RETRY_MAX 16 +int retry =SEAMCALL_RETRY_MAX ; +u64 ret; + +do { + ret =tdx_seamcall (op, in, out); + }while (ret ==TDX_ERROR_SEPT_BUSY && retry-- >0 ); +return ret; +} +static inline u64 tdh_mng_addcx (hpa_t tdr,hpa_t addr) {struct tdx_module_args in = {@@ -74,7 +104,7 @@ static inline u64 tdh_mem_page_add(hpa_t tdr, gpa_t gpa, hpa_t hpa, hpa_t source };clflush_cache_range (__va (hpa),PAGE_SIZE ); -return tdx_seamcall (TDH_MEM_PAGE_ADD , &in, out); +return tdx_seamcall_sept (TDH_MEM_PAGE_ADD , &in, out); } static inline u64 tdh_mem_sept_add(hpa_t tdr, gpa_t gpa, int level, hpa_t page,@@ -87,7 +117,7 @@ static inline u64 tdh_mem_sept_add(hpa_t tdr, gpa_t gpa, int level, hpa_t page, };clflush_cache_range (__va (page),PAGE_SIZE ); -return tdx_seamcall (TDH_MEM_SEPT_ADD , &in, out); +return tdx_seamcall_sept (TDH_MEM_SEPT_ADD , &in, out); } static inline u64 tdh_mem_sept_rd(hpa_t tdr, gpa_t gpa, int level,@@ -98,7 +128,7 @@ static inline u64 tdh_mem_sept_rd(hpa_t tdr, gpa_t gpa, int level, .rdx = tdr, }; -return tdx_seamcall (TDH_MEM_SEPT_RD , &in, out); +return tdx_seamcall_sept (TDH_MEM_SEPT_RD , &in, out); } static inline u64 tdh_mem_sept_remove(hpa_t tdr, gpa_t gpa, int level,@@ -109,7 +139,7 @@ static inline u64 tdh_mem_sept_remove(hpa_t tdr, gpa_t gpa, int level, .rdx = tdr, }; -return tdx_seamcall (TDH_MEM_SEPT_REMOVE , &in, out); +return tdx_seamcall_sept (TDH_MEM_SEPT_REMOVE , &in, out); }static inline u64 tdh_vp_addcx (hpa_t tdvpr,hpa_t addr)@@ -133,7 +163,7 @@ static inline u64 tdh_mem_page_relocate(hpa_t tdr, gpa_t gpa, hpa_t hpa, };clflush_cache_range (__va (hpa),PAGE_SIZE ); -return tdx_seamcall (TDH_MEM_PAGE_RELOCATE , &in, out); +return tdx_seamcall_sept (TDH_MEM_PAGE_RELOCATE , &in, out); } static inline u64 tdh_mem_page_aug(hpa_t tdr, gpa_t gpa, hpa_t hpa,@@ -146,7 +176,7 @@ static inline u64 tdh_mem_page_aug(hpa_t tdr, gpa_t gpa, hpa_t hpa, };clflush_cache_range (__va (hpa),PAGE_SIZE ); -return tdx_seamcall (TDH_MEM_PAGE_AUG , &in, out); +return tdx_seamcall_sept (TDH_MEM_PAGE_AUG , &in, out); } static inline u64 tdh_mem_range_block(hpa_t tdr, gpa_t gpa, int level,@@ -157,7 +187,7 @@ static inline u64 tdh_mem_range_block(hpa_t tdr, gpa_t gpa, int level, .rdx = tdr, }; -return tdx_seamcall (TDH_MEM_RANGE_BLOCK , &in, out); +return tdx_seamcall_sept (TDH_MEM_RANGE_BLOCK , &in, out); }static inline u64 tdh_mng_key_config (hpa_t tdr)@@ -307,7 +337,7 @@ static inline u64 tdh_mem_page_remove(hpa_t tdr, gpa_t gpa, int level, .rdx = tdr, }; -return tdx_seamcall (TDH_MEM_PAGE_REMOVE , &in, out); +return tdx_seamcall_sept (TDH_MEM_PAGE_REMOVE , &in, out); }static inline u64 tdh_sys_lp_shutdown (void )@@ -335,7 +365,7 @@ static inline u64 tdh_mem_range_unblock(hpa_t tdr, gpa_t gpa, int level, .rdx = tdr, }; -return tdx_seamcall (TDH_MEM_RANGE_UNBLOCK , &in, out); +return tdx_seamcall_sept (TDH_MEM_RANGE_UNBLOCK , &in, out); }static inline u64 tdh_phymem_cache_wb (bool resume)
PATCH v18 59/121 KVM: TDX: Require TDP MMU and mmio caching for TDX
In reply to [email protected]From: Isaku Yamahata <[email protected]>
As TDP MMU is becoming main stream than the legacy MMU, the legacy MMU support for TDX isn't implemented. TDX requires KVM mmio caching. Disable TDX support when TDP MMU or mmio caching aren't supported.
--- arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/vmx/main.c | 11 +++++++++++ 2 files changed, 12 insertions(+)
arch/x86/kvm/mmu/mmu.c +1 -0
@@ -104,6 +104,7 @@ module_param_named(flush_on_reuse, force_flush_and_sync_on_reuse, bool, 0644);* If the hardware supports that we don't need to do shadow paging. */ bool tdp_enabled = false; +EXPORT_SYMBOL_GPL (tdp_enabled);static bool __ro_after_init tdp_mmu_allowed;
arch/x86/kvm/vmx/main.c +11 -0
@@ -62,6 +62,17 @@ static __init int vt_hardware_setup(void)if (enable_ept)kvm_mmu_set_ept_masks (enable_ept_ad_bits,cpu_has_vmx_ept_execute_only ()); +/* TDX requires KVM TDP MMU. */ +if (enable_tdx && !tdp_enabled) { + enable_tdx = false; +pr_warn_ratelimited ("TDX requires TDP MMU. Please enable TDP MMU for TDX.\n" ); + } + +/* TDX requires MMIO caching. */ +if (enable_tdx && !enable_mmio_caching) { + enable_tdx = false; +pr_warn_ratelimited ("TDX requires mmio caching. Please enable mmio caching for TDX.\n" ); + } enable_tdx = enable_tdx && !tdx_hardware_setup (&vt_x86_ops);
Signature · 1 line
2.25.1
PATCH v18 60/121 KVM: TDX: TDP MMU TDX support
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Implement hooks of TDP MMU for TDX backend. TLB flush, TLB shootdown, propagating the change private EPT entry to Secure EPT and freeing Secure EPT page. TLB flush handles both shared EPT and private EPT. It flushes shared EPT same as VMX. It also waits for the TDX TLB shootdown. For the hook to free Secure EPT page, unlinks the Secure EPT page from the Secure EPT so that the page can be freed to OS.
Propagate the entry change to Secure EPT. The possible entry changes are present -> non-present(zapping) and non-present -> present(population). On population just link the Secure EPT page or the private guest page to the Secure EPT by TDX SEAMCALL. Because TDP MMU allows concurrent zapping/population, zapping requires synchronous TLB shoot down with the frozen EPT entry. It zaps the secure entry, increments TLB counter, sends IPI to remote vcpus to trigger TLB flush, and then unlinks the private guest page from the Secure EPT. For simplicity, batched zapping with exclude lock is handled as concurrent zapping. Although it's inefficient, it can be optimized in the future.
For MMIO SPTE, the spte value changes as follows.
initial value (suppress VE bit is set)
-> Guest issues MMIO and triggers EPT violation
-> KVM updates SPTE value to MMIO value (suppress VE bit is cleared) -> Guest MMIO resumes. It triggers VE exception in guest TD -> Guest VE handler issues TDG.VP.VMCALL<MMIO>
-> KVM handles MMIO
-> Guest VE handler resumes its execution after MMIO instruction
---
v18:
- rename tdx_sept_page_aug() -> tdx_mem_page_aug()
- checkpatch: space => tab
v15 -> v16:
- Add the handling of TD_ATTR_SEPT_VE_DISABLE case.
v14 -> v15:
- Implemented tdx_flush_tlb_current()
- Removed unnecessary invept in tdx_flush_tlb(). It was carry over from the very old code base.
--- arch/x86/kvm/mmu/spte.c | 3 +- arch/x86/kvm/vmx/main.c | 71 +++++++- arch/x86/kvm/vmx/tdx.c | 342 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 2 +- arch/x86/kvm/vmx/tdx_ops.h | 6 + arch/x86/kvm/vmx/x86_ops.h | 6 + 6 files changed, 424 insertions(+), 6 deletions(-)
arch/x86/kvm/mmu/spte.c +2 -1
@@ -74,7 +74,8 @@ u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access)u64 spte =generation_mmio_spte_mask (gen);u64 gpa = gfn <<PAGE_SHIFT ; -WARN_ON_ONCE (!vcpu->kvm->arch.shadow_mmio_value); +WARN_ON_ONCE (!vcpu->kvm->arch.shadow_mmio_value && + !kvm_gfn_shared_mask (vcpu->kvm)); access &= shadow_mmio_access_mask; spte |= vcpu->kvm->arch.shadow_mmio_value | access;
arch/x86/kvm/vmx/main.c +67 -4
97 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.c +342 -0
374 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +1 -1
@@ -18,6 +18,7 @@ struct kvm_tdx {int hkid;bool finalized; +atomic_t tdh_mem_track;u64 tsc_offset; };
@@ -165,7 +166,6 @@ static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 fiel }return out.r8; } -#else struct kvm_tdx { struct kvm kvm;
arch/x86/kvm/vmx/tdx_ops.h +6 -0
@@ -52,6 +52,12 @@ static inline u64 tdx_seamcall(u64 op, struct tdx_module_args *in,void pr_tdx_error (u64 op,u64 error_code,const struct tdx_module_args *out);#endif +static inline int pg_level_to_tdx_sept_level (enum pg_level level) +{ +WARN_ON_ONCE (level ==PG_LEVEL_NONE ); +return level -1 ; +} +/* * TDX module acquires its internal lock for resources. It doesn't spin to get * locks because of its restrictions of allowed execution time. Instead, it
arch/x86/kvm/vmx/x86_ops.h +6 -0
@@ -153,6 +153,9 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp); +void tdx_flush_tlb (struct kvm_vcpu *vcpu); +void tdx_flush_tlb_current (struct kvm_vcpu *vcpu); +int tdx_sept_flush_remote_tlbs (struct kvm *kvm);void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -EOPNOTSUPP ; }
@@ -176,6 +179,9 @@ static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; } +static inline void tdx_flush_tlb (struct kvm_vcpu *vcpu) {} +static inline void tdx_flush_tlb_current (struct kvm_vcpu *vcpu) {} +static inline int tdx_sept_flush_remote_tlbs (struct kvm *kvm) {return 0 ; }static inline void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level) {}#endif
Signature · 1 line
2.25.1
Re: [PATCH v18 060/121] KVM: TDX: TDP MMU TDX support
Quoted 91 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Implement hooks of TDP MMU for TDX backend. TLB flush, TLB shootdown, propagating the change private EPT entry to Secure EPT and freeing Secure EPT page. TLB flush handles both shared EPT and private EPT. It flushes shared EPT same as VMX. It also waits for the TDX TLB shootdown. For the hook to free Secure EPT page, unlinks the Secure EPT page from the Secure EPT so that the page can be freed to OS.
Propagate the entry change to Secure EPT. The possible entry changes are present -> non-present(zapping) and non-present -> present(population). On population just link the Secure EPT page or the private guest page to the Secure EPT by TDX SEAMCALL. Because TDP MMU allows concurrent zapping/population, zapping requires synchronous TLB shoot down with the frozen EPT entry. It zaps the secure entry, increments TLB counter, sends IPI to remote vcpus to trigger TLB flush, and then unlinks the private guest page from the Secure EPT. For simplicity, batched zapping with exclude lock is handled as concurrent zapping. Although it's inefficient, it can be optimized in the future.
For MMIO SPTE, the spte value changes as follows.
initial value (suppress VE bit is set)
-> Guest issues MMIO and triggers EPT violation
-> KVM updates SPTE value to MMIO value (suppress VE bit is cleared) -> Guest MMIO resumes. It triggers VE exception in guest TD -> Guest VE handler issues TDG.VP.VMCALL<MMIO>
-> KVM handles MMIO
-> Guest VE handler resumes its execution after MMIO instructionSigned-off-by Isaku Yamahata <[email protected]>---
v18:
- renametdx_sept_page_aug()->tdx_mem_page_aug()
- checkpatch: space => tabv15 -> v16:
- Add the handling ofTD_ATTR_SEPT_VE_DISABLEcase.v14 -> v15:
- Implementedtdx_flush_tlb_current()
- Removed unnecessary invept intdx_flush_tlb(). It was carry over from the very old code base.--- arch/x86/kvm/mmu/spte.c | 3 +- arch/x86/kvm/vmx/main.c | 71 +++++++- arch/x86/kvm/vmx/tdx.c | 342 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 2 +- arch/x86/kvm/vmx/tdx_ops.h | 6 + arch/x86/kvm/vmx/x86_ops.h | 6 + 6 files changed, 424 insertions(+), 6 deletions(-)arch/x86/kvm/mmu/spte.c +2
-1@@ -74,7 +74,8 @@ u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access)u64 spte =generation_mmio_spte_mask (gen);u64 gpa = gfn <<PAGE_SHIFT ; -WARN_ON_ONCE (!vcpu->kvm->arch.shadow_mmio_value); +WARN_ON_ONCE (!vcpu->kvm->arch.shadow_mmio_value && + !kvm_gfn_shared_mask (vcpu->kvm)); access &= shadow_mmio_access_mask; spte |= vcpu->kvm->arch.shadow_mmio_value | access;arch/x86/kvm/vmx/main.c +10
-0@@ -28,6 +28,7 @@ static int vt_max_vcpus(struct kvm *kvm)return kvm->max_vcpus; } +static int vt_flush_remote_tlbs (struct kvm *kvm);static int vt_hardware_enable (void ) {@@ -74,8 +75,22 @@ static __init int vt_hardware_setup(void)pr_warn_ratelimited ("TDX requires mmio caching. Please enable mmio caching for TDX.\n" ); } +/* +* TDX KVM overrides flush_remote_tlbs method and assumes +* flush_remote_tlbs_range = NULL that falls back to +* flush_remote_tlbs. Disable TDX if there are conflicts. +*/ +if (vt_x86_ops.flush_remote_tlbs || + vt_x86_ops.flush_remote_tlbs_range) { + enable_tdx = false; +pr_warn_ratelimited ("TDX requires baremetal. Not Supported on VMM guest.\n" );
arch/x86/kvm/vmx/main.c static __init int vt_hardware_setup(void) · line 96
Quoted 10 lines
arch/x86/kvm/vmx/main.c +5
-0+ } + enable_tdx = enable_tdx && !tdx_hardware_setup (&vt_x86_ops); +if (enable_tdx) + vt_x86_ops.flush_remote_tlbs = vt_flush_remote_tlbs; +return 0 ; }
I hit some build issues when CONFIG_HYPERV=n:
error: ‘struct kvm_x86_ops’ has no member named ‘flush_remote_tlbs’
error: ‘struct kvm_x86_ops’ has no member named ‘flush_remote_tlbs_range’
I think it should be related to the commit
https://lore.kernel.org/all/[email protected]/
Re: [PATCH v18 060/121] KVM: TDX: TDP MMU TDX support
In reply to [email protected]Quoted 51 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Implement hooks of TDP MMU for TDX backend. TLB flush, TLB shootdown, propagating the change private EPT entry to Secure EPT and freeing Secure EPT page. TLB flush handles both shared EPT and private EPT. It flushes shared EPT same as VMX. It also waits for the TDX TLB shootdown. For the hook to free Secure EPT page, unlinks the Secure EPT page from the Secure EPT so that the page can be freed to OS.
Propagate the entry change to Secure EPT. The possible entry changes are present -> non-present(zapping) and non-present -> present(population). On population just link the Secure EPT page or the private guest page to the Secure EPT by TDX SEAMCALL. Because TDP MMU allows concurrent zapping/population, zapping requires synchronous TLB shoot down with the frozen EPT entry. It zaps the secure entry, increments TLB counter, sends IPI to remote vcpus to trigger TLB flush, and then unlinks the private guest page from the Secure EPT. For simplicity, batched zapping with exclude lock is handled as concurrent zapping. Although it's inefficient, it can be optimized in the future.
For MMIO SPTE, the spte value changes as follows.
initial value (suppress VE bit is set)
-> Guest issues MMIO and triggers EPT violation
-> KVM updates SPTE value to MMIO value (suppress VE bit is cleared) -> Guest MMIO resumes. It triggers VE exception in guest TD -> Guest VE handler issues TDG.VP.VMCALL<MMIO>
-> KVM handles MMIO
-> Guest VE handler resumes its execution after MMIO instructionSigned-off-by Isaku Yamahata <[email protected]>---
v18:
- renametdx_sept_page_aug()->tdx_mem_page_aug()
- checkpatch: space => tabv15 -> v16:
- Add the handling ofTD_ATTR_SEPT_VE_DISABLEcase.v14 -> v15:
- Implementedtdx_flush_tlb_current()
- Removed unnecessary invept intdx_flush_tlb(). It was carry over from the very old code base.--- arch/x86/kvm/mmu/spte.c | 3 +- arch/x86/kvm/vmx/main.c | 71 +++++++- arch/x86/kvm/vmx/tdx.c | 342 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 2 +- arch/x86/kvm/vmx/tdx_ops.h | 6 + arch/x86/kvm/vmx/x86_ops.h | 6 + 6 files changed, 424 insertions(+), 6 deletions(-)
[...]
Quoted 33 lines
arch/x86/kvm/vmx/tdx.c +33
-0+ +/* +* TLB shoot down procedure: +* There is a global epoch counter and each vcpu has local epoch counter. +* - TDH.MEM.RANGE.BLOCK(TDR. level, range) on one vcpu +* This blocks the subsequenct creation of TLB translation on that range. +* This corresponds to clear the present bit(all RXW) in EPT entry +* - TDH.MEM.TRACK(TDR): advances the epoch counter which is global. +* - IPI to remote vcpus +* - TDExit and re-entry with TDH.VP.ENTER on remote vcpus +* - On re-entry, TDX module compares the local epoch counter with the global +* epoch counter. If the local epoch counter is older than the global epoch +* counter, update the local epoch counter and flushes TLB. +*/ +static void tdx_track (struct kvm *kvm) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +u64 err; + +KVM_BUG_ON (!is_hkid_assigned (kvm_tdx), kvm); +/* If TD isn't finalized, it's before any vcpu running. */ +if (unlikely (!is_td_finalized (kvm_tdx))) +return ; + +/* +* tdx_flush_tlb() waits for this function to issue TDH.MEM.TRACK() by +* the counter. The counter is used instead of bool because multiple +* TDH_MEM_TRACK() can be issued concurrently by multiple vcpus. +*/ +atomic_inc (&kvm_tdx->tdh_mem_track); +/* +* KVM_REQ_TLB_FLUSH waits for the empty IPI handler, ack_flush(), with +* KVM_REQUEST_WAIT.
arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 743
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+*/ +kvm_make_all_cpus_request (kvm,KVM_REQ_TLB_FLUSH ); + +do { +/* +* kvm_flush_remote_tlbs() doesn't allow to return error and +* retry. +*/ + err =tdh_mem_track (kvm_tdx->tdr_pa); + }while (unlikely ((err &TDX_SEAMCALL_STATUS_MASK ) ==TDX_OPERAND_BUSY ));
Why the sequence of the code is different from the description of the function. In the description, do the TDH.MEM.TRACK before IPIs.
But in the code, do TDH.MEM.TRACK after IPIs?
Quoted 9 lines
arch/x86/kvm/vmx/tdx.c +9
-0+ +/* Release remote vcpu waiting for TDH.MEM.TRACK in tdx_flush_tlb(). */ +atomic_dec (&kvm_tdx->tdh_mem_track); + +if (KVM_BUG_ON (err, kvm)) +pr_tdx_error (TDH_MEM_TRACK , err,NULL ); + +} +
[...]
Re: [PATCH v18 060/121] KVM: TDX: TDP MMU TDX support
Quoted 40 lines · Binbin Wu
Quoted 40 lines
arch/x86/kvm/vmx/tdx.c +40
-0+ +/* +* TLB shoot down procedure: +* There is a global epoch counter and each vcpu has local epoch counter. +* - TDH.MEM.RANGE.BLOCK(TDR. level, range) on one vcpu +* This blocks the subsequenct creation of TLB translation on that range. +* This corresponds to clear the present bit(all RXW) in EPT entry +* - TDH.MEM.TRACK(TDR): advances the epoch counter which is global. +* - IPI to remote vcpus +* - TDExit and re-entry with TDH.VP.ENTER on remote vcpus +* - On re-entry, TDX module compares the local epoch counter with the global +* epoch counter. If the local epoch counter is older than the global epoch +* counter, update the local epoch counter and flushes TLB. +*/ +static void tdx_track (struct kvm *kvm) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +u64 err; + +KVM_BUG_ON (!is_hkid_assigned (kvm_tdx), kvm); +/* If TD isn't finalized, it's before any vcpu running. */ +if (unlikely (!is_td_finalized (kvm_tdx))) +return ; + +/* +* tdx_flush_tlb() waits for this function to issue TDH.MEM.TRACK() by +* the counter. The counter is used instead of bool because multiple +* TDH_MEM_TRACK() can be issued concurrently by multiple vcpus. +*/ +atomic_inc (&kvm_tdx->tdh_mem_track); +/* +* KVM_REQ_TLB_FLUSH waits for the empty IPI handler, ack_flush(), with +* KVM_REQUEST_WAIT. +*/ +kvm_make_all_cpus_request (kvm,KVM_REQ_TLB_FLUSH ); + + do { + /* + * kvm_flush_remote_tlbs() doesn't allow to return error and + * retry.
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 743
Quoted 3 lines
arch/x86/kvm/vmx/tdx.c +3
-0+*/ + err =tdh_mem_track (kvm_tdx->tdr_pa); + }while (unlikely ((err &TDX_SEAMCALL_STATUS_MASK ) ==TDX_OPERAND_BUSY ));Why the sequence of the code is different from the description of the function.
In the description, do the TDH.MEM.TRACK before IPIs.
But in the code, do TDH.MEM.TRACK after IPIs?
It's intentional to handle IPI in parallel as we already introduced tdh_mem_track.
Signature · 1 line
Isaku Yamahata <[email protected]>
PATCH v18 61/121 KVM: TDX: MTRR: implement get_mt_mask() for TDX
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because TDX virtualize cpuid[0x1].EDX[MTRR: bit 12] to fixed 1, guest TD thinks MTRR is supported. Although TDX supports only WB for private GPA, it's desirable to support MTRR for shared GPA. As guest access to MTRR MSRs causes #VE and KVM/x86 tracks the values of MTRR MSRs, the remining part is to implement get_mt_mask method for TDX for shared GPA.
Pass around shared bit from kvm fault handler to get_mt_mask method so that it can determine if the gfn is shared or private. Implement get_mt_mask() following vmx case for shared GPA and return WB for private GPA. the existing vmx_get_mt_mask() can't be directly used as CPU state(CR0.CD) is protected. GFN passed to kvm_mtrr_check_gfn_range_consistency() should include shared bit.
--- arch/x86/kvm/vmx/main.c | 10 +++++++++- arch/x86/kvm/vmx/tdx.c | 23 +++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ 3 files changed, 34 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/main.c +9 -1
@@ -232,6 +232,14 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); } +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_mt_mask (vcpu, gfn, is_mmio); + +return vmx_get_mt_mask (vcpu, gfn, is_mmio); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -351,7 +359,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_tss_addr = vmx_set_tss_addr, .set_identity_map_addr = vmx_set_identity_map_addr, - .get_mt_mask = vmx_get_mt_mask, + .get_mt_mask = vt_get_mt_mask, .get_exit_info = vmx_get_exit_info,
arch/x86/kvm/vmx/tdx.c +23 -0
@@ -439,6 +439,29 @@ int tdx_vm_init(struct kvm *kvm)return 0 ; } +u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +{ +if (is_mmio) +return MTRR_TYPE_UNCACHABLE <<VMX_EPT_MT_EPTE_SHIFT ; + +if (!kvm_arch_has_noncoherent_dma (vcpu->kvm)) +return (MTRR_TYPE_WRBACK <<VMX_EPT_MT_EPTE_SHIFT ) |VMX_EPT_IPAT_BIT ; + +/* +* TDX enforces CR0.CD = 0 and KVM MTRR emulation enforces writeback. +* TODO: implement MTRR MSR emulation so that +* MTRRCap: SMRR=0: SMRR interface unsupported +* WC=0: write combining unsupported +* FIX=0: Fixed range registers unsupported +* VCNT=0: number of variable range regitsers = 0 +* MTRRDefType: E=1, FE=0, type=writeback only. Don't allow other value. +* E=1: enable MTRR +* FE=0: disable fixed range MTRRs +* type: default memory type=writeback +*/ +return MTRR_TYPE_WRBACK <<VMX_EPT_MT_EPTE_SHIFT ; +} +int tdx_vcpu_create (struct kvm_vcpu *vcpu) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (vcpu->kvm);
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -150,6 +150,7 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);int tdx_vcpu_create (struct kvm_vcpu *vcpu);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event); +u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -176,6 +177,7 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPstatic inline int tdx_vcpu_create (struct kvm_vcpu *vcpu) {return -EOPNOTSUPP ; }static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {} +static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
Re: [PATCH v18 061/121] KVM: TDX: MTRR: implement get_mt_mask() for TDX
Quoted 6 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Because TDX virtualize cpuid[0x1].EDX[MTRR: bit 12] to fixed 1, guest TD thinks MTRR is supported. Although TDX supports only WB for private GPA, it's desirable to support MTRR for shared GPA. As guest access to MTRR MSRs causes #VE and KVM/x86 tracks the values of MTRR MSRs, the remining
s/remining/remaining
Quoted 5 lines
part is to implement
get_mt_maskmethod for TDX for shared GPA.Pass around shared bit from kvm fault handler to
get_mt_maskmethod so that it can determine if the gfn is shared or private. Implementget_mt_mask()following vmx case for shared GPA and return WB for private GPA.
But the shared bit is not consumed in get_mt_mask()?
Quoted 94 lines
the existing
vmx_get_mt_mask()can't be directly used as CPU state(CR0.CD) is protected. GFN passed tokvm_mtrr_check_gfn_range_consistency()should include shared bit.Suggested-by Kai Huang <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/kvm/vmx/main.c | 10 +++++++++- arch/x86/kvm/vmx/tdx.c | 23 +++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ 3 files changed, 34 insertions(+), 1 deletion(-)arch/x86/kvm/vmx/main.c +9
-1@@ -232,6 +232,14 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); } +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_mt_mask (vcpu, gfn, is_mmio); + +return vmx_get_mt_mask (vcpu, gfn, is_mmio); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))@@ -351,7 +359,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_tss_addr = vmx_set_tss_addr, .set_identity_map_addr = vmx_set_identity_map_addr, - .get_mt_mask = vmx_get_mt_mask, + .get_mt_mask = vt_get_mt_mask, .get_exit_info = vmx_get_exit_info,arch/x86/kvm/vmx/tdx.c +23
-0@@ -439,6 +439,29 @@ int tdx_vm_init(struct kvm *kvm)return 0 ; } +u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +{ +if (is_mmio) +return MTRR_TYPE_UNCACHABLE <<VMX_EPT_MT_EPTE_SHIFT ; + +if (!kvm_arch_has_noncoherent_dma (vcpu->kvm)) +return (MTRR_TYPE_WRBACK <<VMX_EPT_MT_EPTE_SHIFT ) |VMX_EPT_IPAT_BIT ; + +/* +* TDX enforces CR0.CD = 0 and KVM MTRR emulation enforces writeback. +* TODO: implement MTRR MSR emulation so that +* MTRRCap: SMRR=0: SMRR interface unsupported +* WC=0: write combining unsupported +* FIX=0: Fixed range registers unsupported +* VCNT=0: number of variable range regitsers = 0 +* MTRRDefType: E=1, FE=0, type=writeback only. Don't allow other value. +* E=1: enable MTRR +* FE=0: disable fixed range MTRRs +* type: default memory type=writeback +*/ +return MTRR_TYPE_WRBACK <<VMX_EPT_MT_EPTE_SHIFT ; +} +int tdx_vcpu_create (struct kvm_vcpu *vcpu) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (vcpu->kvm);arch/x86/kvm/vmx/x86_ops.h +2
-0@@ -150,6 +150,7 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);int tdx_vcpu_create (struct kvm_vcpu *vcpu);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event); +u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);@@ -176,6 +177,7 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPstatic inline int tdx_vcpu_create (struct kvm_vcpu *vcpu) {return -EOPNOTSUPP ; }static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {} +static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Re: [PATCH v18 061/121] KVM: TDX: MTRR: implement get_mt_mask() for TDX
Quoted 12 lines · Binbin Wu
Quoted 6 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Because TDX virtualize cpuid[0x1].EDX[MTRR: bit 12] to fixed 1, guest TD thinks MTRR is supported. Although TDX supports only WB for private GPA, it's desirable to support MTRR for shared GPA. As guest access to MTRR MSRs causes #VE and KVM/x86 tracks the values of MTRR MSRs, the remining
s/remining/remaining
Quoted 7 lines
Quoted 5 lines
part is to implement
get_mt_maskmethod for TDX for shared GPA.Pass around shared bit from kvm fault handler to
get_mt_maskmethod so that it can determine if the gfn is shared or private. Implementget_mt_mask()following vmx case for shared GPA and return WB for private GPA.But the shared bit is not consumed in
get_mt_mask()?
This paragraph became stale. I eliminated this paragraph.
Signature · 1 line
Isaku Yamahata <[email protected]>
PATCH v18 62/121 The start of TDX KVM patch series: TD finalization
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TD finalization.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +3 -2
@@ -11,6 +11,7 @@ What qemu can do - TDX VM TYPE is exposed to Qemu. - Qemu can create/destroy guest of TDX vm type. - Qemu can create/destroy vcpu of TDX vm type. +- Qemu can populate initial guest memory image. Patch Layer status ------------------
@@ -20,8 +21,8 @@ Patch Layer status * TDX architectural definitions: Applied * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied -* TDX EPT violation: Applying -* TD finalization: Not yet +* TDX EPT violation: Applied +* TD finalization: Applying * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet
Signature · 1 line
2.25.1
PATCH v18 63/121 KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by TDX
In reply to [email protected]From: Sean Christopherson <[email protected]>
Introduce a helper to directly (pun intended) fault-in a TDP page without having to go through the full page fault path. This allows TDX to get the resulting pfn and also allows the RET_PF_* enums to stay in mmu.c where they belong.
---
v14 -> v15:
- Remove loop in kvm_mmu_map_tdp_page() and return error code based on RET_FP_xxx value to avoid potential infinite loop. The caller should loop on -EAGAIN instead now.
--- arch/x86/kvm/mmu.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 57 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+)
arch/x86/kvm/mmu.h +3 -0
@@ -184,6 +184,9 @@ static inline void kvm_mmu_refresh_passthrough_bits(struct kvm_vcpu *vcpu,__kvm_mmu_refresh_passthrough_bits (vcpu, mmu); } +int kvm_mmu_map_tdp_page (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code, +int max_level); +/* * Check if a given access (described through the I/D, W/R and U/S bits of a * page fault error code pfec) causes a permission fault with the given PTE
arch/x86/kvm/mmu/mmu.c +57 -0
63 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v18 063/121] KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by TDX
Quoted 68 lines · On Mon, Jan 22, 2024 at 3:56 PM <[email protected]> wrote:
From: Sean Christopherson <[email protected]>
Introduce a helper to directly (pun intended) fault-in a TDP page without having to go through the full page fault path. This allows TDX to get the resulting pfn and also allows the RET_PF_* enums to stay in
mmu.cwhere they belong.Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>---
v14 -> v15:
- Remove loop inkvm_mmu_map_tdp_page()and return error code based onRET_FP_xxxvalue to avoid potential infinite loop. The caller should loop on-EAGAINinstead now.--- arch/x86/kvm/mmu.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 57 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+)arch/x86/kvm/mmu.h +3
-0@@ -184,6 +184,9 @@ static inline void kvm_mmu_refresh_passthrough_bits(struct kvm_vcpu *vcpu,__kvm_mmu_refresh_passthrough_bits (vcpu, mmu); } +int kvm_mmu_map_tdp_page (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code, +int max_level); +/* * Check if a given access (described through the I/D, W/R and U/S bits of a * page fault error code pfec) causes a permission fault with the given PTE arch/x86/kvm/mmu/mmu.c +26
-0@@ -4635,6 +4635,63 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return direct_page_fault (vcpu, fault); } +int kvm_mmu_map_tdp_page (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code, +int max_level) +{ +int r; +struct kvm_page_fault fault = (struct kvm_page_fault ) { + .addr = gpa, + .error_code = error_code, + .exec = error_code &PFERR_FETCH_MASK , + .write = error_code &PFERR_WRITE_MASK , + .present = error_code &PFERR_PRESENT_MASK , + .rsvd = error_code &PFERR_RSVD_MASK , + .user = error_code &PFERR_USER_MASK , + .prefetch = false, + .is_tdp = true, + .is_private = error_code &PFERR_GUEST_ENC_MASK , + .nx_huge_page_workaround_enabled =is_nx_huge_page_enabled (vcpu->kvm), + }; + +WARN_ON_ONCE (!vcpu->arch.mmu->root_role.direct); + fault.gfn =gpa_to_gfn (fault.addr) & ~kvm_gfn_shared_mask (vcpu->kvm); + fault.slot =kvm_vcpu_gfn_to_memslot (vcpu, fault.gfn); + + r =mmu_topup_memory_caches (vcpu, false); +if (r) +return r; +
arch/x86/kvm/mmu/mmu.c int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · line 4673
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10
-0+ fault.max_level = max_level; + fault.req_level =PG_LEVEL_4K ; + fault.goal_level =PG_LEVEL_4K ; + +#ifdef CONFIG_X86_64 +if (tdp_mmu_enabled) + r =kvm_tdp_mmu_page_fault (vcpu, &fault); + else +#endif + r =direct_page_fault (vcpu, &fault);
Are we ever going to hit the direct_page_fault case? I thought TDX only supported tdp_mmu?
Quoted 27 lines
arch/x86/kvm/mmu/mmu.c +21
-1+ +if (is_error_noslot_pfn (fault.pfn) || vcpu->kvm->vm_bugged) +return -EFAULT ; + +switch (r) { +case RET_PF_RETRY : +return -EAGAIN ; + +case RET_PF_FIXED : +case RET_PF_SPURIOUS : +return 0 ; + +case RET_PF_CONTINUE : +case RET_PF_EMULATE : +case RET_PF_INVALID : +default : +return -EIO ; + } +} +EXPORT_SYMBOL_GPL (kvm_mmu_map_tdp_page); +static void nonpaging_init_context (struct kvm_mmu *context) { context->page_fault = nonpaging_page_fault; --2.25.1
PATCH v18 64/121 KVM: TDX: Create initial guest memory
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because the guest memory is protected in TDX, the creation of the initial guest memory requires a dedicated TDX module API, tdh_mem_page_add, instead of directly copying the memory contents into the guest memory in the case of the default VM type. KVM MMU page fault handler callback, private_page_add, handles it.
Define new subcommand, KVM_TDX_INIT_MEM_REGION, of VM-scoped KVM_MEMORY_ENCRYPT_OP. It assigns the guest page, copies the initial memory contents into the guest memory, encrypts the guest memory. At the same time, optionally it extends memory measurement of the TDX guest. It calls the KVM MMU page fault(EPT-violation) handler to trigger the callbacks for it.
---
v18:
- rename tdx_sept_page_add() -> tdx_mem_page_add().
- open code tdx_measure_page() into tdx_mem_page_add().
- remove the change of tools/arch/x86/include/uapi/asm/kvm.h.
v15 -> v16:
- add check if nr_pages isn't large with
(nr_page << PAGE_SHIFT) >> PAGE_SHIFT
v14 -> v15:
- add a check if TD is finalized or not to tdx_init_mem_region()
- return -EAGAIN when partial population
--- arch/x86/include/uapi/asm/kvm.h | 9 ++ arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/vmx/tdx.c | 160 +++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 2 + 4 files changed, 169 insertions(+), 3 deletions(-)
arch/x86/include/uapi/asm/kvm.h +9 -0
@@ -572,6 +572,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_CAPABILITIES =0 ,KVM_TDX_INIT_VM ,KVM_TDX_INIT_VCPU , +KVM_TDX_INIT_MEM_REGION , KVM_TDX_CMD_NR_MAX, };
@@ -649,4 +650,12 @@ struct kvm_tdx_init_vm { struct kvm_cpuid2 cpuid; }; +#define KVM_TDX_MEASURE_MEMORY_REGION (1UL << 0) + +struct kvm_tdx_init_mem_region { +__u64 source_addr; +__u64 gpa; +__u64 nr_pages; +}; +#endif /* _ASM_X86_KVM_H */
arch/x86/kvm/mmu/mmu.c +1 -0
@@ -5663,6 +5663,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) out:return r; } +EXPORT_SYMBOL (kvm_mmu_load);void kvm_mmu_unload (struct kvm_vcpu *vcpu) {
arch/x86/kvm/vmx/tdx.c +157 -3
184 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +2 -0
@@ -17,6 +17,8 @@ struct kvm_tdx {u64 xfam;int hkid; +hpa_t source_pa; +bool finalized;atomic_t tdh_mem_track;
Signature · 1 line
2.25.1
Sean Christopherson
maintainerRe: [PATCH v18 064/121] KVM: TDX: Create initial guest memory
Quoted 26 lines · On Mon, Jan 22, 2024, [email protected] wrote:
arch/x86/kvm/vmx/tdx.c +18
-0@@ -591,6 +591,69 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,return 0 ; } +static int tdx_mem_page_add (struct kvm *kvm,gfn_t gfn, +enum pg_level level,kvm_pfn_t pfn) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +hpa_t hpa =pfn_to_hpa (pfn); +gpa_t gpa =gfn_to_gpa (gfn); +struct tdx_module_args out; +hpa_t source_pa; +bool measure; +u64 err; +int i; + +/* +* KVM_INIT_MEM_REGION, tdx_init_mem_region(), supports only 4K page +* because tdh_mem_page_add() supports only 4K page. +*/ +if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) +return -EINVAL ;
arch/x86/kvm/vmx/tdx.c static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, · line 621
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+ +/* +* In case of TDP MMU, fault handler can run concurrently. Note +* 'source_pa' is a TD scope variable, meaning if there are multiple +* threads reaching here with all needing to access 'source_pa', it +* will break. However fortunately this won't happen, because below +* TDH_MEM_PAGE_ADD code path is only used when VM is being created +* before it is running, using KVM_TDX_INIT_MEM_REGION ioctl (which +* always uses vcpu 0's page table and protected by vcpu->mutex). +*/
Most of the above is superflous. tdx_mem_page_add() is called if and only if the TD is finalized, and the TDX module disallow running vCPUs before the TD is finalized. That's it. And maybe throw in a lockdep to assert that kvm->lock is held.
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5
-0+if (KVM_BUG_ON (kvm_tdx->source_pa ==INVALID_PAGE , kvm)) { +tdx_unpin (kvm, pfn); +return -EINVAL ; + } +
arch/x86/kvm/vmx/tdx.c static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, · line 636
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+ source_pa = kvm_tdx->source_pa & ~KVM_TDX_MEASURE_MEMORY_REGION ; + measure = kvm_tdx->source_pa &KVM_TDX_MEASURE_MEMORY_REGION ; + kvm_tdx->source_pa =INVALID_PAGE ; + + do { + err = tdh_mem_page_add(kvm_tdx->tdr_pa, gpa, hpa, source_pa, + &out); + /* + * This path is executed during populating initial guest memory + * image. i.e. before running any vcpu. Race is rare.
How are races possible at all?
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4
-0+*/ + }while (unlikely (err ==TDX_ERROR_SEPT_BUSY )); +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_MEM_PAGE_ADD , err, &out);
arch/x86/kvm/vmx/tdx.c static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, · line 650
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+tdx_unpin (kvm, pfn); +return -EIO ; + }else if (measure) { +for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) { + err =tdh_mr_extend (kvm_tdx->tdr_pa, gpa + i, &out); +if (KVM_BUG_ON (err, &kvm_tdx->kvm)) { +pr_tdx_error (TDH_MR_EXTEND , err, &out); +break ; + } + }
Why is measurement done deep within the MMU? At a glance, I don't see why this can't be done up in the ioctl, outside of a spinlock.
And IIRC, the order affects the measurement but doesn't truly matter, e.g. KVM could choose to completely separate tdh_mr_extend() from tdh_mem_page_add(), no?
Quoted 26 lines
arch/x86/kvm/vmx/tdx.c +26
-0+static int tdx_init_mem_region (struct kvm *kvm,struct kvm_tdx_cmd *cmd) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +struct kvm_tdx_init_mem_region region; +struct kvm_vcpu *vcpu; +struct page *page; +int idx, ret =0 ; +bool added = false; + +/* Once TD is finalized, the initial guest memory is fixed. */ +if (is_td_finalized (kvm_tdx)) +return -EINVAL ; + +/* The BSP vCPU must be created before initializing memory regions. */ +if (!atomic_read (&kvm->online_vcpus)) +return -EINVAL ; + +if (cmd->flags & ~KVM_TDX_MEASURE_MEMORY_REGION ) +return -EINVAL ; + +if (copy_from_user (®ion, (void __user *)cmd->data,sizeof (region))) +return -EFAULT ; + +/* Sanity check */ + if (!IS_ALIGNED(region.source_addr, PAGE_SIZE) || + !IS_ALIGNED(region.gpa, PAGE_SIZE) ||
arch/x86/kvm/vmx/tdx.c void tdx_flush_tlb_current(struct kvm_vcpu *vcpu) · line 1423
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+ !region.nr_pages || + region.nr_pages &GENMASK_ULL (63 ,63 -PAGE_SHIFT ) || + region.gpa + (region.nr_pages <<PAGE_SHIFT ) <= region.gpa || + !kvm_is_private_gpa (kvm, region.gpa) || + !kvm_is_private_gpa(kvm, region.gpa + (region.nr_pages << PAGE_SHIFT))) + return -EINVAL; + + vcpu =kvm_get_vcpu (kvm,0 ); +if (mutex_lock_killable (&vcpu->mutex)) +return -EINTR ;
The real reason for this drive-by pseudo-review is that I am hoping/wishing we can turn this into a generic KVM ioctl() to allow userspace to pre-map guest memory[*].
If we're going to carry non-trivial code, we might as well squeeze as much use out of it as we can.
Beyond wanting to shove this into KVM_MEMORY_ENCRYPT_OP, is there any reason why this is a VM ioctl() and not a vCPU ioctl()? Very roughly, couldn't we use a struct like this as input to a vCPU ioctl() that maps memory, and optionally initializes memory from @source?
struct kvm_memory_mapping {
__u64 base_gfn;
__u64 nr_pages;
__u64 flags;
__u64 source;
}TDX would need to do special things for copying the source, but beyond that most of the code in this function is generic.
Re: [PATCH v18 064/121] KVM: TDX: Create initial guest memory
+Vipin Sharma
Quoted 21 lines · Sean Christopherson
On Mon, Jan 22, 2024, [email protected] wrote:
The real reason for this drive-by pseudo-review is that I am hoping/wishing we can turn this into a generic KVM
ioctl()to allow userspace to pre-map guest memory[*].If we're going to carry non-trivial code, we might as well squeeze as much use out of it as we can.
Beyond wanting to shove this into
KVM_MEMORY_ENCRYPT_OP, is there any reason why this is a VMioctl()and not a vCPUioctl()? Very roughly, couldn't we use a struct like this as input to a vCPUioctl()that maps memory, and optionally initializes memory from @source?struct kvm_memory_mapping { __u64 base_gfn; __u64 nr_pages; __u64 flags; __u64 source; }
Quoted 4 lines
TDX would need to do special things for copying the source, but beyond that most of the code in this function is generic.
We would also be interested in such an API to reduce the guest performance impact of intra-host migration.
Re: [PATCH v18 064/121] KVM: TDX: Create initial guest memory
Quoted 24 lines · David Matlack
+Vipin Sharma
Quoted 21 lines · Sean Christopherson
On Mon, Jan 22, 2024, [email protected] wrote:
The real reason for this drive-by pseudo-review is that I am hoping/wishing we can turn this into a generic KVM
ioctl()to allow userspace to pre-map guest memory[*].If we're going to carry non-trivial code, we might as well squeeze as much use out of it as we can.
Beyond wanting to shove this into
KVM_MEMORY_ENCRYPT_OP, is there any reason why this is a VMioctl()and not a vCPUioctl()? Very roughly, couldn't we use a struct like this as input to a vCPUioctl()that maps memory, and optionally initializes memory from @source?struct kvm_memory_mapping { __u64 base_gfn; __u64 nr_pages; __u64 flags; __u64 source; }
Quoted 7 lines
Quoted 4 lines
TDX would need to do special things for copying the source, but beyond that most of the code in this function is generic.
We would also be interested in such an API to reduce the guest performance impact of intra-host migration.
I introduce KVM_MEMORY_MAPPING and KVM_CAP_MEMORY_MAPPING with v19. We can continue the discussion there.
Signature · 1 line
Isaku Yamahata <[email protected]>
Sean Christopherson
maintainerRe: [PATCH v18 064/121] KVM: TDX: Create initial guest memory
Quoted 30 lines · Isaku Yamahata
Quoted 27 lines · David Matlack
+Vipin Sharma
Quoted 24 lines · Sean Christopherson
On Mon, Jan 22, 2024, [email protected] wrote:
The real reason for this drive-by pseudo-review is that I am hoping/wishing we can turn this into a generic KVM
ioctl()to allow userspace to pre-map guest memory[*].If we're going to carry non-trivial code, we might as well squeeze as much use out of it as we can.
Beyond wanting to shove this into
KVM_MEMORY_ENCRYPT_OP, is there any reason why this is a VMioctl()and not a vCPUioctl()? Very roughly, couldn't we use a struct like this as input to a vCPUioctl()that maps memory, and optionally initializes memory from @source?struct kvm_memory_mapping { __u64 base_gfn; __u64 nr_pages; __u64 flags; __u64 source; }TDX would need to do special things for copying the source, but beyond that most of the code in this function is generic.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
We would also be interested in such an API to reduce the guest performance impact of intra-host migration.
I introduce
KVM_MEMORY_MAPPINGandKVM_CAP_MEMORY_MAPPINGwith v19.
KVM_MEMORY_MAPPING is not a good ioctl() name. There needs to be a verb in there somewhere, e.g. KVM_MAP_MEMORY, KVM_FAULTIN_MEMORY, etc.
Quoted 1 line
We can continue the discussion there.
No, we absolutely cannot continue the conversation there. That is not how kernel
development works.
Enough is enough. I am archiving v19 and not touching it.
Please post an RFC for _just_ this functionality, and follow-up in existing, pre-v19 conversations for anything else that changed between v18 and v19 and might need additional input/discussion.
Re: [PATCH v18 064/121] KVM: TDX: Create initial guest memory
Quoted 3 lines · Sean Christopherson
Please post an RFC for _just_ this functionality, and follow-up in existing, pre-v19 conversations for anything else that changed between v18 and v19 and might need additional input/discussion.
Sure, will post it. My plan is as follow for input/discussion
- Review SEV-SNP patches by Paolo for commonality
- RFC patch to KVM_MAP_MEMORY or KVM_FAULTIN_MEMORY
- RFC patch for uKVM for confidential VM
Signature · 1 line
Isaku Yamahata <[email protected]>
Sean Christopherson
maintainerRe: [PATCH v18 064/121] KVM: TDX: Create initial guest memory
Quoted 11 lines · Isaku Yamahata
Quoted 3 lines · Sean Christopherson
Please post an RFC for _just_ this functionality, and follow-up in existing, pre-v19 conversations for anything else that changed between v18 and v19 and might need additional input/discussion.
Sure, will post it. My plan is as follow for input/discussion
- Review SEV-SNP patches by Paolo for commonality
- RFC patch toKVM_MAP_MEMORYorKVM_FAULTIN_MEMORY
- RFC patch for uKVM for confidential VM
uKVM?
Re: [PATCH v18 064/121] KVM: TDX: Create initial guest memory
Quoted 8 lines · Sean Christopherson
Quoted 7 lines · Isaku Yamahata
Quoted 3 lines · Sean Christopherson
Please post an RFC for _just_ this functionality, and follow-up in existing, pre-v19 conversations for anything else that changed between v18 and v19 and might need additional input/discussion.
Quoted 6 lines
Quoted 4 lines
Sure, will post it. My plan is as follow for input/discussion
- Review SEV-SNP patches by Paolo for commonality
- RFC patch toKVM_MAP_MEMORYorKVM_FAULTIN_MEMORY
- RFC patch for uKVM for confidential VMuKVM?
I meant uAPI, sorry for typo.
Although I looked into a unified uAPI with SEV, the gain seem to be small or none. I'm currently planning to drop it based on the feedback at
https://lore.kernel.org/kvm/ZL%[email protected]/
Signature · 1 line
Isaku Yamahata <[email protected]>
Re: [PATCH v18 064/121] KVM: TDX: Create initial guest memory
In reply to [email protected]Quoted 7 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Because the guest memory is protected in TDX, the creation of the initial guest memory requires a dedicated TDX module API,
tdh_mem_page_add, instead of directly copying the memory contents into the guest memory in the case of the default VM type. KVM MMU page fault handler callback,private_page_add, handles it.
The changelog is stale? Do you mean "set_private_spte"?
Quoted 111 lines
Define new subcommand,
KVM_TDX_INIT_MEM_REGION, of VM-scopedKVM_MEMORY_ENCRYPT_OP. It assigns the guest page, copies the initial memory contents into the guest memory, encrypts the guest memory. At the same time, optionally it extends memory measurement of the TDX guest. It calls the KVM MMU page fault(EPT-violation) handler to trigger the callbacks for it.Reported-by [email protected]Signed-off-by Isaku Yamahata <[email protected]>---
v18:
- renametdx_sept_page_add()->tdx_mem_page_add().
- open codetdx_measure_page()intotdx_mem_page_add().
- remove the change oftools/arch/x86/include/uapi/asm/kvm.h.v15 -> v16:
- add check ifnr_pagesisn't large with
(nr_page<<PAGE_SHIFT) >>PAGE_SHIFTv14 -> v15:
- add a check if TD is finalized or not totdx_init_mem_region()
- return-EAGAINwhen partial population--- arch/x86/include/uapi/asm/kvm.h | 9 ++ arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/vmx/tdx.c | 160 +++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 2 + 4 files changed, 169 insertions(+), 3 deletions(-)arch/x86/include/uapi/asm/kvm.h +9
-0@@ -572,6 +572,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_CAPABILITIES =0 ,KVM_TDX_INIT_VM ,KVM_TDX_INIT_VCPU , +KVM_TDX_INIT_MEM_REGION , KVM_TDX_CMD_NR_MAX, };@@ -649,4 +650,12 @@ struct kvm_tdx_init_vm { struct kvm_cpuid2 cpuid; }; +#define KVM_TDX_MEASURE_MEMORY_REGION (1UL << 0) + +struct kvm_tdx_init_mem_region { +__u64 source_addr; +__u64 gpa; +__u64 nr_pages; +}; +#endif /* _ASM_X86_KVM_H */ arch/x86/kvm/mmu/mmu.c +1
-0@@ -5663,6 +5663,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) out:return r; } +EXPORT_SYMBOL (kvm_mmu_load);void kvm_mmu_unload (struct kvm_vcpu *vcpu) {arch/x86/kvm/vmx/tdx.c +35
-0@@ -591,6 +591,69 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,return 0 ; } +static int tdx_mem_page_add (struct kvm *kvm,gfn_t gfn, +enum pg_level level,kvm_pfn_t pfn) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +hpa_t hpa =pfn_to_hpa (pfn); +gpa_t gpa =gfn_to_gpa (gfn); +struct tdx_module_args out; +hpa_t source_pa; +bool measure; +u64 err; +int i; + +/* +* KVM_INIT_MEM_REGION, tdx_init_mem_region(), supports only 4K page +* because tdh_mem_page_add() supports only 4K page. +*/ +if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) +return -EINVAL ; + +/* +* In case of TDP MMU, fault handler can run concurrently. Note +* 'source_pa' is a TD scope variable, meaning if there are multiple +* threads reaching here with all needing to access 'source_pa', it +* will break. However fortunately this won't happen, because below +* TDH_MEM_PAGE_ADD code path is only used when VM is being created +* before it is running, using KVM_TDX_INIT_MEM_REGION ioctl (which +* always uses vcpu 0's page table and protected by vcpu->mutex). +*/ +if (KVM_BUG_ON (kvm_tdx->source_pa ==INVALID_PAGE , kvm)) { +tdx_unpin (kvm, pfn); +return -EINVAL ; + } + + source_pa = kvm_tdx->source_pa & ~KVM_TDX_MEASURE_MEMORY_REGION ; + measure = kvm_tdx->source_pa &KVM_TDX_MEASURE_MEMORY_REGION ;
arch/x86/kvm/vmx/tdx.c static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, · line 638
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+ kvm_tdx->source_pa =INVALID_PAGE ; + +do { + err =tdh_mem_page_add (kvm_tdx->tdr_pa, gpa, hpa, source_pa, + &out); +/* +* This path is executed during populating initial guest memory +* image. i.e. before running any vcpu. Race is rare. +*/ + }while (unlikely (err ==TDX_ERROR_SEPT_BUSY ));
For page add, since pages are added one by one, there should be no such error, right?
Quoted 152 lines
arch/x86/kvm/vmx/tdx.c +112
-3+if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_MEM_PAGE_ADD , err, &out); +tdx_unpin (kvm, pfn); +return -EIO ; + }else if (measure) { +for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) { + err =tdh_mr_extend (kvm_tdx->tdr_pa, gpa + i, &out); +if (KVM_BUG_ON (err, &kvm_tdx->kvm)) { +pr_tdx_error (TDH_MR_EXTEND , err, &out); +break ; + } + } + } + +return 0 ; + +} +static int tdx_sept_set_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level,kvm_pfn_t pfn) {@@ -613,9 +676,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (likely (is_td_finalized (kvm_tdx)))return tdx_mem_page_aug (kvm, gfn, level, pfn); -/* TODO: tdh_mem_page_add() comes here for the initial memory. */ - -return 0 ; +return tdx_mem_page_add (kvm, gfn, level, pfn); } static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,@@ -1322,6 +1383,96 @@ void tdx_flush_tlb_current(struct kvm_vcpu *vcpu)tdx_track (vcpu->kvm); } +#define TDX_SEPT_PFERR (PFERR_WRITE_MASK | PFERR_GUEST_ENC_MASK) + +static int tdx_init_mem_region (struct kvm *kvm,struct kvm_tdx_cmd *cmd) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +struct kvm_tdx_init_mem_region region; +struct kvm_vcpu *vcpu; +struct page *page; +int idx, ret =0 ; +bool added = false; + +/* Once TD is finalized, the initial guest memory is fixed. */ +if (is_td_finalized (kvm_tdx)) +return -EINVAL ; + +/* The BSP vCPU must be created before initializing memory regions. */ +if (!atomic_read (&kvm->online_vcpus)) +return -EINVAL ; + +if (cmd->flags & ~KVM_TDX_MEASURE_MEMORY_REGION ) +return -EINVAL ; + +if (copy_from_user (®ion, (void __user *)cmd->data,sizeof (region))) +return -EFAULT ; + +/* Sanity check */ +if (!IS_ALIGNED (region.source_addr,PAGE_SIZE ) || + !IS_ALIGNED (region.gpa,PAGE_SIZE ) || + !region.nr_pages || + region.nr_pages &GENMASK_ULL (63 ,63 -PAGE_SHIFT ) || + region.gpa + (region.nr_pages <<PAGE_SHIFT ) <= region.gpa || + !kvm_is_private_gpa (kvm, region.gpa) || + !kvm_is_private_gpa (kvm, region.gpa + (region.nr_pages <<PAGE_SHIFT ))) +return -EINVAL ; + + vcpu =kvm_get_vcpu (kvm,0 ); +if (mutex_lock_killable (&vcpu->mutex)) +return -EINTR ; + +vcpu_load (vcpu); + idx =srcu_read_lock (&kvm->srcu); + +kvm_mmu_reload (vcpu); + +while (region.nr_pages) { +if (signal_pending (current)) { + ret = -ERESTARTSYS ; +break ; + } + +if (need_resched ()) +cond_resched (); + +/* Pin the source page. */ + ret =get_user_pages_fast (region.source_addr,1 ,0 , &page); +if (ret <0 ) +break ; +if (ret !=1 ) { + ret = -ENOMEM ; +break ; + } + + kvm_tdx->source_pa =pfn_to_hpa (page_to_pfn (page)) | + (cmd->flags &KVM_TDX_MEASURE_MEMORY_REGION ); + + ret =kvm_mmu_map_tdp_page (vcpu, region.gpa,TDX_SEPT_PFERR , +PG_LEVEL_4K ); +put_page (page); +if (ret) +break ; + + region.source_addr +=PAGE_SIZE ; + region.gpa +=PAGE_SIZE ; + region.nr_pages--; + added = true; + } + +srcu_read_unlock (&kvm->srcu, idx); +vcpu_put (vcpu); + +mutex_unlock (&vcpu->mutex); + +if (added && region.nr_pages >0 ) + ret = -EAGAIN ; +if (copy_to_user ((void __user *)cmd->data, ®ion,sizeof (region))) + ret = -EFAULT ; + +return ret; +} +int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {struct kvm_tdx_cmd tdx_cmd;@@ -1341,6 +1492,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)case KVM_TDX_INIT_VM : r =tdx_td_init (kvm, &tdx_cmd);break ; +case KVM_TDX_INIT_MEM_REGION : + r =tdx_init_mem_region (kvm, &tdx_cmd); +break ;default : r = -EINVAL ; goto out;arch/x86/kvm/vmx/tdx.h +2
-0@@ -17,6 +17,8 @@ struct kvm_tdx {u64 xfam;int hkid; +hpa_t source_pa; +bool finalized;atomic_t tdh_mem_track;
PATCH v18 65/121 KVM: TDX: Finalize VM initialization
In reply to [email protected]From: Isaku Yamahata <[email protected]>
To protect the initial contents of the guest TD, the TDX module measures the guest TD during the build process as SHA-384 measurement. The measurement of the guest TD contents needs to be completed to make the guest TD ready to run.
Add a new subcommand, KVM_TDX_FINALIZE_VM, for VM-scoped KVM_MEMORY_ENCRYPT_OP to finalize the measurement and mark the TDX VM ready to run.
---
v18:
- Remove the change of tools/arch/x86/include/uapi/asm/kvm.h.
v14 -> v15:
- removed unconditional tdx_track() by tdx_flush_tlb_current() that does tdx_track().
--- arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/vmx/tdx.c | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+)
arch/x86/include/uapi/asm/kvm.h +1 -0
@@ -573,6 +573,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_INIT_VM ,KVM_TDX_INIT_VCPU ,KVM_TDX_INIT_MEM_REGION , +KVM_TDX_FINALIZE_VM , KVM_TDX_CMD_NR_MAX, };
arch/x86/kvm/vmx/tdx.c +21 -0
@@ -1473,6 +1473,24 @@ static int tdx_init_mem_region(struct kvm *kvm, struct kvm_tdx_cmd *cmd)return ret; } +static int tdx_td_finalizemr (struct kvm *kvm) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +u64 err; + +if (!is_hkid_assigned (kvm_tdx) ||is_td_finalized (kvm_tdx)) +return -EINVAL ; + + err =tdh_mr_finalize (kvm_tdx->tdr_pa); +if (WARN_ON_ONCE (err)) { +pr_tdx_error (TDH_MR_FINALIZE , err,NULL ); +return -EIO ; + } + + kvm_tdx->finalized = true; +return 0 ; +} +int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {struct kvm_tdx_cmd tdx_cmd;
@@ -1495,6 +1513,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)case KVM_TDX_INIT_MEM_REGION : r =tdx_init_mem_region (kvm, &tdx_cmd);break ; +case KVM_TDX_FINALIZE_VM : + r =tdx_td_finalizemr (kvm); +break ;default : r = -EINVAL ; goto out;
Signature · 1 line
2.25.1
PATCH v18 66/121 The start of TDX KVM patch series: TD vcpu enter/exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TD vcpu enter/exit.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +3 -2
@@ -12,6 +12,7 @@ What qemu can do - Qemu can create/destroy guest of TDX vm type. - Qemu can create/destroy vcpu of TDX vm type. - Qemu can populate initial guest memory image. +- Qemu can finalize guest TD. Patch Layer status ------------------
@@ -22,8 +23,8 @@ Patch Layer status * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied * TDX EPT violation: Applied -* TD finalization: Applying -* TD vcpu enter/exit: Not yet +* TD finalization: Applied +* TD vcpu enter/exit: Applying * TD vcpu interrupts/exit/hypercall: Not yet * KVM MMU GPA shared bits: Applied
Signature · 1 line
2.25.1
PATCH v18 67/121 KVM: TDX: Implement TDX vcpu enter/exit path
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This patch implements running TDX vcpu. Once vcpu runs on the logical processor (LP), the TDX vcpu is associated with it. When the TDX vcpu moves to another LP, the TDX vcpu needs to flush its status on the LP. When destroying TDX vcpu, it needs to complete flush and flush cpu memory cache. Track which LP the TDX vcpu run and flush it as necessary.
Do nothing on sched_in event as TDX doesn't support pause loop.
TDX vcpu execution requires restoring PMU debug store after returning back to KVM because the TDX module unconditionally resets the value. To reuse the existing code, export perf_restore_debug_store.
---
Changes v15 -> v16:
- use __seamcall_saved_ret()
- As struct tdx_module_args doesn't match with vcpu.arch.regs, copy regs before/after calling __seamcall_saved_ret().
--- arch/x86/kvm/vmx/main.c | 21 +++++++++- arch/x86/kvm/vmx/tdx.c | 84 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 33 +++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 2 + arch/x86/kvm/x86.c | 1 + 5 files changed, 139 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +19 -2
@@ -173,6 +173,23 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)vmx_vcpu_reset (vcpu, init_event); } +static int vt_vcpu_pre_run (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +/* Unconditionally continue to vcpu_run(). */ +return 1 ; + +return vmx_vcpu_pre_run (vcpu); +} + +static fastpath_t vt_vcpu_run (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_vcpu_run (vcpu); + +return vmx_vcpu_run (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) {
@@ -325,8 +342,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .flush_tlb_gva = vt_flush_tlb_gva, .flush_tlb_guest = vt_flush_tlb_guest, - .vcpu_pre_run = vmx_vcpu_pre_run, - .vcpu_run = vmx_vcpu_run, + .vcpu_pre_run = vt_vcpu_pre_run, + .vcpu_run = vt_vcpu_run, .handle_exit = vmx_handle_exit, .skip_emulated_instruction = vmx_skip_emulated_instruction, .update_emulated_instruction = vmx_update_emulated_instruction,
arch/x86/kvm/vmx/tdx.c +84 -0
96 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +33 -0
@@ -25,6 +25,37 @@ struct kvm_tdx {u64 tsc_offset; }; +union tdx_exit_reason { +struct { +/* 31:0 mirror the VMX Exit Reason format */ +u64 basic :16 ; +u64 reserved16 :1 ; +u64 reserved17 :1 ; +u64 reserved18 :1 ; +u64 reserved19 :1 ; +u64 reserved20 :1 ; +u64 reserved21 :1 ; +u64 reserved22 :1 ; +u64 reserved23 :1 ; +u64 reserved24 :1 ; +u64 reserved25 :1 ; +u64 bus_lock_detected :1 ; +u64 enclave_mode :1 ; +u64 smi_pending_mtf :1 ; +u64 smi_from_vmx_root :1 ; +u64 reserved30 :1 ; +u64 failed_vmentry :1 ; + +/* 63:32 are TDX specific */ +u64 details_l1 :8 ; +u64 class :8 ; +u64 reserved61_48 :14 ; +u64 non_recoverable :1 ; +u64 error :1 ; + }; +u64 full; +}; +struct vcpu_tdx {struct kvm_vcpu vcpu;
@@ -32,6 +63,8 @@ struct vcpu_tdx {unsigned long *tdvpx_pa;bool td_vcpu_created; +union tdx_exit_reason exit_reason; +bool initialized;/*
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -150,6 +150,7 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);int tdx_vcpu_create (struct kvm_vcpu *vcpu);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event); +fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -177,6 +178,7 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPstatic inline int tdx_vcpu_create (struct kvm_vcpu *vcpu) {return -EOPNOTSUPP ; }static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {} +static inline fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) {return EXIT_FASTPATH_NONE ; }static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
arch/x86/kvm/x86.c +1 -0
@@ -315,6 +315,7 @@ const struct kvm_stats_header kvm_vcpu_stats_header = { };u64 __read_mostly host_xcr0; +EXPORT_SYMBOL_GPL (host_xcr0);static struct kmem_cache *x86_emulator_cache;
Signature · 1 line
2.25.1
Re: [PATCH v18 067/121] KVM: TDX: Implement TDX vcpu enter/exit path
Quoted 8 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
This patch implements running TDX vcpu. Once vcpu runs on the logical processor (LP), the TDX vcpu is associated with it. When the TDX vcpu moves to another LP, the TDX vcpu needs to flush its status on the LP. When destroying TDX vcpu, it needs to complete flush and flush cpu memory cache. Track which LP the TDX vcpu run and flush it as necessary.
Quoted 5 lines
Do nothing on
sched_inevent as TDX doesn't support pause loop.TDX vcpu execution requires restoring PMU debug store after returning back to KVM because the TDX module unconditionally resets the value. To reuse the existing code, export
perf_restore_debug_store.
The changelog doesn't match the code implemented in this patch.
Quoted 233 lines
Signed-off-by Isaku Yamahata <[email protected]>---
Changes v15 -> v16:
- use__seamcall_saved_ret()
- As structtdx_module_argsdoesn't match with vcpu.arch.regs, copy regs before/after calling__seamcall_saved_ret().--- arch/x86/kvm/vmx/main.c | 21 +++++++++- arch/x86/kvm/vmx/tdx.c | 84 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 33 +++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 2 + arch/x86/kvm/x86.c | 1 + 5 files changed, 139 insertions(+), 2 deletions(-)arch/x86/kvm/vmx/main.c +19
-2@@ -173,6 +173,23 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)vmx_vcpu_reset (vcpu, init_event); } +static int vt_vcpu_pre_run (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +/* Unconditionally continue to vcpu_run(). */ +return 1 ; + +return vmx_vcpu_pre_run (vcpu); +} + +static fastpath_t vt_vcpu_run (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_vcpu_run (vcpu); + +return vmx_vcpu_run (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) {@@ -325,8 +342,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .flush_tlb_gva = vt_flush_tlb_gva, .flush_tlb_guest = vt_flush_tlb_guest, - .vcpu_pre_run = vmx_vcpu_pre_run, - .vcpu_run = vmx_vcpu_run, + .vcpu_pre_run = vt_vcpu_pre_run, + .vcpu_run = vt_vcpu_run, .handle_exit = vmx_handle_exit, .skip_emulated_instruction = vmx_skip_emulated_instruction, .update_emulated_instruction = vmx_update_emulated_instruction,arch/x86/kvm/vmx/tdx.c +84
-0@@ -11,6 +11,9 @@#include "vmx.h" #include "x86.h" +#include <trace/events/kvm.h> +#include "trace.h" +#undef pr_fmt#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt@@ -541,6 +544,87 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)*/ } +static noinstr void tdx_vcpu_enter_exit(struct vcpu_tdx *tdx) +{ +struct tdx_module_args args; + +/* +* Avoid section mismatch with to_tdx() with KVM_VM_BUG(). The caller +* should call to_tdx(). +*/ +struct kvm_vcpu *vcpu = &tdx->vcpu; + +guest_state_enter_irqoff (); + +/* +* TODO: optimization: +* - Eliminate copy between args and vcpu->arch.regs. +* - copyin/copyout registers only if (tdx->tdvmvall.regs_mask != 0) +* which means TDG.VP.VMCALL. +*/ + args = (struct tdx_module_args ) { + .rcx = tdx->tdvpr_pa, +#define REG(reg, REG) .reg = vcpu->arch.regs[VCPU_REGS_ ## REG] +REG (rdx,RDX ), +REG (r8,R8 ), +REG (r9,R9 ), +REG (r10,R10 ), +REG (r11,R11 ), +REG (r12,R12 ), +REG (r13,R13 ), +REG (r14,R14 ), +REG (r15,R15 ), +REG (rbx,RBX ), +REG (rdi,RDI ), +REG (rsi,RSI ), +#undef REG + }; + + tdx->exit_reason.full =__seamcall_saved_ret (TDH_VP_ENTER , &args); + +#define REG (reg,REG ) vcpu->arch.regs[VCPU_REGS_ ## REG] = args.reg +REG (rcx,RCX ); +REG (rdx,RDX ); +REG (r8,R8 ); +REG (r9,R9 ); +REG (r10,R10 ); +REG (r11,R11 ); +REG (r12,R12 ); +REG (r13,R13 ); +REG (r14,R14 ); +REG (r15,R15 ); +REG (rbx,RBX ); +REG (rdi,RDI ); +REG (rsi,RSI ); +#undef REG + +WARN_ON_ONCE (!kvm_rebooting && + (tdx->exit_reason.full &TDX_SW_ERROR ) ==TDX_SW_ERROR ); + +guest_state_exit_irqoff (); +} + +fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); + +if (unlikely (!tdx->initialized)) +return -EINVAL ; +if (unlikely (vcpu->kvm->vm_bugged)) { + tdx->exit_reason.full =TDX_NON_RECOVERABLE_VCPU ; +return EXIT_FASTPATH_NONE ; + } + +trace_kvm_entry (vcpu); + +tdx_vcpu_enter_exit (tdx); + + vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ; +trace_kvm_exit (vcpu,KVM_ISA_VMX ); + +return EXIT_FASTPATH_NONE ; +} +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) {td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK );arch/x86/kvm/vmx/tdx.h +33
-0@@ -25,6 +25,37 @@ struct kvm_tdx {u64 tsc_offset; }; +union tdx_exit_reason { +struct { +/* 31:0 mirror the VMX Exit Reason format */ +u64 basic :16 ; +u64 reserved16 :1 ; +u64 reserved17 :1 ; +u64 reserved18 :1 ; +u64 reserved19 :1 ; +u64 reserved20 :1 ; +u64 reserved21 :1 ; +u64 reserved22 :1 ; +u64 reserved23 :1 ; +u64 reserved24 :1 ; +u64 reserved25 :1 ; +u64 bus_lock_detected :1 ; +u64 enclave_mode :1 ; +u64 smi_pending_mtf :1 ; +u64 smi_from_vmx_root :1 ; +u64 reserved30 :1 ; +u64 failed_vmentry :1 ; + +/* 63:32 are TDX specific */ +u64 details_l1 :8 ; +u64 class :8 ; +u64 reserved61_48 :14 ; +u64 non_recoverable :1 ; +u64 error :1 ; + }; +u64 full; +}; +struct vcpu_tdx {struct kvm_vcpu vcpu;@@ -32,6 +63,8 @@ struct vcpu_tdx {unsigned long *tdvpx_pa;bool td_vcpu_created; +union tdx_exit_reason exit_reason; +bool initialized;/* arch/x86/kvm/vmx/x86_ops.h +2
-0@@ -150,6 +150,7 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);int tdx_vcpu_create (struct kvm_vcpu *vcpu);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event); +fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);@@ -177,6 +178,7 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPstatic inline int tdx_vcpu_create (struct kvm_vcpu *vcpu) {return -EOPNOTSUPP ; }static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {} +static inline fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) {return EXIT_FASTPATH_NONE ; }static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c const struct kvm_stats_header kvm_vcpu_stats_header = { · line 318
Quoted 4 lines
arch/x86/kvm/x86.c +1
-0@@ -315,6 +315,7 @@ const struct kvm_stats_header kvm_vcpu_stats_header = { };u64 __read_mostly host_xcr0; +EXPORT_SYMBOL_GPL (host_xcr0);
It's not used in this patch, can it be moved to where it is used?
Quoted 3 lines
arch/x86/kvm/x86.c +0
-0static struct kmem_cache *x86_emulator_cache;
PATCH v18 68/121 KVM: TDX: vcpu_run: save/restore host state(host kernel gs)
In reply to [email protected]From: Isaku Yamahata <[email protected]>
On entering/exiting TDX vcpu, Preserved or clobbered CPU state is different from VMX case. Add TDX hooks to save/restore host/guest CPU state. Save/restore kernel GS base MSR.
--- arch/x86/kvm/vmx/main.c | 30 +++++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 42 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 4 ++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ 4 files changed, 78 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +28 -2
@@ -173,6 +173,32 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)vmx_vcpu_reset (vcpu, init_event); } +static void vt_prepare_switch_to_guest (struct kvm_vcpu *vcpu) +{ +/* +* All host state is saved/restored across SEAMCALL/SEAMRET, and the +* guest state of a TD is obviously off limits. Deferring MSRs and DRs +* is pointless because the TDX module needs to load *something* so as +* not to expose guest state. +*/ +if (is_td_vcpu (vcpu)) { +tdx_prepare_switch_to_guest (vcpu); +return ; + } + +vmx_prepare_switch_to_guest (vcpu); +} + +static void vt_vcpu_put (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +tdx_vcpu_put (vcpu); +return ; + } + +vmx_vcpu_put (vcpu); +} +static int vt_vcpu_pre_run (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -308,9 +334,9 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_free = vt_vcpu_free, .vcpu_reset = vt_vcpu_reset, - .prepare_switch_to_guest = vmx_prepare_switch_to_guest, + .prepare_switch_to_guest = vt_prepare_switch_to_guest, .vcpu_load = vmx_vcpu_load, - .vcpu_put = vmx_vcpu_put, + .vcpu_put = vt_vcpu_put, .update_exception_bitmap = vmx_update_exception_bitmap, .get_msr_feature = vmx_get_msr_feature,
arch/x86/kvm/vmx/tdx.c +42 -0
68 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +4 -0
@@ -67,6 +67,10 @@ struct vcpu_tdx {bool initialized; +bool host_state_need_save; +bool host_state_need_restore; +u64 msr_host_kernel_gs_base; +/* * Dummy to make pmu_intel not corrupt memory. * TODO: Support PMU for TDX. Future work.
arch/x86/kvm/vmx/x86_ops.h +4 -0
@@ -151,6 +151,8 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event);fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu); +void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu); +void tdx_vcpu_put (struct kvm_vcpu *vcpu);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -179,6 +181,8 @@ static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTSUPP; }static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {}static inline fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) {return EXIT_FASTPATH_NONE ; } +static inline void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {} +static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {}static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v18 69/121 KVM: TDX: restore host xsave state when exit from the guest TD
In reply to [email protected]From: Isaku Yamahata <[email protected]>
On exiting from the guest TD, xsave state is clobbered. Restore xsave state on TD exit.
---
v15 -> v16:
- Added CET flag mask
--- arch/x86/kvm/vmx/tdx.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
arch/x86/kvm/vmx/tdx.c +19 -0
@@ -2,6 +2,7 @@#include <linux/cpu.h> #include <linux/mmu_context.h> +#include <asm/fpu/xcr.h> #include <asm/tdx.h> #include "capabilities.h"
@@ -584,6 +585,23 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)*/ } +static void tdx_restore_host_xsave_state (struct kvm_vcpu *vcpu) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (vcpu->kvm); + +if (static_cpu_has (X86_FEATURE_XSAVE ) && + host_xcr0 != (kvm_tdx->xfam & kvm_caps.supported_xcr0)) +xsetbv (XCR_XFEATURE_ENABLED_MASK , host_xcr0); +if (static_cpu_has (X86_FEATURE_XSAVES ) && +/* PT can be exposed to TD guest regardless of KVM's XSS support */ + host_xss != (kvm_tdx->xfam & + (kvm_caps.supported_xss |XFEATURE_MASK_PT |TDX_TD_XFAM_CET ))) +wrmsrl (MSR_IA32_XSS , host_xss); +if (static_cpu_has (X86_FEATURE_PKU ) && + (kvm_tdx->xfam &XFEATURE_MASK_PKRU )) +write_pkru (vcpu->arch.host_pkru); +} + static noinstr void tdx_vcpu_enter_exit(struct vcpu_tdx *tdx) {struct tdx_module_args args;
@@ -659,6 +677,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (tdx); +tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true; vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ;
Signature · 1 line
2.25.1
Re: [PATCH v18 069/121] KVM: TDX: restore host xsave state when exit from the guest TD
Quoted 36 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
On exiting from the guest TD, xsave state is clobbered. Restore xsave state on TD exit.
Signed-off-by Isaku Yamahata <[email protected]>---
v15 -> v16:
- Added CET flag mask--- arch/x86/kvm/vmx/tdx.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)arch/x86/kvm/vmx/tdx.c +8
-0@@ -2,6 +2,7 @@#include <linux/cpu.h> #include <linux/mmu_context.h> +#include <asm/fpu/xcr.h> #include <asm/tdx.h> #include "capabilities.h" @@ -584,6 +585,23 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)*/ } +static void tdx_restore_host_xsave_state (struct kvm_vcpu *vcpu) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (vcpu->kvm); + +if (static_cpu_has (X86_FEATURE_XSAVE ) && + host_xcr0 != (kvm_tdx->xfam & kvm_caps.supported_xcr0)) +xsetbv (XCR_XFEATURE_ENABLED_MASK , host_xcr0);
arch/x86/kvm/vmx/tdx.c void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) · line 604
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+if (static_cpu_has (X86_FEATURE_XSAVES ) && +/* PT can be exposed to TD guest regardless of KVM's XSS support */ + host_xss != (kvm_tdx->xfam & + (kvm_caps.supported_xss |XFEATURE_MASK_PT |TDX_TD_XFAM_CET ))) +wrmsrl (MSR_IA32_XSS , host_xss); +if (static_cpu_has (X86_FEATURE_PKU ) && + (kvm_tdx->xfam &XFEATURE_MASK_PKRU )) +write_pkru (vcpu->arch.host_pkru); +} +
The export of host_xcr0 in patch 67 can be moved to this path.
u64 __read_mostly host_xcr0;
+EXPORT_SYMBOL_GPL(host_xcr0);
Quoted 11 lines
arch/x86/kvm/vmx/tdx.c +1
-0static noinstr void tdx_vcpu_enter_exit(struct vcpu_tdx *tdx) {struct tdx_module_args args;@@ -659,6 +677,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (tdx); +tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true; vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ;
PATCH v18 70/121 KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o wrmsr
In reply to [email protected]From: Chao Gao <[email protected]>
Several MSRs are constant and only used in userspace(ring 3). But VMs may have different values. KVM uses kvm_set_user_return_msr() to switch to guest's values and leverages user return notifier to restore them when the kernel is to return to userspace. To eliminate unnecessary wrmsr, KVM also caches the value it wrote to an MSR last time.
TDX module unconditionally resets some of these MSRs to architectural INIT state on TD exit. It makes the cached values in kvm_user_return_msrs are inconsistent with values in hardware. This inconsistency needs to be fixed. Otherwise, it may mislead kvm_on_user_return() to skip restoring some MSRs to the host's values. kvm_set_user_return_msr() can help correct this case, but it is not optimal as it always does a wrmsr. So, introduce a variation of kvm_set_user_return_msr() to update cached values and skip that wrmsr.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -2268,6 +2268,7 @@ int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low,int kvm_add_user_return_msr (u32 msr);int kvm_find_user_return_msr (u32 msr);int kvm_set_user_return_msr (unsigned index,u64 val,u64 mask); +void kvm_user_return_update_cache (unsigned int index,u64 val);static inline bool kvm_is_supported_user_return_msr (u32 msr) {
arch/x86/kvm/x86.c +20 -5
@@ -440,6 +440,15 @@ static void kvm_user_return_msr_cpu_online(void) } } +static void kvm_user_return_register_notifier (struct kvm_user_return_msrs *msrs) +{ +if (!msrs->registered) { + msrs->urn.on_user_return = kvm_on_user_return; +user_return_notifier_register (&msrs->urn); + msrs->registered = true; + } +} +int kvm_set_user_return_msr (unsigned slot,u64 value,u64 mask) {unsigned int cpu =smp_processor_id ();
@@ -454,15 +463,21 @@ int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask)return 1 ; msrs->values[slot].curr = value; -if (!msrs->registered) { - msrs->urn.on_user_return = kvm_on_user_return; -user_return_notifier_register (&msrs->urn); - msrs->registered = true; - } +kvm_user_return_register_notifier (msrs);return 0 ; }EXPORT_SYMBOL_GPL (kvm_set_user_return_msr); +/* Update the cache, "curr", and register the notifier */ +void kvm_user_return_update_cache (unsigned int slot,u64 value) +{ +struct kvm_user_return_msrs *msrs =this_cpu_ptr (user_return_msrs); + + msrs->values[slot].curr = value; +kvm_user_return_register_notifier (msrs); +} +EXPORT_SYMBOL_GPL (kvm_user_return_update_cache); +static void drop_user_return_notifiers (void ) {unsigned int cpu =smp_processor_id ();
Signature · 1 line
2.25.1
PATCH v18 71/121 KVM: TDX: restore user ret MSRs
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Several user ret MSRs are clobbered on TD exit. Restore those values on TD exit and before returning to ring 3. Because TSX_CTRL requires special treat, this patch doesn't address it.
--- arch/x86/kvm/vmx/tdx.c | 43 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
arch/x86/kvm/vmx/tdx.c +43 -0
@@ -585,6 +585,28 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)*/ } +struct tdx_uret_msr { +u32 msr; +unsigned int slot; +u64 defval; +}; + +static struct tdx_uret_msr tdx_uret_msrs[] = { + {.msr =MSR_SYSCALL_MASK , .defval =0x20200 }, + {.msr =MSR_STAR ,}, + {.msr =MSR_LSTAR ,}, + {.msr =MSR_TSC_AUX ,}, +}; + +static void tdx_user_return_update_cache (void ) +{ +int i; + +for (i =0 ; i <ARRAY_SIZE (tdx_uret_msrs); i++) +kvm_user_return_update_cache (tdx_uret_msrs[i].slot, + tdx_uret_msrs[i].defval); +} +static void tdx_restore_host_xsave_state (struct kvm_vcpu *vcpu) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (vcpu->kvm);
@@ -677,6 +699,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (tdx); +tdx_user_return_update_cache ();tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true;
@@ -1936,6 +1959,26 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)return -EINVAL ; } +for (i =0 ; i <ARRAY_SIZE (tdx_uret_msrs); i++) { +/* +* Here it checks if MSRs (tdx_uret_msrs) can be saved/restored +* before returning to user space. +* +* this_cpu_ptr(user_return_msrs)->registered isn't checked +* because the registration is done at vcpu runtime by +* kvm_set_user_return_msr(). +* Here is setting up cpu feature before running vcpu, +* registered is already false. +*/ + tdx_uret_msrs[i].slot =kvm_find_user_return_msr (tdx_uret_msrs[i].msr); +if (tdx_uret_msrs[i].slot ==-1 ) { +/* If any MSR isn't supported, it is a KVM bug */ +pr_err ("MSR %x isn't included by kvm_find_user_return_msr\n" , + tdx_uret_msrs[i].msr); +return -EIO ; + } + } + max_pkgs =topology_max_packages (); tdx_mng_key_config_lock =kcalloc (max_pkgs,sizeof (*tdx_mng_key_config_lock),GFP_KERNEL );
Signature · 1 line
2.25.1
Re: [PATCH v18 071/121] KVM: TDX: restore user ret MSRs
Quoted 55 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Several user ret MSRs are clobbered on TD exit. Restore those values on TD exit and before returning to ring 3. Because
TSX_CTRLrequires special treat, this patch doesn't address it.Signed-off-by Isaku Yamahata <[email protected]>Reviewed-by Paolo Bonzini <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 43 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)arch/x86/kvm/vmx/tdx.c +23
-0@@ -585,6 +585,28 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)*/ } +struct tdx_uret_msr { +u32 msr; +unsigned int slot; +u64 defval; +}; + +static struct tdx_uret_msr tdx_uret_msrs[] = { + {.msr =MSR_SYSCALL_MASK , .defval =0x20200 }, + {.msr =MSR_STAR ,}, + {.msr =MSR_LSTAR ,}, + {.msr =MSR_TSC_AUX ,}, +}; + +static void tdx_user_return_update_cache (void ) +{ +int i; + +for (i =0 ; i <ARRAY_SIZE (tdx_uret_msrs); i++) +kvm_user_return_update_cache (tdx_uret_msrs[i].slot, + tdx_uret_msrs[i].defval); +} +static void tdx_restore_host_xsave_state (struct kvm_vcpu *vcpu) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (vcpu->kvm);@@ -677,6 +699,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (tdx); +tdx_user_return_update_cache ();tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true;@@ -1936,6 +1959,26 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)return -EINVAL ;
arch/x86/kvm/vmx/tdx.c int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops) · line 1969
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +8
-0} +for (i =0 ; i <ARRAY_SIZE (tdx_uret_msrs); i++) { +/* +* Here it checks if MSRs (tdx_uret_msrs) can be saved/restored +* before returning to user space. +* +* this_cpu_ptr(user_return_msrs)->registered isn't checked +* because the registration is done at vcpu runtime by +* kvm_set_user_return_msr().
For tdx, it's done by kvm_user_return_update_cache(), right?
Quoted 15 lines
arch/x86/kvm/vmx/tdx.c +12
-0+* Here is setting up cpu feature before running vcpu, +* registered is already false. +*/ + tdx_uret_msrs[i].slot =kvm_find_user_return_msr (tdx_uret_msrs[i].msr); +if (tdx_uret_msrs[i].slot ==-1 ) { +/* If any MSR isn't supported, it is a KVM bug */ +pr_err ("MSR %x isn't included by kvm_find_user_return_msr\n" , + tdx_uret_msrs[i].msr); +return -EIO ; + } + } + max_pkgs =topology_max_packages (); tdx_mng_key_config_lock =kcalloc (max_pkgs,sizeof (*tdx_mng_key_config_lock),GFP_KERNEL );
Re: [PATCH v18 071/121] KVM: TDX: restore user ret MSRs
Quoted 5 lines · Binbin Wu
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +2
-0@@ -1936,6 +1959,26 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)return -EINVAL ; } +for (i =0 ; i <ARRAY_SIZE (tdx_uret_msrs); i++) { +/*
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops) · line 1969
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +6
-0+* Here it checks if MSRs (tdx_uret_msrs) can be saved/restored +* before returning to user space. +* +* this_cpu_ptr(user_return_msrs)->registered isn't checked +* because the registration is done at vcpu runtime by +* kvm_set_user_return_msr(). For tdx, it's done by
kvm_user_return_update_cache(), right?
Right, fixed the comment.
Signature · 1 line
Isaku Yamahata <[email protected]>
PATCH v18 72/121 KVM: TDX: Add TSX_CTRL msr into uret_msrs list
In reply to [email protected]From: Yang Weijiang <[email protected]>
TDX module resets the TSX_CTRL MSR to 0 at TD exit if TSX is enabled for TD. Or it preserves the TSX_CTRL MSR if TSX is disabled for TD. VMM can rely on uret_msrs mechanism to defer the reload of host value until exiting to user space.
--- arch/x86/kvm/vmx/tdx.c | 33 +++++++++++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx.h | 8 ++++++++ 2 files changed, 39 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/tdx.c +31 -2
@@ -597,14 +597,21 @@ static struct tdx_uret_msr tdx_uret_msrs[] = { {.msr = MSR_LSTAR,}, {.msr = MSR_TSC_AUX,}, }; +static unsigned int tdx_uret_tsx_ctrl_slot; -static void tdx_user_return_update_cache (void ) +static void tdx_user_return_update_cache (struct kvm_vcpu *vcpu) {int i;for (i =0 ; i <ARRAY_SIZE (tdx_uret_msrs); i++)kvm_user_return_update_cache (tdx_uret_msrs[i].slot, tdx_uret_msrs[i].defval); +/* +* TSX_CTRL is reset to 0 if guest TSX is supported. Otherwise +* preserved. +*/ +if (to_kvm_tdx (vcpu->kvm)->tsx_supported && tdx_uret_tsx_ctrl_slot !=-1 ) +kvm_user_return_update_cache (tdx_uret_tsx_ctrl_slot,0 ); }static void tdx_restore_host_xsave_state (struct kvm_vcpu *vcpu)
@@ -699,7 +706,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (tdx); -tdx_user_return_update_cache (); +tdx_user_return_update_cache (vcpu);tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true;
@@ -1212,6 +1219,22 @@ static int setup_tdparams_xfam(struct kvm_cpuid2 *cpuid, struct td_params *td_pareturn 0 ; } +static bool tdparams_tsx_supported (struct kvm_cpuid2 *cpuid) +{ +const struct kvm_cpuid_entry2 *entry; +u64 mask; +u32 ebx; + + entry =kvm_find_cpuid_entry2 (cpuid->entries, cpuid->nent,0x7 ,0 ); +if (entry) + ebx = entry->ebx; +else + ebx =0 ; + + mask =__feature_bit (X86_FEATURE_HLE ) |__feature_bit (X86_FEATURE_RTM ); +return ebx & mask; +} +static int setup_tdparams (struct kvm *kvm,struct td_params *td_params,struct kvm_tdx_init_vm *init_vm) {
@@ -1253,6 +1276,7 @@ static int setup_tdparams(struct kvm *kvm, struct td_params *td_params,MEMCPY_SAME_SIZE (td_params->mrowner, init_vm->mrowner);MEMCPY_SAME_SIZE (td_params->mrownerconfig, init_vm->mrownerconfig); +to_kvm_tdx (kvm)->tsx_supported =tdparams_tsx_supported (cpuid);return 0 ; }
@@ -1978,6 +2002,11 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)return -EIO ; } } + tdx_uret_tsx_ctrl_slot =kvm_find_user_return_msr (MSR_IA32_TSX_CTRL ); +if (tdx_uret_tsx_ctrl_slot ==-1 &&boot_cpu_has (X86_FEATURE_MSR_TSX_CTRL )) { +pr_err ("MSR_IA32_TSX_CTRL isn't included by kvm_find_user_return_msr\n" ); +return -EIO ; + } max_pkgs =topology_max_packages (); tdx_mng_key_config_lock = kcalloc(max_pkgs, sizeof(*tdx_mng_key_config_lock),
arch/x86/kvm/vmx/tdx.h +8 -0
@@ -17,6 +17,14 @@ struct kvm_tdx {u64 xfam;int hkid; +/* +* Used on each TD-exit, see tdx_user_return_update_cache(). +* TSX_CTRL value on TD exit +* - set 0 if guest TSX enabled +* - preserved if guest TSX disabled +*/ +bool tsx_supported; +hpa_t source_pa;bool finalized;
Signature · 1 line
2.25.1
Re: [PATCH v18 072/121] KVM: TDX: Add TSX_CTRL msr into uret_msrs list
Quoted 19 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Yang Weijiang <[email protected]>
TDX module resets the
TSX_CTRLMSR to 0 at TD exit if TSX is enabled for TD. Or it preserves theTSX_CTRLMSR if TSX is disabled for TD. VMM can rely onuret_msrsmechanism to defer the reload of host value until exiting to user space.Signed-off-by Yang Weijiang <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 33 +++++++++++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx.h | 8 ++++++++ 2 files changed, 39 insertions(+), 2 deletions(-)arch/x86/kvm/vmx/tdx.c +0
-0
arch/x86/kvm/vmx/tdx.c static struct tdx_uret_msr tdx_uret_msrs[] = { · line 600
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +1
-0@@ -597,14 +597,21 @@ static struct tdx_uret_msr tdx_uret_msrs[] = { {.msr = MSR_LSTAR,}, {.msr = MSR_TSC_AUX,}, }; +static unsigned int tdx_uret_tsx_ctrl_slot;
It should use "int" instead of "unsigned int" since the return type of kvm_find_user_return_msr() is int.
Not a good code style to compare between unsigned int and int.
Quoted 89 lines
arch/x86/kvm/vmx/tdx.c +30
-2-static void tdx_user_return_update_cache (void ) +static void tdx_user_return_update_cache (struct kvm_vcpu *vcpu) {int i;for (i =0 ; i <ARRAY_SIZE (tdx_uret_msrs); i++)kvm_user_return_update_cache (tdx_uret_msrs[i].slot, tdx_uret_msrs[i].defval); +/* +* TSX_CTRL is reset to 0 if guest TSX is supported. Otherwise +* preserved. +*/ +if (to_kvm_tdx (vcpu->kvm)->tsx_supported && tdx_uret_tsx_ctrl_slot !=-1 ) +kvm_user_return_update_cache (tdx_uret_tsx_ctrl_slot,0 ); }static void tdx_restore_host_xsave_state (struct kvm_vcpu *vcpu)@@ -699,7 +706,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (tdx); -tdx_user_return_update_cache (); +tdx_user_return_update_cache (vcpu);tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true;@@ -1212,6 +1219,22 @@ static int setup_tdparams_xfam(struct kvm_cpuid2 *cpuid, struct td_params *td_pareturn 0 ; } +static bool tdparams_tsx_supported (struct kvm_cpuid2 *cpuid) +{ +const struct kvm_cpuid_entry2 *entry; +u64 mask; +u32 ebx; + + entry =kvm_find_cpuid_entry2 (cpuid->entries, cpuid->nent,0x7 ,0 ); +if (entry) + ebx = entry->ebx; +else + ebx =0 ; + + mask =__feature_bit (X86_FEATURE_HLE ) |__feature_bit (X86_FEATURE_RTM ); +return ebx & mask; +} +static int setup_tdparams (struct kvm *kvm,struct td_params *td_params,struct kvm_tdx_init_vm *init_vm) {@@ -1253,6 +1276,7 @@ static int setup_tdparams(struct kvm *kvm, struct td_params *td_params,MEMCPY_SAME_SIZE (td_params->mrowner, init_vm->mrowner);MEMCPY_SAME_SIZE (td_params->mrownerconfig, init_vm->mrownerconfig); +to_kvm_tdx (kvm)->tsx_supported =tdparams_tsx_supported (cpuid);return 0 ; }@@ -1978,6 +2002,11 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)return -EIO ; } } + tdx_uret_tsx_ctrl_slot =kvm_find_user_return_msr (MSR_IA32_TSX_CTRL ); +if (tdx_uret_tsx_ctrl_slot ==-1 &&boot_cpu_has (X86_FEATURE_MSR_TSX_CTRL )) { +pr_err ("MSR_IA32_TSX_CTRL isn't included by kvm_find_user_return_msr\n" ); +return -EIO ; + } max_pkgs =topology_max_packages (); tdx_mng_key_config_lock = kcalloc(max_pkgs, sizeof(*tdx_mng_key_config_lock),arch/x86/kvm/vmx/tdx.h +8
-0@@ -17,6 +17,14 @@ struct kvm_tdx {u64 xfam;int hkid; +/* +* Used on each TD-exit, see tdx_user_return_update_cache(). +* TSX_CTRL value on TD exit +* - set 0 if guest TSX enabled +* - preserved if guest TSX disabled +*/ +bool tsx_supported; +hpa_t source_pa;bool finalized;
PATCH v18 73/121 The start of TDX KVM patch series: TD vcpu exits/interrupts/hypercalls
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TD vcpu exits, interrupts, and hypercalls.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -1
@@ -13,6 +13,7 @@ What qemu can do - Qemu can create/destroy vcpu of TDX vm type. - Qemu can populate initial guest memory image. - Qemu can finalize guest TD. +- Qemu can start to run vcpu. But vcpu can not make progress yet. Patch Layer status ------------------
@@ -24,7 +25,7 @@ Patch Layer status * TD vcpu creation/destruction: Applied * TDX EPT violation: Applied * TD finalization: Applied -* TD vcpu enter/exit: Applying +* TD vcpu enter/exit: Applied * TD vcpu interrupts/exit/hypercall: Not yet * KVM MMU GPA shared bits: Applied
Signature · 1 line
2.25.1
PATCH v18 74/121 KVM: TDX: complete interrupts after tdexit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This corresponds to VMX __vmx_complete_interrupts(). Because TDX virtualize vAPIC, KVM only needs to care NMI injection.
--- arch/x86/kvm/vmx/tdx.c | 10 ++++++++++ arch/x86/kvm/vmx/tdx.h | 2 ++ 2 files changed, 12 insertions(+)
arch/x86/kvm/vmx/tdx.c +10 -0
@@ -585,6 +585,14 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)*/ } +static void tdx_complete_interrupts (struct kvm_vcpu *vcpu) +{ +/* Avoid costly SEAMCALL if no nmi was injected */ +if (vcpu->arch.nmi_injected) + vcpu->arch.nmi_injected =td_management_read8 (to_tdx (vcpu), +TD_VCPU_PEND_NMI ); +} +struct tdx_uret_msr {u32 msr;unsigned int slot;
@@ -713,6 +721,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ;trace_kvm_exit (vcpu,KVM_ISA_VMX ); +tdx_complete_interrupts (vcpu); +return EXIT_FASTPATH_NONE ; }
arch/x86/kvm/vmx/tdx.h +2 -0
@@ -201,6 +201,8 @@ TDX_BUILD_TDVPS_ACCESSORS(16, VMCS, vmcs);TDX_BUILD_TDVPS_ACCESSORS (32 ,VMCS , vmcs);TDX_BUILD_TDVPS_ACCESSORS (64 ,VMCS , vmcs); +TDX_BUILD_TDVPS_ACCESSORS (8 ,MANAGEMENT , management); + static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field) {struct tdx_module_args out;
Signature · 1 line
2.25.1
Re: [PATCH v18 074/121] KVM: TDX: complete interrupts after tdexit
Quoted 4 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
This corresponds to VMX
__vmx_complete_interrupts(). Because TDX virtualize vAPIC, KVM only needs to care NMI injection.
Nit: complete -> Complete in shortlog.
Quoted 49 lines
Signed-off-by Isaku Yamahata <[email protected]>Reviewed-by Paolo Bonzini <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 10 ++++++++++ arch/x86/kvm/vmx/tdx.h | 2 ++ 2 files changed, 12 insertions(+)arch/x86/kvm/vmx/tdx.c +10
-0@@ -585,6 +585,14 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)*/ } +static void tdx_complete_interrupts (struct kvm_vcpu *vcpu) +{ +/* Avoid costly SEAMCALL if no nmi was injected */ +if (vcpu->arch.nmi_injected) + vcpu->arch.nmi_injected =td_management_read8 (to_tdx (vcpu), +TD_VCPU_PEND_NMI ); +} +struct tdx_uret_msr {u32 msr;unsigned int slot;@@ -713,6 +721,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ;trace_kvm_exit (vcpu,KVM_ISA_VMX ); +tdx_complete_interrupts (vcpu); +return EXIT_FASTPATH_NONE ; }arch/x86/kvm/vmx/tdx.h +2
-0@@ -201,6 +201,8 @@ TDX_BUILD_TDVPS_ACCESSORS(16, VMCS, vmcs);TDX_BUILD_TDVPS_ACCESSORS (32 ,VMCS , vmcs);TDX_BUILD_TDVPS_ACCESSORS (64 ,VMCS , vmcs); +TDX_BUILD_TDVPS_ACCESSORS (8 ,MANAGEMENT , management); + static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field) {struct tdx_module_args out;
PATCH v18 75/121 KVM: TDX: restore debug store when TD exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because debug store is clobbered, restore it on TD exit.
--- arch/x86/events/intel/ds.c | 1 + arch/x86/kvm/vmx/tdx.c | 1 + 2 files changed, 2 insertions(+)
arch/x86/events/intel/ds.c +1 -0
@@ -2428,3 +2428,4 @@ void perf_restore_debug_store(void)wrmsrl (MSR_IA32_DS_AREA , (unsigned long )ds); } +EXPORT_SYMBOL_GPL (perf_restore_debug_store);
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -715,6 +715,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (tdx);tdx_user_return_update_cache (vcpu); +perf_restore_debug_store ();tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true;
Signature · 1 line
2.25.1
PATCH v18 76/121 KVM: TDX: handle vcpu migration over logical processor
In reply to [email protected]From: Isaku Yamahata <[email protected]>
For vcpu migration, in the case of VMX, VMCS is flushed on the source pcpu, and load it on the target pcpu. There are corresponding TDX SEAMCALL APIs, call them on vcpu migration. The logic is mostly same as VMX except the TDX SEAMCALLs are used.
When shutting down the machine, (VMX or TDX) vcpus needs to be shutdown on each pcpu. Do the similar for TDX with TDX SEAMCALL APIs.
--- arch/x86/kvm/vmx/main.c | 32 ++++++- arch/x86/kvm/vmx/tdx.c | 190 ++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 2 + arch/x86/kvm/vmx/x86_ops.h | 4 + 4 files changed, 221 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/main.c +29 -3
@@ -48,6 +48,14 @@ static int vt_hardware_enable(void)return ret; } +static void vt_hardware_disable (void ) +{ +/* Note, TDX *and* VMX need to be disabled if TDX is enabled. */ +if (enable_tdx) +tdx_hardware_disable (); +vmx_hardware_disable (); +} +static __init int vt_hardware_setup (void ) {int ret;
@@ -216,6 +224,16 @@ static fastpath_t vt_vcpu_run(struct kvm_vcpu *vcpu)return vmx_vcpu_run (vcpu); } +static void vt_vcpu_load (struct kvm_vcpu *vcpu,int cpu) +{ +if (is_td_vcpu (vcpu)) { +tdx_vcpu_load (vcpu, cpu); +return ; + } + +vmx_vcpu_load (vcpu, cpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) {
@@ -275,6 +293,14 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); } +static void vt_sched_in (struct kvm_vcpu *vcpu,int cpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_sched_in (vcpu, cpu); +} +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {if (is_td_vcpu (vcpu))
@@ -317,7 +343,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .offline_cpu = tdx_offline_cpu, .hardware_enable = vt_hardware_enable, - .hardware_disable = vmx_hardware_disable, + .hardware_disable = vt_hardware_disable, .has_emulated_msr = vmx_has_emulated_msr, .is_vm_type_supported = vt_is_vm_type_supported,
@@ -335,7 +361,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_reset = vt_vcpu_reset, .prepare_switch_to_guest = vt_prepare_switch_to_guest, - .vcpu_load = vmx_vcpu_load, + .vcpu_load = vt_vcpu_load, .vcpu_put = vt_vcpu_put, .update_exception_bitmap = vmx_update_exception_bitmap,
@@ -422,7 +448,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .request_immediate_exit = vmx_request_immediate_exit, - .sched_in = vmx_sched_in, + .sched_in = vt_sched_in, .cpu_dirty_log_size = PML_ENTITY_NUM, .update_cpu_dirty_logging = vmx_update_cpu_dirty_logging,
arch/x86/kvm/vmx/tdx.c +186 -4
268 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +2 -0
@@ -71,6 +71,8 @@ struct vcpu_tdx {unsigned long *tdvpx_pa;bool td_vcpu_created; +struct list_head cpu_list; +union tdx_exit_reason exit_reason;bool initialized;
arch/x86/kvm/vmx/x86_ops.h +4 -0
@@ -137,6 +137,7 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu);#ifdef CONFIG_INTEL_TDX_HOST int __init tdx_hardware_setup (struct kvm_x86_ops *x86_ops);void tdx_hardware_unsetup (void ); +void tdx_hardware_disable (void );bool tdx_is_vm_type_supported (unsigned long type);int tdx_offline_cpu (void );
@@ -153,6 +154,7 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu);void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu);void tdx_vcpu_put (struct kvm_vcpu *vcpu); +void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -164,6 +166,7 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -EOPNOTSUPP ; }static inline void tdx_hardware_unsetup (void ) {} +static inline void tdx_hardware_disable (void ) {}static inline bool tdx_is_vm_type_supported (unsigned long type) {return false; }static inline int tdx_offline_cpu (void ) {return 0 ; }
@@ -183,6 +186,7 @@ static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}static inline fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) {return EXIT_FASTPATH_NONE ; }static inline void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {} +static inline void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {}static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v18 77/121 KVM: x86: Add a switch_db_regs flag to handle TDX's auto-switched behavior
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add a flag, KVM_DEBUGREG_AUTO_SWITCHED_GUEST, to skip saving/restoring DRs irrespective of any other flags. TDX-SEAM unconditionally saves and restores guest DRs and reset to architectural INIT state on TD exit. So, KVM needs to save host DRs before TD enter without restoring guest DRs and restore host DRs after TD exit.
Opportunistically convert the KVM_DEBUGREG_* definitions to use BIT().
--- arch/x86/include/asm/kvm_host.h | 10 ++++++++-- arch/x86/kvm/vmx/tdx.c | 1 + arch/x86/kvm/x86.c | 11 ++++++++--- 3 files changed, 17 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +8 -2
@@ -626,8 +626,14 @@ struct kvm_pmu {struct kvm_pmu_ops ;enum { -KVM_DEBUGREG_BP_ENABLED =1 , -KVM_DEBUGREG_WONT_EXIT =2 , +KVM_DEBUGREG_BP_ENABLED =BIT (0 ), +KVM_DEBUGREG_WONT_EXIT =BIT (1 ), +/* +* Guest debug registers (DR0-3 and DR6) are saved/restored by hardware +* on exit from or enter to guest. KVM needn't switch them. Because DR7 +* is cleared on exit from guest, DR7 need to be saved/restored. +*/ +KVM_DEBUGREG_AUTO_SWITCH =BIT (2 ), };struct kvm_mtrr_range {
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -636,6 +636,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.efer =EFER_SCE |EFER_LME |EFER_LMA |EFER_NX ; + vcpu->arch.switch_db_regs =KVM_DEBUGREG_AUTO_SWITCH ; vcpu->arch.cr0_guest_owned_bits =-1ul ; vcpu->arch.cr4_guest_owned_bits =-1ul ;
arch/x86/kvm/x86.c +8 -3
@@ -10973,7 +10973,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)if (vcpu->arch.guest_fpu.xfd_err)wrmsrl (MSR_IA32_XFD_ERR , vcpu->arch.guest_fpu.xfd_err); -if (unlikely (vcpu->arch.switch_db_regs)) { +if (unlikely (vcpu->arch.switch_db_regs & ~KVM_DEBUGREG_AUTO_SWITCH )) {set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg (vcpu->arch.eff_db[1 ],1 );
@@ -11019,6 +11019,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)*/ if (unlikely (vcpu->arch.switch_db_regs &KVM_DEBUGREG_WONT_EXIT )) {WARN_ON (vcpu->guest_debug &KVM_GUESTDBG_USE_HW_BP ); +WARN_ON (vcpu->arch.switch_db_regs &KVM_DEBUGREG_AUTO_SWITCH );static_call (kvm_x86_sync_dirty_debug_regs)(vcpu);kvm_update_dr0123 (vcpu);kvm_update_dr7 (vcpu);
@@ -11031,8 +11032,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)* care about the messed up debug address registers. But if * we have some of them active, restore the old state. */ -if (hw_breakpoint_active ()) -hw_breakpoint_restore (); +if (hw_breakpoint_active ()) { +if (!(vcpu->arch.switch_db_regs &KVM_DEBUGREG_AUTO_SWITCH )) +hw_breakpoint_restore (); +else +set_debugreg (__this_cpu_read (cpu_dr7),7 ); + } vcpu->arch.last_vmentry_cpu = vcpu->cpu; vcpu->arch.last_guest_tsc =kvm_read_l1_tsc (vcpu,rdtsc ());
Signature · 1 line
2.25.1
Re: [PATCH v18 077/121] KVM: x86: Add a switch_db_regs flag to handle TDX's auto-switched behavior
Quoted 7 lines · On 1/23/2024 7:53 AM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Add a flag,
KVM_DEBUGREG_AUTO_SWITCHED_GUEST, to skip saving/restoring DRs irrespective of any other flags. TDX-SEAM unconditionally saves and restores guest DRs and reset to architectural INIT state on TD exit. So, KVM needs to save host DRs before TD enter without restoring guest DRs and restore host DRs after TD exit.
The description here is different from the implementation. The changelog needs to be updated?
Quoted 83 lines
Opportunistically convert the KVM_DEBUGREG_* definitions to use
BIT().Reported-by Xiaoyao Li <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Co-developed-by Chao Gao <[email protected]>Signed-off-by Chao Gao <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/include/asm/kvm_host.h | 10 ++++++++-- arch/x86/kvm/vmx/tdx.c | 1 + arch/x86/kvm/x86.c | 11 ++++++++--- 3 files changed, 17 insertions(+), 5 deletions(-)arch/x86/include/asm/kvm_host.h +8
-2@@ -626,8 +626,14 @@ struct kvm_pmu {struct kvm_pmu_ops ;enum { -KVM_DEBUGREG_BP_ENABLED =1 , -KVM_DEBUGREG_WONT_EXIT =2 , +KVM_DEBUGREG_BP_ENABLED =BIT (0 ), +KVM_DEBUGREG_WONT_EXIT =BIT (1 ), +/* +* Guest debug registers (DR0-3 and DR6) are saved/restored by hardware +* on exit from or enter to guest. KVM needn't switch them. Because DR7 +* is cleared on exit from guest, DR7 need to be saved/restored. +*/ +KVM_DEBUGREG_AUTO_SWITCH =BIT (2 ), };struct kvm_mtrr_range {arch/x86/kvm/vmx/tdx.c +1
-0@@ -636,6 +636,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.efer =EFER_SCE |EFER_LME |EFER_LMA |EFER_NX ; + vcpu->arch.switch_db_regs =KVM_DEBUGREG_AUTO_SWITCH ; vcpu->arch.cr0_guest_owned_bits =-1ul ; vcpu->arch.cr4_guest_owned_bits =-1ul ;arch/x86/kvm/x86.c +8
-3@@ -10973,7 +10973,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)if (vcpu->arch.guest_fpu.xfd_err)wrmsrl (MSR_IA32_XFD_ERR , vcpu->arch.guest_fpu.xfd_err); -if (unlikely (vcpu->arch.switch_db_regs)) { +if (unlikely (vcpu->arch.switch_db_regs & ~KVM_DEBUGREG_AUTO_SWITCH )) {set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg (vcpu->arch.eff_db[1 ],1 );@@ -11019,6 +11019,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)*/ if (unlikely (vcpu->arch.switch_db_regs &KVM_DEBUGREG_WONT_EXIT )) {WARN_ON (vcpu->guest_debug &KVM_GUESTDBG_USE_HW_BP ); +WARN_ON (vcpu->arch.switch_db_regs &KVM_DEBUGREG_AUTO_SWITCH );static_call (kvm_x86_sync_dirty_debug_regs)(vcpu);kvm_update_dr0123 (vcpu);kvm_update_dr7 (vcpu);@@ -11031,8 +11032,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)* care about the messed up debug address registers. But if * we have some of them active, restore the old state. */ -if (hw_breakpoint_active ()) -hw_breakpoint_restore (); +if (hw_breakpoint_active ()) { +if (!(vcpu->arch.switch_db_regs &KVM_DEBUGREG_AUTO_SWITCH )) +hw_breakpoint_restore (); +else +set_debugreg (__this_cpu_read (cpu_dr7),7 ); + } vcpu->arch.last_vmentry_cpu = vcpu->cpu; vcpu->arch.last_guest_tsc =kvm_read_l1_tsc (vcpu,rdtsc ());
PATCH v18 78/121 KVM: TDX: Add support for find pending IRQ in a protected local APIC
In reply to [email protected]From: Sean Christopherson <[email protected]>
Add flag and hook to KVM's local APIC management to support determining whether or not a TDX guest as a pending IRQ. For TDX vCPUs, the virtual APIC page is owned by the TDX module and cannot be accessed by KVM. As a result, registers that are virtualized by the CPU, e.g. PPR, cannot be read or written by KVM. To deliver interrupts for TDX guests, KVM must send an IRQ to the CPU on the posted interrupt notification vector. And to determine if TDX vCPU has a pending interrupt, KVM must check if there is an outstanding notification.
Return "no interrupt" in kvm_apic_has_interrupt() if the guest APIC is protected to short-circuit the various other flows that try to pull an IRQ out of the vAPIC, the only valid operation is querying _if_ an IRQ is pending, KVM can't do anything based on _which_ IRQ is pending.
Intentionally omit sanity checks from other flows, e.g. PPR update, so as not to degrade non-TDX guests with unnecessary checks. A well-behaved KVM and userspace will never reach those flows for TDX guests, but reaching them is not fatal if something does go awry.
Note, this doesn't handle interrupts that have been delivered to the vCPU but not yet recognized by the core, i.e. interrupts that are sitting in vmcs.GUEST_INTR_STATUS. Querying that state requires a SEAMCALL and will be supported in a future patch.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/irq.c | 3 +++ arch/x86/kvm/lapic.c | 3 +++ arch/x86/kvm/lapic.h | 2 ++ arch/x86/kvm/vmx/main.c | 10 ++++++++++ arch/x86/kvm/vmx/tdx.c | 6 ++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ 8 files changed, 28 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -124,6 +124,7 @@ KVM_X86_OP_OPTIONAL(pi_start_assignment) KVM_X86_OP_OPTIONAL(apicv_pre_state_restore) KVM_X86_OP_OPTIONAL(apicv_post_state_restore) KVM_X86_OP_OPTIONAL_RET0(dy_apicv_has_pending_interrupt) +KVM_X86_OP_OPTIONAL(protected_apic_has_interrupt) KVM_X86_OP_OPTIONAL(set_hv_timer) KVM_X86_OP_OPTIONAL(cancel_hv_timer) KVM_X86_OP(setup_mce)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1817,6 +1817,7 @@ struct kvm_x86_ops {void (*apicv_pre_state_restore)(struct kvm_vcpu *vcpu);void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);bool (*dy_apicv_has_pending_interrupt)(struct kvm_vcpu *vcpu); +bool (*protected_apic_has_interrupt)(struct kvm_vcpu *vcpu);int (*set_hv_timer)(struct kvm_vcpu *vcpu,u64 guest_deadline_tsc,bool *expired);
arch/x86/kvm/irq.c +3 -0
@@ -100,6 +100,9 @@ int kvm_cpu_has_interrupt(struct kvm_vcpu *v)if (kvm_cpu_has_extint (v))return 1 ; +if (lapic_in_kernel (v) && v->arch.apic->guest_apic_protected) +return static_call (kvm_x86_protected_apic_has_interrupt)(v); +return kvm_apic_has_interrupt (v) !=-1 ;/* LAPIC */ }EXPORT_SYMBOL_GPL (kvm_cpu_has_interrupt);
arch/x86/kvm/lapic.c +3 -0
@@ -2860,6 +2860,9 @@ int kvm_apic_has_interrupt(struct kvm_vcpu *vcpu)if (!kvm_apic_present (vcpu))return -1 ; +if (apic->guest_apic_protected) +return -1 ; +__apic_update_ppr (apic, &ppr);return apic_has_interrupt_for_ppr (apic, ppr); }
arch/x86/kvm/lapic.h +2 -0
@@ -66,6 +66,8 @@ struct kvm_lapic {bool sw_enabled;bool irr_pending;bool lvt0_in_nmi_mode; +/* Select registers in the vAPIC cannot be read/written. */ +bool guest_apic_protected;/* Number of bits set in ISR. */ s16 isr_count;/* The highest vector set in ISR; if -1 - invalid, must scan ISR. */
arch/x86/kvm/vmx/main.c +10 -0
@@ -98,6 +98,8 @@ static __init int vt_hardware_setup(void)if (enable_tdx) vt_x86_ops.flush_remote_tlbs = vt_flush_remote_tlbs; +else + vt_x86_ops.protected_apic_has_interrupt =NULL ;return 0 ; }
@@ -234,6 +236,13 @@ static void vt_vcpu_load(struct kvm_vcpu *vcpu, int cpu)vmx_vcpu_load (vcpu, cpu); } +static bool vt_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) +{ +KVM_BUG_ON (!is_td_vcpu (vcpu), vcpu->kvm); + +return tdx_protected_apic_has_interrupt (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) {
@@ -425,6 +434,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .sync_pir_to_irr = vmx_sync_pir_to_irr, .deliver_interrupt = vmx_deliver_interrupt, .dy_apicv_has_pending_interrupt = pi_has_pending_interrupt, + .protected_apic_has_interrupt = vt_protected_apic_has_interrupt, .set_tss_addr = vmx_set_tss_addr, .set_identity_map_addr = vmx_set_identity_map_addr,
arch/x86/kvm/vmx/tdx.c +6 -0
@@ -633,6 +633,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu)return -EINVAL ;fpstate_set_confidential (&vcpu->arch.guest_fpu); + vcpu->arch.apic->guest_apic_protected = true; vcpu->arch.efer =EFER_SCE |EFER_LME |EFER_LMA |EFER_NX ;
@@ -674,6 +675,11 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu)local_irq_enable (); } +bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) +{ +return pi_has_pending_interrupt (vcpu); +} +void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {struct vcpu_tdx *tdx =to_tdx (vcpu);
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -155,6 +155,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu);void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu);void tdx_vcpu_put (struct kvm_vcpu *vcpu);void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu); +bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -187,6 +188,7 @@ static inline fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) { return EXIT_FASTPstatic inline void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {} +static inline bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) {return false; }static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v18 79/121 KVM: x86: Assume timer IRQ was injected if APIC state is proteced
In reply to [email protected]From: Sean Christopherson <[email protected]>
If APIC state is protected, i.e. the vCPU is a TDX guest, assume a timer IRQ was injected when deciding whether or not to busy wait in the "timer advanced" path. The "real" vIRR is not readable/writable, so trying to query for a pending timer IRQ will return garbage.
Note, TDX can scour the PIR if it wants to be more precise and skip the "wait" call entirely.
--- arch/x86/kvm/lapic.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c +10 -1
@@ -1774,8 +1774,17 @@ static void apic_update_lvtt(struct kvm_lapic *apic)static bool lapic_timer_int_injected (struct kvm_vcpu *vcpu) {struct kvm_lapic *apic = vcpu->arch.apic; -u32 reg =kvm_lapic_get_reg (apic,APIC_LVTT ); +u32 reg; +/* +* Assume a timer IRQ was "injected" if the APIC is protected. KVM's +* copy of the vIRR is bogus, it's the responsibility of the caller to +* precisely check whether or not a timer IRQ is pending. +*/ +if (apic->guest_apic_protected) +return true; + + reg =kvm_lapic_get_reg (apic,APIC_LVTT );if (kvm_apic_hw_enabled (apic)) {int vec = reg &APIC_VECTOR_MASK ;void *bitmap = apic->regs +APIC_ISR ;
Signature · 1 line
2.25.1
PATCH v18 80/121 KVM: TDX: remove use of struct vcpu_vmx from posted_interrupt.c
In reply to [email protected]From: Isaku Yamahata <[email protected]>
As TDX will use posted_interrupt.c, the use of struct vcpu_vmx is a blocker. Because the members of struct pi_desc pi_desc and struct list_head pi_wakeup_list are only used in posted_interrupt.c, introduce common structure, struct vcpu_pi, make vcpu_vmx and vcpu_tdx has same layout in the top of structure.
To minimize the diff size, avoid code conversion like, vmx->pi_desc => vmx->common->pi_desc. Instead add compile time check if the layout is expected.
--- arch/x86/kvm/vmx/posted_intr.c | 41 ++++++++++++++++++++++++++-------- arch/x86/kvm/vmx/posted_intr.h | 11 +++++++++ arch/x86/kvm/vmx/tdx.c | 1 + arch/x86/kvm/vmx/tdx.h | 8 +++++++ arch/x86/kvm/vmx/vmx.h | 14 +++++++----- 5 files changed, 60 insertions(+), 15 deletions(-)
arch/x86/kvm/vmx/posted_intr.c +32 -9
@@ -11,6 +11,7 @@#include "posted_intr.h" #include "trace.h" #include "vmx.h" +#include "tdx.h" /* * Maintain a per-CPU list of vCPUs that need to be awakened by wakeup_handler()
@@ -31,9 +32,29 @@ static DEFINE_PER_CPU(struct list_head, wakeup_vcpus_on_cpu);*/ static DEFINE_PER_CPU(raw_spinlock_t, wakeup_vcpus_on_cpu_lock); +/* +* The layout of the head of struct vcpu_vmx and struct vcpu_tdx must match with +* struct vcpu_pi. +*/ +static_assert (offsetof(struct vcpu_pi , pi_desc) == + offsetof(struct vcpu_vmx , pi_desc)); +static_assert (offsetof(struct vcpu_pi , pi_wakeup_list) == + offsetof(struct vcpu_vmx , pi_wakeup_list)); +#ifdef CONFIG_INTEL_TDX_HOST +static_assert (offsetof(struct vcpu_pi , pi_desc) == + offsetof(struct vcpu_tdx , pi_desc)); +static_assert (offsetof(struct vcpu_pi , pi_wakeup_list) == + offsetof(struct vcpu_tdx , pi_wakeup_list)); +#endif + +static inline struct vcpu_pi *vcpu_to_pi (struct kvm_vcpu *vcpu) +{ +return (struct vcpu_pi *)vcpu; +} +static inline struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu) { -return &(to_vmx (vcpu)->pi_desc); +return &vcpu_to_pi (vcpu)->pi_desc; }static int pi_try_set_control (struct pi_desc *pi_desc,u64 *pold,u64 new)
@@ -52,8 +73,8 @@ static int pi_try_set_control(struct pi_desc *pi_desc, u64 *pold, u64 new)void vmx_vcpu_pi_load (struct kvm_vcpu *vcpu,int cpu) { -struct pi_desc *pi_desc =vcpu_to_pi_desc (vcpu); -struct vcpu_vmx *vmx =to_vmx (vcpu); +struct vcpu_pi *vcpu_pi =vcpu_to_pi (vcpu); +struct pi_desc *pi_desc = &vcpu_pi->pi_desc;struct pi_desc old, new;unsigned long flags;unsigned int dest;
@@ -90,7 +111,7 @@ void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)*/ if (pi_desc->nv ==POSTED_INTR_WAKEUP_VECTOR ) {raw_spin_lock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); -list_del (&vmx->pi_wakeup_list); +list_del (&vcpu_pi->pi_wakeup_list);raw_spin_unlock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); }
@@ -145,15 +166,15 @@ static bool vmx_can_use_vtd_pi(struct kvm *kvm)*/ static void pi_enable_wakeup_handler (struct kvm_vcpu *vcpu) { -struct pi_desc *pi_desc =vcpu_to_pi_desc (vcpu); -struct vcpu_vmx *vmx =to_vmx (vcpu); +struct vcpu_pi *vcpu_pi =vcpu_to_pi (vcpu); +struct pi_desc *pi_desc = &vcpu_pi->pi_desc;struct pi_desc old, new;unsigned long flags;local_irq_save (flags);raw_spin_lock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); -list_add_tail (&vmx->pi_wakeup_list, +list_add_tail (&vcpu_pi->pi_wakeup_list, &per_cpu (wakeup_vcpus_on_cpu, vcpu->cpu));raw_spin_unlock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu));
@@ -190,7 +211,8 @@ static bool vmx_needs_pi_wakeup(struct kvm_vcpu *vcpu)* notification vector is switched to the one that calls * back to the pi_wakeup_handler() function. */ -return vmx_can_use_ipiv (vcpu) ||vmx_can_use_vtd_pi (vcpu->kvm); +return (vmx_can_use_ipiv (vcpu) && !is_td_vcpu (vcpu)) || +vmx_can_use_vtd_pi (vcpu->kvm); }void vmx_vcpu_pi_put (struct kvm_vcpu *vcpu)
@@ -200,7 +222,8 @@ void vmx_vcpu_pi_put(struct kvm_vcpu *vcpu)if (!vmx_needs_pi_wakeup (vcpu))return ; -if (kvm_vcpu_is_blocking (vcpu) && !vmx_interrupt_blocked (vcpu)) +if (kvm_vcpu_is_blocking (vcpu) && + (is_td_vcpu (vcpu) || !vmx_interrupt_blocked (vcpu)))pi_enable_wakeup_handler (vcpu);/*
arch/x86/kvm/vmx/posted_intr.h +11 -0
@@ -94,6 +94,17 @@ static inline bool pi_test_sn(struct pi_desc *pi_desc) (unsigned long *)&pi_desc->control); } +struct vcpu_pi { +struct kvm_vcpu vcpu; + +/* Posted interrupt descriptor */ +struct pi_desc pi_desc; + +/* Used if this vCPU is waiting for PI notification wakeup. */ +struct list_head pi_wakeup_list; +/* Until here common layout betwwn vcpu_vmx and vcpu_tdx. */ +}; +void vmx_vcpu_pi_load (struct kvm_vcpu *vcpu,int cpu);void vmx_vcpu_pi_put (struct kvm_vcpu *vcpu);void pi_wakeup_handler (void );
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -634,6 +634,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu)fpstate_set_confidential (&vcpu->arch.guest_fpu); vcpu->arch.apic->guest_apic_protected = true; +INIT_LIST_HEAD (&tdx->pi_wakeup_list); vcpu->arch.efer =EFER_SCE |EFER_LME |EFER_LMA |EFER_NX ;
arch/x86/kvm/vmx/tdx.h +8 -0
@@ -4,6 +4,7 @@#ifdef CONFIG_INTEL_TDX_HOST +#include "posted_intr.h" #include "pmu_intel.h" #include "tdx_ops.h"
@@ -67,6 +68,13 @@ union tdx_exit_reason {struct vcpu_tdx {struct kvm_vcpu vcpu; +/* Posted interrupt descriptor */ +struct pi_desc pi_desc; + +/* Used if this vCPU is waiting for PI notification wakeup. */ +struct list_head pi_wakeup_list; +/* Until here same layout to struct vcpu_pi. */ +unsigned long tdvpr_pa;unsigned long *tdvpx_pa;bool td_vcpu_created;
arch/x86/kvm/vmx/vmx.h +8 -6
@@ -235,6 +235,14 @@ struct nested_vmx {struct vcpu_vmx {struct kvm_vcpu vcpu; + +/* Posted interrupt descriptor */ +struct pi_desc pi_desc; + +/* Used if this vCPU is waiting for PI notification wakeup. */ +struct list_head pi_wakeup_list; +/* Until here same layout to struct vcpu_pi. */ +u8 fail;u8 x2apic_msr_bitmap_mode;
@@ -304,12 +312,6 @@ struct vcpu_vmx {union vmx_exit_reason exit_reason; -/* Posted interrupt descriptor */ -struct pi_desc pi_desc; - -/* Used if this vCPU is waiting for PI notification wakeup. */ -struct list_head pi_wakeup_list; -/* Support for a guest hypervisor (nested VMX) */ struct nested_vmx nested;
Signature · 1 line
2.25.1
PATCH v18 81/121 KVM: TDX: Implement interrupt injection
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX supports interrupt inject into vcpu with posted interrupt. Wire up the corresponding kvm x86 operations to posted interrupt. Move kvm_vcpu_trigger_posted_interrupt() from vmx.c to common.h to share the code.
VMX can inject interrupt by setting interrupt information field, VM_ENTRY_INTR_INFO_FIELD, of VMCS. TDX supports interrupt injection only by posted interrupt. Ignore the execution path to access VM_ENTRY_INTR_INFO_FIELD.
As cpu state is protected and apicv is enabled for the TDX guest, VMM can inject interrupt by updating posted interrupt descriptor. Treat interrupt can be injected always.
--- arch/x86/kvm/vmx/common.h | 71 ++++++++++++++++++++++++++ arch/x86/kvm/vmx/main.c | 93 ++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/posted_intr.c | 2 +- arch/x86/kvm/vmx/posted_intr.h | 2 + arch/x86/kvm/vmx/tdx.c | 25 +++++++++ arch/x86/kvm/vmx/vmx.c | 67 +----------------------- arch/x86/kvm/vmx/x86_ops.h | 7 ++- 7 files changed, 190 insertions(+), 77 deletions(-)
arch/x86/kvm/vmx/common.h +71 -0
@@ -4,6 +4,7 @@#include <linux/kvm_host.h> +#include "posted_intr.h" #include "mmu.h" static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t gpa,
@@ -30,4 +31,74 @@ static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t gpa, return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0); } +static inline void kvm_vcpu_trigger_posted_interrupt (struct kvm_vcpu *vcpu, +int pi_vec) +{ +#ifdef CONFIG_SMP +if (vcpu->mode ==IN_GUEST_MODE ) { +/* +* The vector of the virtual has already been set in the PIR. +* Send a notification event to deliver the virtual interrupt +* unless the vCPU is the currently running vCPU, i.e. the +* event is being sent from a fastpath VM-Exit handler, in +* which case the PIR will be synced to the vIRR before +* re-entering the guest. +* +* When the target is not the running vCPU, the following +* possibilities emerge: +* +* Case 1: vCPU stays in non-root mode. Sending a notification +* event posts the interrupt to the vCPU. +* +* Case 2: vCPU exits to root mode and is still runnable. The +* PIR will be synced to the vIRR before re-entering the guest. +* Sending a notification event is ok as the host IRQ handler +* will ignore the spurious event. +* +* Case 3: vCPU exits to root mode and is blocked. vcpu_block() +* has already synced PIR to vIRR and never blocks the vCPU if +* the vIRR is not empty. Therefore, a blocked vCPU here does +* not wait for any requested interrupts in PIR, and sending a +* notification event also results in a benign, spurious event. +*/ + +if (vcpu !=kvm_get_running_vcpu ()) +__apic_send_IPI_mask (get_cpu_mask (vcpu->cpu), pi_vec); +return ; + } +#endif +/* +* The vCPU isn't in the guest; wake the vCPU in case it is blocking, +* otherwise do nothing as KVM will grab the highest priority pending +* IRQ via ->sync_pir_to_irr() in vcpu_enter_guest(). +*/ +kvm_vcpu_wake_up (vcpu); +} + +/* +* Send interrupt to vcpu via posted interrupt way. +* 1. If target vcpu is running(non-root mode), send posted interrupt +* notification to vcpu and hardware will sync PIR to vIRR atomically. +* 2. If target vcpu isn't running(root mode), kick it to pick up the +* interrupt from PIR in next vmentry. +*/ +static inline void __vmx_deliver_posted_interrupt (struct kvm_vcpu *vcpu, +struct pi_desc *pi_desc,int vector) +{ +if (pi_test_and_set_pir (vector, pi_desc)) +return ; + +/* If a previous notification has sent the IPI, nothing to do. */ +if (pi_test_and_set_on (pi_desc)) +return ; + +/* +* The implied barrier in pi_test_and_set_on() pairs with the smp_mb_*() +* after setting vcpu->mode in vcpu_enter_guest(), thus the vCPU is +* guaranteed to see PID.ON=1 and sync the PIR to IRR if triggering a +* posted interrupt "fails" because vcpu->mode != IN_GUEST_MODE. +*/ +kvm_vcpu_trigger_posted_interrupt (vcpu,POSTED_INTR_VECTOR ); +} +#endif /* __KVM_X86_VMX_COMMON_H */
arch/x86/kvm/vmx/main.c +84 -9
127 lines not on this page · raw message
arch/x86/kvm/vmx/posted_intr.c +1 -1
@@ -52,7 +52,7 @@ static inline struct vcpu_pi *vcpu_to_pi(struct kvm_vcpu *vcpu)return (struct vcpu_pi *)vcpu; } -static inline struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu) +struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu) {return &vcpu_to_pi (vcpu)->pi_desc; }
arch/x86/kvm/vmx/posted_intr.h +2 -0
@@ -105,6 +105,8 @@ struct vcpu_pi {/* Until here common layout betwwn vcpu_vmx and vcpu_tdx. */ }; +struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu); +void vmx_vcpu_pi_load (struct kvm_vcpu *vcpu,int cpu);void vmx_vcpu_pi_put (struct kvm_vcpu *vcpu);void pi_wakeup_handler (void );
arch/x86/kvm/vmx/tdx.c +25 -0
@@ -7,6 +7,7 @@#include "capabilities.h" #include "x86_ops.h" +#include "common.h" #include "mmu.h" #include "tdx_arch.h" #include "tdx.h"
@@ -653,6 +654,9 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) tdx->host_state_need_save = true; tdx->host_state_need_restore = false; + tdx->pi_desc.nv =POSTED_INTR_VECTOR ; + tdx->pi_desc.sn =1 ; +return 0 ; }
@@ -660,6 +664,7 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) {struct vcpu_tdx *tdx =to_tdx (vcpu); +vmx_vcpu_pi_load (vcpu, cpu);if (vcpu->cpu == cpu)return ;
@@ -898,6 +903,12 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu); +if (pi_test_on (&tdx->pi_desc)) { + apic->send_IPI_self (POSTED_INTR_VECTOR ); + +kvm_wait_lapic_expire (vcpu); + } +tdx_vcpu_enter_exit (tdx);tdx_user_return_update_cache (vcpu);
@@ -1258,6 +1269,16 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,return tdx_sept_drop_private_spte (kvm, gfn, level, pfn); } +void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode, +int trig_mode,int vector) +{ +struct kvm_vcpu *vcpu = apic->vcpu; +struct vcpu_tdx *tdx =to_tdx (vcpu); + +/* TDX supports only posted interrupt. No lapic emulation. */ +__vmx_deliver_posted_interrupt (vcpu, &tdx->pi_desc, vector); +} +static int tdx_get_capabilities (struct kvm_tdx_cmd *cmd) {struct kvm_tdx_capabilities __user *user_caps;
@@ -2045,6 +2066,10 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)if (ret)return ret; +td_vmcs_write16 (tdx,POSTED_INTR_NV ,POSTED_INTR_VECTOR ); +td_vmcs_write64 (tdx,POSTED_INTR_DESC_ADDR ,__pa (&tdx->pi_desc)); +td_vmcs_setbit32 (tdx,PIN_BASED_VM_EXEC_CONTROL ,PIN_BASED_POSTED_INTR ); + tdx->initialized = true;return 0 ; }
arch/x86/kvm/vmx/vmx.c +1 -66
85 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +6 -1
@@ -58,7 +58,6 @@ int vmx_check_intercept(struct kvm_vcpu *vcpu,bool vmx_apic_init_signal_blocked (struct kvm_vcpu *vcpu);void vmx_migrate_timers (struct kvm_vcpu *vcpu);void vmx_set_virtual_apic_mode (struct kvm_vcpu *vcpu); -void vmx_apicv_pre_state_restore (struct kvm_vcpu *vcpu);bool vmx_check_apicv_inhibit_reasons (enum kvm_apicv_inhibit reason);void vmx_hwapic_irr_update (struct kvm_vcpu *vcpu,int max_irr);void vmx_hwapic_isr_update (int max_isr);
@@ -158,6 +157,9 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu);bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio); +void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode, +int trig_mode,int vector); +int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);void tdx_flush_tlb (struct kvm_vcpu *vcpu);
@@ -191,6 +193,9 @@ static inline void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) {}static inline bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) {return false; }static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; } +static inline void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode, +int trig_mode,int vector) {} +static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }static inline void tdx_flush_tlb (struct kvm_vcpu *vcpu) {}
Signature · 1 line
2.25.1
PATCH v18 82/121 KVM: TDX: Implements vcpu request_immediate_exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Now we are able to inject interrupts into TDX vcpu, it's ready to block TDX vcpu. Wire up kvm x86 methods for blocking/unblocking vcpu for TDX. To unblock on pending events, request immediate exit methods is also needed.
--- arch/x86/kvm/vmx/main.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/main.c +11 -1
@@ -385,6 +385,16 @@ static void vt_enable_irq_window(struct kvm_vcpu *vcpu)vmx_enable_irq_window (vcpu); } +static void vt_request_immediate_exit (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +__kvm_request_immediate_exit (vcpu); +return ; + } + +vmx_request_immediate_exit (vcpu); +} +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {if (is_td_vcpu (vcpu))
@@ -531,7 +541,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .check_intercept = vmx_check_intercept, .handle_exit_irqoff = vmx_handle_exit_irqoff, - .request_immediate_exit = vmx_request_immediate_exit, + .request_immediate_exit = vt_request_immediate_exit, .sched_in = vt_sched_in,
Signature · 1 line
2.25.1
PATCH v18 83/121 KVM: TDX: Implement methods to inject NMI
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX vcpu control structure defines one bit for pending NMI for VMM to inject NMI by setting the bit without knowing TDX vcpu NMI states. Because the vcpu state is protected, VMM can't know about NMI states of TDX vcpu. The TDX module handles actual injection and NMI states transition.
Add methods for NMI and treat NMI can be injected always.
--- arch/x86/kvm/vmx/main.c | 64 +++++++++++++++++++++++++++++++++++--- arch/x86/kvm/vmx/tdx.c | 6 ++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ 3 files changed, 67 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/main.c +59 -5
@@ -319,6 +319,60 @@ static void vt_flush_tlb_guest(struct kvm_vcpu *vcpu)vmx_flush_tlb_guest (vcpu); } +static void vt_inject_nmi (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +tdx_inject_nmi (vcpu); +return ; + } + +vmx_inject_nmi (vcpu); +} + +static int vt_nmi_allowed (struct kvm_vcpu *vcpu,bool for_injection) +{ +/* +* The TDX module manages NMI windows and NMI reinjection, and hides NMI +* blocking, all KVM can do is throw an NMI over the wall. +*/ +if (is_td_vcpu (vcpu)) +return true; + +return vmx_nmi_allowed (vcpu, for_injection); +} + +static bool vt_get_nmi_mask (struct kvm_vcpu *vcpu) +{ +/* +* Assume NMIs are always unmasked. KVM could query PEND_NMI and treat +* NMIs as masked if a previous NMI is still pending, but SEAMCALLs are +* expensive and the end result is unchanged as the only relevant usage +* of get_nmi_mask() is to limit the number of pending NMIs, i.e. it +* only changes whether KVM or the TDX module drops an NMI. +*/ +if (is_td_vcpu (vcpu)) +return false; + +return vmx_get_nmi_mask (vcpu); +} + +static void vt_set_nmi_mask (struct kvm_vcpu *vcpu,bool masked) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_nmi_mask (vcpu, masked); +} + +static void vt_enable_nmi_window (struct kvm_vcpu *vcpu) +{ +/* Refer the comment in vt_get_nmi_mask(). */ +if (is_td_vcpu (vcpu)) +return ; + +vmx_enable_nmi_window (vcpu); +} +static void vt_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) {
@@ -497,14 +551,14 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .get_interrupt_shadow = vt_get_interrupt_shadow, .patch_hypercall = vmx_patch_hypercall, .inject_irq = vt_inject_irq, - .inject_nmi = vmx_inject_nmi, + .inject_nmi = vt_inject_nmi, .inject_exception = vmx_inject_exception, .cancel_injection = vt_cancel_injection, .interrupt_allowed = vt_interrupt_allowed, - .nmi_allowed = vmx_nmi_allowed, - .get_nmi_mask = vmx_get_nmi_mask, - .set_nmi_mask = vmx_set_nmi_mask, - .enable_nmi_window = vmx_enable_nmi_window, + .nmi_allowed = vt_nmi_allowed, + .get_nmi_mask = vt_get_nmi_mask, + .set_nmi_mask = vt_set_nmi_mask, + .enable_nmi_window = vt_enable_nmi_window, .enable_irq_window = vt_enable_irq_window, .update_cr8_intercept = vmx_update_cr8_intercept, .set_virtual_apic_mode = vmx_set_virtual_apic_mode,
arch/x86/kvm/vmx/tdx.c +6 -0
@@ -924,6 +924,12 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)return EXIT_FASTPATH_NONE ; } +void tdx_inject_nmi (struct kvm_vcpu *vcpu) +{ + ++vcpu->stat.nmi_injections; +td_management_write8 (to_tdx (vcpu),TD_VCPU_PEND_NMI ,1 ); +} +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) {td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK );
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -159,6 +159,7 @@ u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio);void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector); +void tdx_inject_nmi (struct kvm_vcpu *vcpu);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -195,6 +196,7 @@ static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)static inline void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector) {} +static inline void tdx_inject_nmi (struct kvm_vcpu *vcpu) {}static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v18 84/121 KVM: VMX: Modify NMI and INTR handlers to take intr_info as function argument
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX uses different ABI to get information about VM exit. Pass intr_info to the NMI and INTR handlers instead of pulling it from vcpu_vmx in preparation for sharing the bulk of the handlers with TDX.
When the guest TD exits to VMM, RAX holds status and exit reason, RCX holds exit qualification etc rather than the VMCS fields because VMM doesn't have access to the VMCS. The eventual code will be
VMX:
- get exit reason, intr_info, exit_qualification, and etc from VMCS
- call NMI/INTR handlers (common code)
TDX:
- get exit reason, intr_info, exit_qualification, and etc from guest registers
- call NMI/INTR handlers (common code)
--- arch/x86/kvm/vmx/vmx.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-)
arch/x86/kvm/vmx/vmx.c +7 -9
@@ -6912,24 +6912,22 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu)rdmsrl (MSR_IA32_XFD_ERR , vcpu->arch.guest_fpu.xfd_err); } -static void handle_exception_irqoff (struct vcpu_vmx *vmx) +static void handle_exception_irqoff (struct kvm_vcpu *vcpu,u32 intr_info) { -u32 intr_info =vmx_get_intr_info (&vmx->vcpu); -/* if exit due to PF check for async PF */ if (is_page_fault (intr_info)) - vmx->vcpu.arch.apf.host_apf_flags =kvm_read_and_reset_apf_flags (); + vcpu->arch.apf.host_apf_flags =kvm_read_and_reset_apf_flags ();/* if exit due to NM, handle before interrupts are enabled */ else if (is_nm_fault (intr_info)) -handle_nm_fault_irqoff (&vmx->vcpu); +handle_nm_fault_irqoff (vcpu);/* Handle machine checks before interrupts are enabled */ else if (is_machine_check (intr_info))kvm_machine_check (); } -static void handle_external_interrupt_irqoff (struct kvm_vcpu *vcpu) +static void handle_external_interrupt_irqoff (struct kvm_vcpu *vcpu, +u32 intr_info) { -u32 intr_info =vmx_get_intr_info (vcpu);unsigned int vector = intr_info &INTR_INFO_VECTOR_MASK ;gate_desc *desc = (gate_desc *)host_idt_base + vector;
@@ -6952,9 +6950,9 @@ void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu)return ;if (vmx->exit_reason.basic ==EXIT_REASON_EXTERNAL_INTERRUPT ) -handle_external_interrupt_irqoff (vcpu); +handle_external_interrupt_irqoff (vcpu,vmx_get_intr_info (vcpu));else if (vmx->exit_reason.basic ==EXIT_REASON_EXCEPTION_NMI ) -handle_exception_irqoff (vmx); +handle_exception_irqoff (vcpu,vmx_get_intr_info (vcpu)); }/*
Signature · 1 line
2.25.1
PATCH v18 85/121 KVM: VMX: Move NMI/exception handler to common helper
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX mostly handles NMI/exception exit mostly the same to VMX case. The difference is how to retrieve exit qualification. To share the code with TDX, move NMI/exception to a common header, common.h.
--- arch/x86/kvm/vmx/common.h | 59 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 68 +++++---------------------------------- 2 files changed, 67 insertions(+), 60 deletions(-)
arch/x86/kvm/vmx/common.h +59 -0
@@ -4,8 +4,67 @@#include <linux/kvm_host.h> +#include <asm/traps.h> +#include "posted_intr.h" #include "mmu.h" +#include "vmcs.h" +#include "x86.h" + +extern unsigned long vmx_host_idt_base; +void vmx_do_interrupt_irqoff (unsigned long entry); +void vmx_do_nmi_irqoff (void ); + +static inline void vmx_handle_nm_fault_irqoff (struct kvm_vcpu *vcpu) +{ +/* +* Save xfd_err to guest_fpu before interrupt is enabled, so the +* MSR value is not clobbered by the host activity before the guest +* has chance to consume it. +* +* Do not blindly read xfd_err here, since this exception might +* be caused by L1 interception on a platform which doesn't +* support xfd at all. +* +* Do it conditionally upon guest_fpu::xfd. xfd_err matters +* only when xfd contains a non-zero value. +* +* Queuing exception is done in vmx_handle_exit. See comment there. +*/ +if (vcpu->arch.guest_fpu.fpstate->xfd) +rdmsrl (MSR_IA32_XFD_ERR , vcpu->arch.guest_fpu.xfd_err); +} + +static inline void vmx_handle_exception_irqoff (struct kvm_vcpu *vcpu, +u32 intr_info) +{ +/* if exit due to PF check for async PF */ +if (is_page_fault (intr_info)) + vcpu->arch.apf.host_apf_flags =kvm_read_and_reset_apf_flags (); +/* if exit due to NM, handle before interrupts are enabled */ +else if (is_nm_fault (intr_info)) +vmx_handle_nm_fault_irqoff (vcpu); +/* Handle machine checks before interrupts are enabled */ +else if (is_machine_check (intr_info)) +kvm_machine_check (); +} + +static inline void vmx_handle_external_interrupt_irqoff (struct kvm_vcpu *vcpu, +u32 intr_info) +{ +unsigned int vector = intr_info &INTR_INFO_VECTOR_MASK ; +gate_desc *desc = (gate_desc *)vmx_host_idt_base + vector; + +if (KVM_BUG (!is_external_intr (intr_info), vcpu->kvm, +"unexpected VM-Exit interrupt info: 0x%x" , intr_info)) +return ; + +kvm_before_interrupt (vcpu,KVM_HANDLING_IRQ ); +vmx_do_interrupt_irqoff (gate_offset (desc)); +kvm_after_interrupt (vcpu); + + vcpu->arch.at_instruction_boundary = true; +}static inline int __vmx_handle_ept_violation (struct kvm_vcpu *vcpu,gpa_t gpa,unsigned long exit_qualification)
arch/x86/kvm/vmx/vmx.c +8 -60
111 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v18 86/121 KVM: x86: Split core of hypercall emulation to helper function
In reply to [email protected]From: Sean Christopherson <[email protected]>
By necessity, TDX will use a different register ABI for hypercalls. Break out the core functionality so that it may be reused for TDX.
--- arch/x86/include/asm/kvm_host.h | 4 +++ arch/x86/kvm/x86.c | 56 ++++++++++++++++++++++----------- 2 files changed, 42 insertions(+), 18 deletions(-)
arch/x86/include/asm/kvm_host.h +4 -0
@@ -2186,6 +2186,10 @@ static inline void kvm_clear_apicv_inhibit(struct kvm *kvm,kvm_set_or_clear_apicv_inhibit (kvm, reason, false); } +unsigned long __kvm_emulate_hypercall (struct kvm_vcpu *vcpu,unsigned long nr, +unsigned long a0,unsigned long a1, +unsigned long a2,unsigned long a3, +int op_64_bit,int cpl);int kvm_emulate_hypercall (struct kvm_vcpu *vcpu); int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,
arch/x86/kvm/x86.c +38 -18
89 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v18 87/121 KVM: TDX: Add a place holder to handle TDX VM exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up handle_exit and handle_exit_irqoff methods and add a place holder to handle VM exit. Add helper functions to get exit info, exit qualification, etc.
--- arch/x86/kvm/vmx/main.c | 37 ++++++++++++- arch/x86/kvm/vmx/tdx.c | 110 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 10 ++++ 3 files changed, 154 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/main.c +34 -3
@@ -243,6 +243,25 @@ static bool vt_protected_apic_has_interrupt(struct kvm_vcpu *vcpu)return tdx_protected_apic_has_interrupt (vcpu); } +static int vt_handle_exit (struct kvm_vcpu *vcpu, +enum exit_fastpath_completion fastpath) +{ +if (is_td_vcpu (vcpu)) +return tdx_handle_exit (vcpu, fastpath); + +return vmx_handle_exit (vcpu, fastpath); +} + +static void vt_handle_exit_irqoff (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +tdx_handle_exit_irqoff (vcpu); +return ; + } + +vmx_handle_exit_irqoff (vcpu); +} +static void vt_apicv_pre_state_restore (struct kvm_vcpu *vcpu) {struct pi_desc *pi =vcpu_to_pi_desc (vcpu);
@@ -449,6 +468,18 @@ static void vt_request_immediate_exit(struct kvm_vcpu *vcpu)vmx_request_immediate_exit (vcpu); } +static void vt_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason, +u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code) +{ +if (is_td_vcpu (vcpu)) { +tdx_get_exit_info (vcpu, reason, info1, info2, intr_info, + error_code); +return ; + } + +vmx_get_exit_info (vcpu, reason, info1, info2, intr_info, error_code); +} +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {if (is_td_vcpu (vcpu))
@@ -544,7 +575,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_pre_run = vt_vcpu_pre_run, .vcpu_run = vt_vcpu_run, - .handle_exit = vmx_handle_exit, + .handle_exit = vt_handle_exit, .skip_emulated_instruction = vmx_skip_emulated_instruction, .update_emulated_instruction = vmx_update_emulated_instruction, .set_interrupt_shadow = vt_set_interrupt_shadow,
@@ -579,7 +610,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_identity_map_addr = vmx_set_identity_map_addr, .get_mt_mask = vt_get_mt_mask, - .get_exit_info = vmx_get_exit_info, + .get_exit_info = vt_get_exit_info, .vcpu_after_set_cpuid = vmx_vcpu_after_set_cpuid,
@@ -593,7 +624,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .load_mmu_pgd = vt_load_mmu_pgd, .check_intercept = vmx_check_intercept, - .handle_exit_irqoff = vmx_handle_exit_irqoff, + .handle_exit_irqoff = vt_handle_exit_irqoff, .request_immediate_exit = vt_request_immediate_exit,
arch/x86/kvm/vmx/tdx.c +110 -0
134 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +10 -0
@@ -155,11 +155,16 @@ void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu);void tdx_vcpu_put (struct kvm_vcpu *vcpu);void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu);bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu); +void tdx_handle_exit_irqoff (struct kvm_vcpu *vcpu); +int tdx_handle_exit (struct kvm_vcpu *vcpu, +enum exit_fastpath_completion fastpath);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector);void tdx_inject_nmi (struct kvm_vcpu *vcpu); +void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason, +u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -192,11 +197,16 @@ static inline void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {}static inline bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) {return false; } +static inline void tdx_handle_exit_irqoff (struct kvm_vcpu *vcpu) {} +static inline int tdx_handle_exit (struct kvm_vcpu *vcpu, +enum exit_fastpath_completion fastpath) {return 0 ; }static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector) {}static inline void tdx_inject_nmi (struct kvm_vcpu *vcpu) {} +static inline void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason,u64 *info1, +u64 *info2,u32 *intr_info,u32 *error_code) {}static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v18 88/121 KVM: TDX: Handle vmentry failure for INTEL TD guest
In reply to [email protected]From: Yao Yuan <[email protected]>
TDX module passes control back to VMM if it failed to vmentry for a TD, use same exit reason to notify user space, align with VMX.
If VMM corrupted TD VMCS, machine check during entry can happens. vm exit reason will be EXIT_REASON_MCE_DURING_VMENTRY. If VMM corrupted TD VMCS with debug TD by TDH.VP.WR, the exit reason would be EXIT_REASON_INVALID_STATE or EXIT_REASON_MSR_LOAD_FAIL.
--- arch/x86/kvm/vmx/tdx.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/x86/kvm/vmx/tdx.c +22 -0
@@ -1365,6 +1365,28 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) goto unhandled_exit; } +/* +* When TDX module saw VMEXIT_REASON_FAILED_VMENTER_MC etc, TDH.VP.ENTER +* returns with TDX_SUCCESS | exit_reason with failed_vmentry = 1. +* Because TDX module maintains TD VMCS correctness, usually vmentry +* failure shouldn't happen. In some corner cases it can happen. For +* example +* - machine check during entry: EXIT_REASON_MCE_DURING_VMENTRY +* - TDH.VP.WR with debug TD. VMM can corrupt TD VMCS +* - EXIT_REASON_INVALID_STATE +* - EXIT_REASON_MSR_LOAD_FAIL +*/ +if (unlikely (exit_reason.failed_vmentry)) { +pr_err ("TDExit: exit_reason 0x%016llx qualification=%016lx ext_qualification=%016lx\n" , + exit_reason.full,tdexit_exit_qual (vcpu),tdexit_ext_exit_qual (vcpu)); + vcpu->run->exit_reason =KVM_EXIT_FAIL_ENTRY ; + vcpu->run->fail_entry.hardware_entry_failure_reason + = exit_reason.full; + vcpu->run->fail_entry.cpu = vcpu->arch.last_vmentry_cpu; + +return 0 ; + } +WARN_ON_ONCE (fastpath !=EXIT_FASTPATH_NONE );switch (exit_reason.basic) {
Signature · 1 line
2.25.1
PATCH v18 89/121 KVM: TDX: handle EXIT_REASON_OTHER_SMI
In reply to [email protected]From: Isaku Yamahata <[email protected]>
If the control reaches EXIT_REASON_OTHER_SMI, #SMI is delivered and handled right after returning from the TDX module to KVM nothing needs to be done in KVM. Continue TDX vcpu execution.
--- arch/x86/include/uapi/asm/vmx.h | 1 + arch/x86/kvm/vmx/tdx.c | 7 +++++++ 2 files changed, 8 insertions(+)
arch/x86/include/uapi/asm/vmx.h +1 -0
@@ -34,6 +34,7 @@#define EXIT_REASON_TRIPLE_FAULT 2#define EXIT_REASON_INIT_SIGNAL 3#define EXIT_REASON_SIPI_SIGNAL 4 +#define EXIT_REASON_OTHER_SMI 6#define EXIT_REASON_INTERRUPT_WINDOW 7#define EXIT_REASON_NMI_WINDOW 8
arch/x86/kvm/vmx/tdx.c +7 -0
@@ -1390,6 +1390,13 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath)WARN_ON_ONCE (fastpath !=EXIT_FASTPATH_NONE );switch (exit_reason.basic) { +case EXIT_REASON_OTHER_SMI : +/* +* If reach here, it's not a Machine Check System Management +* Interrupt(MSMI). #SMI is delivered and handled right after +* SEAMRET, nothing needs to be done in KVM. +*/ +return 1 ;default :break ; }
Signature · 1 line
2.25.1
PATCH v18 90/121 KVM: TDX: handle ept violation/misconfig exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
On EPT violation, call a common function, __vmx_handle_ept_violation() to trigger x86 MMU code. On EPT misconfiguration, exit to ring 3 with KVM_EXIT_UNKNOWN. because EPT misconfiguration can't happen as MMIO is trigged by TDG.VP.VMCALL. No point to set a misconfiguration value for the fast path.
---
v14 -> v15:
- use PFERR_GUEST_ENC_MASK to tell the fault is private
--- arch/x86/kvm/vmx/common.h | 3 +++ arch/x86/kvm/vmx/tdx.c | 49 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+)
arch/x86/kvm/vmx/common.h +3 -0
@@ -87,6 +87,9 @@ static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t gpa, error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ?PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; +if (kvm_is_private_gpa (vcpu->kvm, gpa)) + error_code |=PFERR_GUEST_ENC_MASK ; +return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); }
arch/x86/kvm/vmx/tdx.c +49 -0
61 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v18 91/121 KVM: TDX: handle EXCEPTION_NMI and EXTERNAL_INTERRUPT
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because guest TD state is protected, exceptions in guest TDs can't be intercepted. TDX VMM doesn't need to handle exceptions. tdx_handle_exit_irqoff() handles NMI and machine check. Ignore NMI and machine check and continue guest TD execution.
For external interrupt, increment stats same to the VMX case.
--- arch/x86/kvm/vmx/tdx.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
arch/x86/kvm/vmx/tdx.c +23 -0
@@ -968,6 +968,25 @@ void tdx_handle_exit_irqoff(struct kvm_vcpu *vcpu)vmx_handle_exception_irqoff (vcpu,tdexit_intr_info (vcpu)); } +static int tdx_handle_exception (struct kvm_vcpu *vcpu) +{ +u32 intr_info =tdexit_intr_info (vcpu); + +if (is_nmi (intr_info) ||is_machine_check (intr_info)) +return 1 ; + +kvm_pr_unimpl ("unexpected exception 0x%x(exit_reason 0x%llx qual 0x%lx)\n" , + intr_info, +to_tdx (vcpu)->exit_reason.full,tdexit_exit_qual (vcpu)); +return -EFAULT ; +} + +static int tdx_handle_external_interrupt (struct kvm_vcpu *vcpu) +{ + ++vcpu->stat.irq_exits; +return 1 ; +} +static int tdx_handle_triple_fault (struct kvm_vcpu *vcpu) { vcpu->run->exit_reason =KVM_EXIT_SHUTDOWN ;
@@ -1435,6 +1454,10 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath)WARN_ON_ONCE (fastpath !=EXIT_FASTPATH_NONE );switch (exit_reason.basic) { +case EXIT_REASON_EXCEPTION_NMI : +return tdx_handle_exception (vcpu); +case EXIT_REASON_EXTERNAL_INTERRUPT : +return tdx_handle_external_interrupt (vcpu);case EXIT_REASON_EPT_VIOLATION :return tdx_handle_ept_violation (vcpu);case EXIT_REASON_EPT_MISCONFIG :
Signature · 1 line
2.25.1
PATCH v18 92/121 KVM: TDX: Handle EXIT_REASON_OTHER_SMI with MSMI
In reply to [email protected]From: Isaku Yamahata <[email protected]>
When BIOS eMCA MCE-SMI morphing is enabled, the #MC is morphed to MSMI (Machine Check System Management Interrupt). Then the SMI causes TD exit with the read reason of EXIT_REASON_OTHER_SMI with MSMI bit set in the exit qualification to KVM instead of EXIT_REASON_EXCEPTION_NMI with MC exception.
Handle EXIT_REASON_OTHER_SMI with MSMI bit set in the exit qualification as MCE(Machine Check Exception) happened during TD guest running.
--- arch/x86/kvm/vmx/tdx.c | 40 ++++++++++++++++++++++++++++++++++--- arch/x86/kvm/vmx/tdx_arch.h | 2 ++ 2 files changed, 39 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/tdx.c +37 -3
@@ -966,6 +966,30 @@ void tdx_handle_exit_irqoff(struct kvm_vcpu *vcpu) tdexit_intr_info(vcpu)); else if (exit_reason == EXIT_REASON_EXCEPTION_NMI) vmx_handle_exception_irqoff(vcpu, tdexit_intr_info(vcpu)); + else if (unlikely(tdx->exit_reason.non_recoverable || + tdx->exit_reason.error)) { +/* +* The only reason it gets EXIT_REASON_OTHER_SMI is there is an +* #MSMI(Machine Check System Management Interrupt) with +* exit_qualification bit 0 set in TD guest. +* The #MSMI is delivered right after SEAMCALL returns, +* and an #MC is delivered to host kernel after SMI handler +* returns. +* +* The #MC right after SEAMCALL is fixed up and skipped in #MC +* handler because it's an #MC happens in TD guest we cannot +* handle it with host's context. +* +* Call KVM's machine check handler explicitly here. +*/ +if (tdx->exit_reason.basic ==EXIT_REASON_OTHER_SMI ) { +unsigned long exit_qual; + + exit_qual =tdexit_exit_qual (vcpu); +if (exit_qual &TD_EXIT_OTHER_SMI_IS_MSMI ) +kvm_machine_check (); + } + } }static int tdx_handle_exception (struct kvm_vcpu *vcpu)
@@ -1426,6 +1450,11 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) exit_reason.full, exit_reason.basic,to_kvm_tdx (vcpu->kvm)->hkid, set_hkid_to_hpa(0, to_kvm_tdx(vcpu->kvm)->hkid)); + +/* +* tdx_handle_exit_irqoff() handled EXIT_REASON_OTHER_SMI. It +* must be handled before enabling preemption because it's #MC. +*/ goto unhandled_exit; }
@@ -1464,9 +1493,14 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath)return tdx_handle_ept_misconfig (vcpu);case EXIT_REASON_OTHER_SMI :/* -* If reach here, it's not a Machine Check System Management -* Interrupt(MSMI). #SMI is delivered and handled right after -* SEAMRET, nothing needs to be done in KVM. +* Unlike VMX, all the SMI in SEAM non-root mode (i.e. when +* TD guest vcpu is running) will cause TD exit to TDX module, +* then SEAMRET to KVM. Once it exits to KVM, SMI is delivered +* and handled right away. +* +* - If it's an Machine Check System Management Interrupt +* (MSMI), it's handled above due to non_recoverable bit set. +* - If it's not an MSMI, don't need to do anything here. */ return 1 ;default :
arch/x86/kvm/vmx/tdx_arch.h +2 -0
@@ -48,6 +48,8 @@#define TDG_VP_VMCALL_REPORT_FATAL_ERROR 0x10003#define TDG_VP_VMCALL_SETUP_EVENT_NOTIFY_INTERRUPT 0x10004 +#define TD_EXIT_OTHER_SMI_IS_MSMI BIT(1) +/* TDX control structure (TDR/TDCS/TDVPS) field access codes */ #define TDX_NON_ARCH BIT_ULL(63)#define TDX_CLASS_SHIFT 56
Signature · 1 line
2.25.1
PATCH v18 93/121 KVM: TDX: Add a place holder for handler of TDX hypercalls (TDG.VP.VMCALL)
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The TDX module specification defines TDG.VP.VMCALL API (TDVMCALL for short) for the guest TD to call hypercall to VMM. When the guest TD issues TDG.VP.VMCALL, the guest TD exits to VMM with a new exit reason of TDVMCALL. The arguments from the guest TD and returned values from the VMM are passed in the guest registers. The guest RCX registers indicates which registers are used. Define helper functions to access those registers as ABI.
Define the TDVMCALL exit reason, which is carved out from the VMX exit reason namespace as the TDVMCALL exit from TDX guest to TDX-SEAM is really just a VM-Exit. Add a place holder to handle TDVMCALL exit.
--- arch/x86/include/uapi/asm/vmx.h | 4 ++- arch/x86/kvm/vmx/tdx.c | 53 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 13 ++++++++ 3 files changed, 69 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/vmx.h +3 -1
@@ -93,6 +93,7 @@#define EXIT_REASON_TPAUSE 68#define EXIT_REASON_BUS_LOCK 74#define EXIT_REASON_NOTIFY 75 +#define EXIT_REASON_TDCALL 77#define VMX_EXIT_REASONS \ { EXIT_REASON_EXCEPTION_NMI, "EXCEPTION_NMI" }, \
@@ -156,7 +157,8 @@ { EXIT_REASON_UMWAIT, "UMWAIT" }, \ { EXIT_REASON_TPAUSE, "TPAUSE" }, \ { EXIT_REASON_BUS_LOCK, "BUS_LOCK" }, \ - { EXIT_REASON_NOTIFY, "NOTIFY" } + { EXIT_REASON_NOTIFY, "NOTIFY" }, \ + { EXIT_REASON_TDCALL, "TDCALL" } #define VMX_EXIT_REASON_FLAGS \ { VMX_EXIT_REASONS_FAILED_VMENTRY, "FAILED_VMENTRY" }
arch/x86/kvm/vmx/tdx.c +53 -0
77 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +13 -0
@@ -81,6 +81,19 @@ struct vcpu_tdx {struct list_head cpu_list; +union { +struct { +union { +struct { +u16 gpr_mask; +u16 xmm_mask; + }; +u32 regs_mask; + }; +u32 reserved; + }; +u64 rcx; + } tdvmcall;union tdx_exit_reason exit_reason;bool initialized;
Signature · 1 line
2.25.1
PATCH v18 94/121 KVM: TDX: handle KVM hypercall with TDG.VP.VMCALL
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The TDX Guest-Host communication interface (GHCI) specification defines
the ABI for the guest TD to issue hypercall. It reserves vendor specific
arguments for VMM specific use. Use it as KVM hypercall and handle it.
--- arch/x86/kvm/vmx/tdx.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
arch/x86/kvm/vmx/tdx.c +33 -0
@@ -1058,8 +1058,41 @@ static int tdx_handle_triple_fault(struct kvm_vcpu *vcpu)return 0 ; } +static int tdx_emulate_vmcall (struct kvm_vcpu *vcpu) +{ +unsigned long nr, a0, a1, a2, a3, ret; + +/* +* ABI for KVM tdvmcall argument: +* In Guest-Hypervisor Communication Interface(GHCI) specification, +* Non-zero leaf number (R10 != 0) is defined to indicate +* vendor-specific. KVM uses this for KVM hypercall. NOTE: KVM +* hypercall number starts from one. Zero isn't used for KVM hypercall +* number. +* +* R10: KVM hypercall number +* arguments: R11, R12, R13, R14. +*/ + nr =kvm_r10_read (vcpu); + a0 =kvm_r11_read (vcpu); + a1 =kvm_r12_read (vcpu); + a2 =kvm_r13_read (vcpu); + a3 =kvm_r14_read (vcpu); + + ret =__kvm_emulate_hypercall (vcpu, nr, a0, a1, a2, a3, true,0 ); + +tdvmcall_set_return_code (vcpu, ret); + +if (nr ==KVM_HC_MAP_GPA_RANGE && !ret) +return 0 ; +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) { +if (tdvmcall_exit_type (vcpu)) +return tdx_emulate_vmcall (vcpu); +switch (tdvmcall_leaf (vcpu)) {default :break ;
Signature · 1 line
2.25.1
PATCH v18 95/121 KVM: TDX: Add KVM Exit for TDX TDG.VP.VMCALL
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Some of TDG.VP.VMCALL require device model, for example, qemu, to handle them on behalf of kvm kernel module. TDG_VP_VMCALL_REPORT_FATAL_ERROR, TDG_VP_VMCALL_MAP_GPA, TDG_VP_VMCALL_SETUP_EVENT_NOTIFY_INTERRUPT, and TDG_VP_VMCALL_GET_QUOTE requires user space VMM handling.
Introduce new kvm exit, KVM_EXIT_TDX, and functions to setup it. TDG_VP_VMCALL_INVALID_OPERAND is set as default return value to avoid random value. Device model should update R10 if necessary.
---
v14 -> v15:
- updated struct kvm_tdx_exit with union
- export constants for reg bitmask
--- arch/x86/kvm/vmx/tdx.c | 84 ++++++++++++++++++++++++++++++++++++- include/uapi/linux/kvm.h | 89 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/tdx.c +82 -2
@@ -1088,6 +1088,78 @@ static int tdx_emulate_vmcall(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_complete_vp_vmcall (struct kvm_vcpu *vcpu) +{ +struct kvm_tdx_vmcall *tdx_vmcall = &vcpu->run->tdx.u.vmcall; +__u64 reg_mask =kvm_rcx_read (vcpu); + +#define COPY_REG (MASK ,REG ) \ + do { \ + if (reg_mask & TDX_VMCALL_REG_MASK_ ## MASK) \ + kvm_## REG ## _write(vcpu, tdx_vmcall->out_ ## REG); \ + } while (0) + + +COPY_REG (R10 , r10); +COPY_REG (R11 , r11); +COPY_REG (R12 , r12); +COPY_REG (R13 , r13); +COPY_REG (R14 , r14); +COPY_REG (R15 , r15); +COPY_REG (RBX , rbx); +COPY_REG (RDI , rdi); +COPY_REG (RSI , rsi); +COPY_REG (R8 , r8); +COPY_REG (R9 , r9); +COPY_REG (RDX , rdx); + +#undef COPY_REG + +return 1 ; +} + +static int tdx_vp_vmcall_to_user (struct kvm_vcpu *vcpu) +{ +struct kvm_tdx_vmcall *tdx_vmcall = &vcpu->run->tdx.u.vmcall; +__u64 reg_mask; + + vcpu->arch.complete_userspace_io = tdx_complete_vp_vmcall; +memset (tdx_vmcall,0 ,sizeof (*tdx_vmcall)); + + vcpu->run->exit_reason =KVM_EXIT_TDX ; + vcpu->run->tdx.type =KVM_EXIT_TDX_VMCALL ; + + reg_mask =kvm_rcx_read (vcpu); + tdx_vmcall->reg_mask = reg_mask; + +#define COPY_REG (MASK ,REG ) \ + do { \ + if (reg_mask & TDX_VMCALL_REG_MASK_ ## MASK) { \ + tdx_vmcall->in_ ## REG = kvm_ ## REG ## _read(vcpu); \ + tdx_vmcall->out_ ## REG = tdx_vmcall->in_ ## REG; \ + } \ + } while (0) + + +COPY_REG (R10 , r10); +COPY_REG (R11 , r11); +COPY_REG (R12 , r12); +COPY_REG (R13 , r13); +COPY_REG (R14 , r14); +COPY_REG (R15 , r15); +COPY_REG (RBX , rbx); +COPY_REG (RDI , rdi); +COPY_REG (RSI , rsi); +COPY_REG (R8 , r8); +COPY_REG (R9 , r9); +COPY_REG (RDX , rdx); + +#undef COPY_REG + +/* notify userspace to handle the request */ +return 0 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1098,8 +1170,16 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)break ; } -tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); -return 1 ; +/* +* Unknown VMCALL. Toss the request to the user space VMM, e.g. qemu, +* as it may know how to handle. +* +* Those VMCALLs require user space VMM: +* TDG_VP_VMCALL_REPORT_FATAL_ERROR, TDG_VP_VMCALL_MAP_GPA, +* TDG_VP_VMCALL_SETUP_EVENT_NOTIFY_INTERRUPT, and +* TDG_VP_VMCALL_GET_QUOTE. +*/ +return tdx_vp_vmcall_to_user (vcpu); }void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level)
include/uapi/linux/kvm.h +89 -0
107 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v18 96/121 KVM: TDX: Handle TDX PV CPUID hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV CPUID hypercall to the KVM backend function.
--- arch/x86/kvm/vmx/tdx.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/x86/kvm/vmx/tdx.c +22 -0
@@ -1160,12 +1160,34 @@ static int tdx_vp_vmcall_to_user(struct kvm_vcpu *vcpu)return 0 ; } +static int tdx_emulate_cpuid (struct kvm_vcpu *vcpu) +{ +u32 eax, ebx, ecx, edx; + +/* EAX and ECX for cpuid is stored in R12 and R13. */ + eax =tdvmcall_a0_read (vcpu); + ecx =tdvmcall_a1_read (vcpu); + +kvm_cpuid (vcpu, &eax, &ebx, &ecx, &edx, false); + +tdvmcall_a0_write (vcpu, eax); +tdvmcall_a1_write (vcpu, ebx); +tdvmcall_a2_write (vcpu, ecx); +tdvmcall_a3_write (vcpu, edx); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); + +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))return tdx_emulate_vmcall (vcpu);switch (tdvmcall_leaf (vcpu)) { +case EXIT_REASON_CPUID : +return tdx_emulate_cpuid (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v18 97/121 KVM: TDX: Handle TDX PV HLT hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV HLT hypercall to the KVM backend function.
---
v18:
- drop buggy_hlt_workaround and use TDH.VP.RD(TD_VCPU_STATE_DETAILS)
--- arch/x86/kvm/vmx/tdx.c | 26 +++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 3 +++ 2 files changed, 28 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/tdx.c +25 -1
@@ -738,7 +738,18 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu)bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) { -return pi_has_pending_interrupt (vcpu); +bool ret =pi_has_pending_interrupt (vcpu); +union tdx_vcpu_state_details details; +struct vcpu_tdx *tdx =to_tdx (vcpu); + +if (ret || vcpu->arch.mp_state !=KVM_MP_STATE_HALTED ) +return true; + +if (tdx->interrupt_disabled_hlt) +return false; + + details.full =td_state_non_arch_read64 (tdx,TD_VCPU_STATE_DETAILS_NON_ARCH ); +return !!details.vmxip; }void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu)
@@ -1180,6 +1191,17 @@ static int tdx_emulate_cpuid(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_emulate_hlt (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); + +/* See tdx_protected_apic_has_interrupt() to avoid heavy seamcall */ + tdx->interrupt_disabled_hlt =tdvmcall_a0_read (vcpu); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return kvm_emulate_halt_noskip (vcpu); +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1188,6 +1210,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)switch (tdvmcall_leaf (vcpu)) {case EXIT_REASON_CPUID :return tdx_emulate_cpuid (vcpu); +case EXIT_REASON_HLT : +return tdx_emulate_hlt (vcpu);default :break ; }
arch/x86/kvm/vmx/tdx.h +3 -0
@@ -102,6 +102,8 @@ struct vcpu_tdx {bool host_state_need_restore;u64 msr_host_kernel_gs_base; +bool interrupt_disabled_hlt; +/* * Dummy to make pmu_intel not corrupt memory. * TODO: Support PMU for TDX. Future work.
@@ -225,6 +227,7 @@ TDX_BUILD_TDVPS_ACCESSORS(32, VMCS, vmcs);TDX_BUILD_TDVPS_ACCESSORS (64 ,VMCS , vmcs);TDX_BUILD_TDVPS_ACCESSORS (8 ,MANAGEMENT , management); +TDX_BUILD_TDVPS_ACCESSORS (64 ,STATE_NON_ARCH , state_non_arch); static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field) {
Signature · 1 line
2.25.1
PATCH v18 98/121 KVM: TDX: Handle TDX PV port io hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV port IO hypercall to the KVM backend function.
---
v18:
- Fix out case to set R10 and R11 correctly when user space handled port out.
--- arch/x86/kvm/vmx/tdx.c | 67 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)
arch/x86/kvm/vmx/tdx.c +67 -0
@@ -1202,6 +1202,71 @@ static int tdx_emulate_hlt(struct kvm_vcpu *vcpu)return kvm_emulate_halt_noskip (vcpu); } +static int tdx_complete_pio_out (struct kvm_vcpu *vcpu) +{ +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +tdvmcall_set_return_val (vcpu,0 ); +return 1 ; +} + +static int tdx_complete_pio_in (struct kvm_vcpu *vcpu) +{ +struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt; +unsigned long val =0 ; +int ret; + +WARN_ON_ONCE (vcpu->arch.pio.count !=1 ); + + ret = ctxt->ops->pio_in_emulated (ctxt, vcpu->arch.pio.size, + vcpu->arch.pio.port, &val,1 ); +WARN_ON_ONCE (!ret); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +tdvmcall_set_return_val (vcpu, val); + +return 1 ; +} + +static int tdx_emulate_io (struct kvm_vcpu *vcpu) +{ +struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt; +unsigned long val =0 ; +unsigned int port; +int size, ret; +bool write; + + ++vcpu->stat.io_exits; + + size =tdvmcall_a0_read (vcpu); + write =tdvmcall_a1_read (vcpu); + port =tdvmcall_a2_read (vcpu); + +if (size !=1 && size !=2 && size !=4 ) { +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } + +if (write) { + val =tdvmcall_a3_read (vcpu); + ret = ctxt->ops->pio_out_emulated (ctxt, size, port, &val,1 ); + +/* No need for a complete_userspace_io callback. */ + vcpu->arch.pio.count =0 ; + }else + ret = ctxt->ops->pio_in_emulated (ctxt, size, port, &val,1 ); + +if (ret) +tdvmcall_set_return_val (vcpu, val); +else { +if (write) + vcpu->arch.complete_userspace_io = tdx_complete_pio_out; +else + vcpu->arch.complete_userspace_io = tdx_complete_pio_in; + } + +return ret; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1212,6 +1277,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_cpuid (vcpu);case EXIT_REASON_HLT :return tdx_emulate_hlt (vcpu); +case EXIT_REASON_IO_INSTRUCTION : +return tdx_emulate_io (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v18 99/121 KVM: TDX: Handle TDX PV MMIO hypercall
In reply to [email protected]From: Sean Christopherson <[email protected]>
Export kvm_io_bus_read and kvm_mmio tracepoint and wire up TDX PV MMIO hypercall to the KVM backend functions.
kvm_io_bus_read/write() searches KVM device emulated in kernel of the given MMIO address and emulates the MMIO. As TDX PV MMIO also needs it, export kvm_io_bus_read(). kvm_io_bus_write() is already exported. TDX PV MMIO emulates some of MMIO itself. To add trace point consistently with x86 kvm, export kvm_mmio tracepoint.
--- arch/x86/kvm/vmx/tdx.c | 114 +++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 1 + virt/kvm/kvm_main.c | 2 + 3 files changed, 117 insertions(+)
arch/x86/kvm/vmx/tdx.c +114 -0
@@ -1267,6 +1267,118 @@ static int tdx_emulate_io(struct kvm_vcpu *vcpu)return ret; } +static int tdx_complete_mmio (struct kvm_vcpu *vcpu) +{ +unsigned long val =0 ; +gpa_t gpa; +int size; + +KVM_BUG_ON (vcpu->mmio_needed !=1 , vcpu->kvm); + vcpu->mmio_needed =0 ; + +if (!vcpu->mmio_is_write) { + gpa = vcpu->mmio_fragments[0 ].gpa; + size = vcpu->mmio_fragments[0 ].len; + +memcpy (&val, vcpu->run->mmio.data, size); +tdvmcall_set_return_val (vcpu, val); +trace_kvm_mmio (KVM_TRACE_MMIO_READ , size, gpa, &val); + } +return 1 ; +} + +static inline int tdx_mmio_write (struct kvm_vcpu *vcpu,gpa_t gpa,int size, +unsigned long val) +{ +if (kvm_iodevice_write (vcpu, &vcpu->arch.apic->dev, gpa, size, &val) && +kvm_io_bus_write (vcpu,KVM_MMIO_BUS , gpa, size, &val)) +return -EOPNOTSUPP ; + +trace_kvm_mmio (KVM_TRACE_MMIO_WRITE , size, gpa, &val); +return 0 ; +} + +static inline int tdx_mmio_read (struct kvm_vcpu *vcpu,gpa_t gpa,int size) +{ +unsigned long val; + +if (kvm_iodevice_read (vcpu, &vcpu->arch.apic->dev, gpa, size, &val) && +kvm_io_bus_read (vcpu,KVM_MMIO_BUS , gpa, size, &val)) +return -EOPNOTSUPP ; + +tdvmcall_set_return_val (vcpu, val); +trace_kvm_mmio (KVM_TRACE_MMIO_READ , size, gpa, &val); +return 0 ; +} + +static int tdx_emulate_mmio (struct kvm_vcpu *vcpu) +{ +struct kvm_memory_slot *slot; +int size, write, r; +unsigned long val; +gpa_t gpa; + +KVM_BUG_ON (vcpu->mmio_needed, vcpu->kvm); + + size =tdvmcall_a0_read (vcpu); + write =tdvmcall_a1_read (vcpu); + gpa =tdvmcall_a2_read (vcpu); + val = write ?tdvmcall_a3_read (vcpu) :0 ; + +if (size !=1 && size !=2 && size !=4 && size !=8 ) + goto error; +if (write !=0 && write !=1 ) + goto error; + +/* Strip the shared bit, allow MMIO with and without it set. */ + gpa = gpa & ~gfn_to_gpa (kvm_gfn_shared_mask (vcpu->kvm)); + +if (size >8u || ((gpa + size -1 ) ^ gpa) &PAGE_MASK ) + goto error; + + slot =kvm_vcpu_gfn_to_memslot (vcpu,gpa_to_gfn (gpa)); +if (slot && !(slot->flags &KVM_MEMSLOT_INVALID )) + goto error; + +if (!kvm_io_bus_write (vcpu,KVM_FAST_MMIO_BUS , gpa,0 ,NULL )) { +trace_kvm_fast_mmio (gpa); +return 1 ; + } + +if (write) + r =tdx_mmio_write (vcpu, gpa, size, val); +else + r =tdx_mmio_read (vcpu, gpa, size); +if (!r) { +/* Kernel completed device emulation. */ +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return 1 ; + } + +/* Request the device emulation to userspace device model. */ + vcpu->mmio_needed =1 ; + vcpu->mmio_is_write = write; + vcpu->arch.complete_userspace_io = tdx_complete_mmio; + + vcpu->run->mmio.phys_addr = gpa; + vcpu->run->mmio.len = size; + vcpu->run->mmio.is_write = write; + vcpu->run->exit_reason =KVM_EXIT_MMIO ; + +if (write) { +memcpy (vcpu->run->mmio.data, &val, size); + }else { + vcpu->mmio_fragments[0 ].gpa = gpa; + vcpu->mmio_fragments[0 ].len = size; +trace_kvm_mmio (KVM_TRACE_MMIO_READ_UNSATISFIED , size, gpa,NULL ); + } +return 0 ; + +error: +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1279,6 +1391,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_hlt (vcpu);case EXIT_REASON_IO_INSTRUCTION :return tdx_emulate_io (vcpu); +case EXIT_REASON_EPT_VIOLATION : +return tdx_emulate_mmio (vcpu);default :break ; }
arch/x86/kvm/x86.c +1 -0
@@ -13935,6 +13935,7 @@ EXPORT_SYMBOL_GPL(kvm_sev_es_string_io);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_entry);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_exit); +EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_mmio);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_fast_mmio);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_inj_virq);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_page_fault);
virt/kvm/kvm_main.c +2 -0
@@ -2689,6 +2689,7 @@ struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn_t gfnreturn NULL ; } +EXPORT_SYMBOL_GPL (kvm_vcpu_gfn_to_memslot);bool kvm_is_visible_gfn (struct kvm *kvm,gfn_t gfn) {
@@ -5925,6 +5926,7 @@ int kvm_io_bus_read(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, gpa_t addr, r =__kvm_io_bus_read (vcpu, bus, &range, val);return r <0 ? r :0 ; } +EXPORT_SYMBOL_GPL (kvm_io_bus_read);int kvm_io_bus_register_dev (struct kvm *kvm,enum kvm_bus bus_idx,gpa_t addr,int len,struct kvm_io_device *dev)
Signature · 1 line
2.25.1
PATCH v18 100/121 KVM: TDX: Implement callbacks for MSR operations for TDX
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Implements set_msr/get_msr/has_emulated_msr methods for TDX to handle hypercall from guest TD for paravirtualized rdmsr and wrmsr. The TDX module virtualizes MSRs. For some MSRs, it injects #VE to the guest TD upon RDMSR or WRMSR. The exact list of such MSRs are defined in the spec.
Upon #VE, the guest TD may execute hypercalls, TDG.VP.VMCALL<INSTRUCTION.RDMSR> and TDG.VP.VMCALL<INSTRUCTION.WRMSR>, which are defined in GHCI (Guest-Host Communication Interface) so that the host VMM (e.g. KVM) can virtualize the MSRs.
There are three classes of MSRs virtualization.
- non-configurable: TDX module directly virtualizes it. VMM can't configure. the value set by KVM_SET_MSR_INDEX_LIST is ignored.
- configurable: TDX module directly virtualizes it. VMM can configure at the VM creation time. The value set by KVM_SET_MSR_INDEX_LIST is used.
- #VE case
Guest TD would issue TDG.VP.VMCALL<INSTRUCTION.{WRMSR,RDMSR> and VMM handles the MSR hypercall. The value set by KVM_SET_MSR_INDEX_LIST is used.
--- arch/x86/kvm/vmx/main.c | 44 +++++++++++++++++++++--- arch/x86/kvm/vmx/tdx.c | 70 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 6 ++++ arch/x86/kvm/x86.c | 1 - arch/x86/kvm/x86.h | 2 ++ 5 files changed, 118 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/main.c +40 -4
@@ -262,6 +262,42 @@ static void vt_handle_exit_irqoff(struct kvm_vcpu *vcpu)vmx_handle_exit_irqoff (vcpu); } +static int vt_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr_info) +{ +if (unlikely (is_td_vcpu (vcpu))) +return tdx_set_msr (vcpu, msr_info); + +return vmx_set_msr (vcpu, msr_info); +} + +/* +* The kvm parameter can be NULL (module initialization, or invocation before +* VM creation). Be sure to check the kvm parameter before using it. +*/ +static bool vt_has_emulated_msr (struct kvm *kvm,u32 index) +{ +if (kvm &&is_td (kvm)) +return tdx_has_emulated_msr (index, true); + +return vmx_has_emulated_msr (kvm, index); +} + +static int vt_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr_info) +{ +if (unlikely (is_td_vcpu (vcpu))) +return tdx_get_msr (vcpu, msr_info); + +return vmx_get_msr (vcpu, msr_info); +} + +static void vt_msr_filter_changed (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_msr_filter_changed (vcpu); +} +static void vt_apicv_pre_state_restore (struct kvm_vcpu *vcpu) {struct pi_desc *pi =vcpu_to_pi_desc (vcpu);
@@ -523,7 +559,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .hardware_enable = vt_hardware_enable, .hardware_disable = vt_hardware_disable, - .has_emulated_msr = vmx_has_emulated_msr, + .has_emulated_msr = vt_has_emulated_msr, .is_vm_type_supported = vt_is_vm_type_supported, .max_vcpus = vt_max_vcpus,
@@ -545,8 +581,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .update_exception_bitmap = vmx_update_exception_bitmap, .get_msr_feature = vmx_get_msr_feature, - .get_msr = vmx_get_msr, - .set_msr = vmx_set_msr, + .get_msr = vt_get_msr, + .set_msr = vt_set_msr, .get_segment_base = vmx_get_segment_base, .get_segment = vmx_get_segment, .set_segment = vmx_set_segment,
@@ -656,7 +692,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .apic_init_signal_blocked = vmx_apic_init_signal_blocked, .migrate_timers = vmx_migrate_timers, - .msr_filter_changed = vmx_msr_filter_changed, + .msr_filter_changed = vt_msr_filter_changed, .complete_emulated_msr = kvm_complete_insn_gp, .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector,
arch/x86/kvm/vmx/tdx.c +70 -0
76 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +6 -0
@@ -165,6 +165,9 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,void tdx_inject_nmi (struct kvm_vcpu *vcpu);void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason,u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code); +bool tdx_has_emulated_msr (u32 index,bool write); +int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr); +int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -207,6 +210,9 @@ static inline void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mostatic inline void tdx_inject_nmi (struct kvm_vcpu *vcpu) {}static inline void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason,u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code) {} +static inline bool tdx_has_emulated_msr (u32 index,bool write) {return false; } +static inline int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; } +static inline int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
arch/x86/kvm/x86.c +0 -1
@@ -90,7 +90,6 @@#include "trace.h" #define MAX_IO_MSRS 256 -#define KVM_MAX_MCE_BANKS 32struct kvm_caps kvm_caps__read_mostly = { .supported_mce_cap =MCG_CTL_P |MCG_SER_P ,
arch/x86/kvm/x86.h +2 -0
@@ -9,6 +9,8 @@#include "kvm_cache_regs.h" #include "kvm_emulate.h" +#define KVM_MAX_MCE_BANKS 32 +bool __kvm_is_vm_type_supported (unsigned long type);struct kvm_caps {
Signature · 1 line
2.25.1
PATCH v18 101/121 KVM: TDX: Handle TDX PV rdmsr/wrmsr hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV rdmsr/wrmsr hypercall to the KVM backend function.
--- arch/x86/kvm/vmx/tdx.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
arch/x86/kvm/vmx/tdx.c +39 -0
@@ -1379,6 +1379,41 @@ static int tdx_emulate_mmio(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_emulate_rdmsr (struct kvm_vcpu *vcpu) +{ +u32 index =tdvmcall_a0_read (vcpu); +u64 data; + +if (!kvm_msr_allowed (vcpu, index,KVM_MSR_FILTER_READ ) || +kvm_get_msr (vcpu, index, &data)) { +trace_kvm_msr_read_ex (index); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } +trace_kvm_msr_read (index, data); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +tdvmcall_set_return_val (vcpu, data); +return 1 ; +} + +static int tdx_emulate_wrmsr (struct kvm_vcpu *vcpu) +{ +u32 index =tdvmcall_a0_read (vcpu); +u64 data =tdvmcall_a1_read (vcpu); + +if (!kvm_msr_allowed (vcpu, index,KVM_MSR_FILTER_WRITE ) || +kvm_set_msr (vcpu, index, data)) { +trace_kvm_msr_write_ex (index, data); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } + +trace_kvm_msr_write (index, data); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1393,6 +1428,10 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_io (vcpu);case EXIT_REASON_EPT_VIOLATION :return tdx_emulate_mmio (vcpu); +case EXIT_REASON_MSR_READ : +return tdx_emulate_rdmsr (vcpu); +case EXIT_REASON_MSR_WRITE : +return tdx_emulate_wrmsr (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v18 102/121 KVM: TDX: Handle MSR MTRRCap and MTRRDefType access
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Handle MTRRCap RO MSR to return all features are unsupported and handle MTRRDefType MSR to accept only E=1,FE=0,type=writeback. enable MTRR, disable Fixed range MTRRs, default memory type=writeback
TDX virtualizes that cpuid to report MTRR to guest TD and TDX enforces guest CR0.CD=0. If guest tries to set CR0.CD=1, it results in #GP. While updating MTRR requires to set CR0.CD=1 (and other cache flushing operations). It means guest TD can't update MTRR. Virtualize MTRR as all features disabled and default memory type as writeback.
--- arch/x86/kvm/vmx/tdx.c | 99 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 82 insertions(+), 17 deletions(-)
arch/x86/kvm/vmx/tdx.c +82 -17
@@ -656,18 +656,7 @@ u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)if (!kvm_arch_has_noncoherent_dma (vcpu->kvm))return (MTRR_TYPE_WRBACK <<VMX_EPT_MT_EPTE_SHIFT ) |VMX_EPT_IPAT_BIT ; -/* -* TDX enforces CR0.CD = 0 and KVM MTRR emulation enforces writeback. -* TODO: implement MTRR MSR emulation so that -* MTRRCap: SMRR=0: SMRR interface unsupported -* WC=0: write combining unsupported -* FIX=0: Fixed range registers unsupported -* VCNT=0: number of variable range regitsers = 0 -* MTRRDefType: E=1, FE=0, type=writeback only. Don't allow other value. -* E=1: enable MTRR -* FE=0: disable fixed range MTRRs -* type: default memory type=writeback -*/ +/* TDX enforces CR0.CD = 0 and KVM MTRR emulation enforces writeback. */ return MTRR_TYPE_WRBACK <<VMX_EPT_MT_EPTE_SHIFT ; }
@@ -1978,7 +1967,9 @@ bool tdx_has_emulated_msr(u32 index, bool write)case MSR_IA32_UCODE_REV :case MSR_IA32_ARCH_CAPABILITIES :case MSR_IA32_POWER_CTL : +case MSR_MTRRcap:case MSR_IA32_CR_PAT : +case MSR_MTRRdefType:case MSR_IA32_TSC_DEADLINE :case MSR_IA32_MISC_ENABLE :case MSR_PLATFORM_INFO :
@@ -2020,16 +2011,47 @@ bool tdx_has_emulated_msr(u32 index, bool write)int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) { -if (tdx_has_emulated_msr (msr->index, false)) -return kvm_get_msr_common (vcpu, msr); -return 1 ; +switch (msr->index) { +case MSR_MTRRcap: +/* +* Override kvm_mtrr_get_msr() which hardcodes the value. +* Report SMRR = 0, WC = 0, FIX = 0 VCNT = 0 to disable MTRR +* effectively. +*/ + msr->data =0 ; +return 0 ; +default : +if (tdx_has_emulated_msr (msr->index, false)) +return kvm_get_msr_common (vcpu, msr); +return 1 ; + } }int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) { -if (tdx_has_emulated_msr (msr->index, true)) +switch (msr->index) { +case MSR_MTRRdefType: +/* +* Allow writeback only for all memory. +* Because it's reported that fixed range MTRR isn't supported +* and VCNT=0, enforce MTRRDefType.FE = 0 and don't care +* variable range MTRRs. Only default memory type matters. +* +* bit 11 E: MTRR enable/disable +* bit 12 FE: Fixed-range MTRRs enable/disable +* (E, FE) = (1, 1): enable MTRR and Fixed range MTRR +* (E, FE) = (1, 0): enable MTRR, disable Fixed range MTRR +* (E, FE) = (0, *): disable all MTRRs. all physical memory +* is UC +*/ +if (msr->data != ((1 <<11 ) |MTRR_TYPE_WRBACK )) +return 1 ;return kvm_set_msr_common (vcpu, msr); -return 1 ; +default : +if (tdx_has_emulated_msr (msr->index, true)) +return kvm_set_msr_common (vcpu, msr); +return 1 ; + } }static int tdx_get_capabilities (struct kvm_tdx_cmd *cmd)
@@ -2778,6 +2800,45 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u64 vcpu_rcx)return ret; } +static int tdx_vcpu_init_mtrr (struct kvm_vcpu *vcpu) +{ +struct msr_data msr; +int ret; +int i; + +/* +* To avoid confusion with reporting VNCT = 0, explicitly disable +* vaiale-range reisters. +*/ +for (i =0 ; i <KVM_NR_VAR_MTRR ; i++) { +/* phymask */ + msr = (struct msr_data ) { + .host_initiated = true, + .index =0x200 +2 * i +1 , + .data =0 ,/* valid = 0 to disable. */ + }; + ret =kvm_set_msr_common (vcpu, &msr); +if (ret) +return -EINVAL ; + } + +/* Set MTRR to use writeback on reset. */ + msr = (struct msr_data ) { + .host_initiated = true, + .index = MSR_MTRRdefType, +/* +* Set E(enable MTRR)=1, FE(enable fixed range MTRR)=0, default +* type=writeback on reset to avoid UC. Note E=0 means all +* memory is UC. +*/ + .data = (1 <<11 ) |MTRR_TYPE_WRBACK , + }; + ret =kvm_set_msr_common (vcpu, &msr); +if (ret) +return -EINVAL ; +return 0 ; +} +int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {struct msr_data apic_base_msr;
@@ -2815,6 +2876,10 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)if (kvm_set_apic_base (vcpu, &apic_base_msr))return -EINVAL ; + ret =tdx_vcpu_init_mtrr (vcpu); +if (ret) +return ret; + ret =tdx_td_vcpu_init (vcpu, (u64 )cmd.data);if (ret)return ret;
Signature · 1 line
2.25.1
PATCH v18 103/121 KVM: TDX: Handle MSR IA32_FEAT_CTL MSR and IA32_MCG_EXT_CTL
In reply to [email protected]From: Isaku Yamahata <[email protected]>
MCE and MCA is advertised via cpuid based on the TDX module spec. Guest kernel can access IA32_FEAT_CTL for checking if LMCE is enabled by platform and IA32_MCG_EXT_CTL to enable LMCE. Make TDX KVM handle them. Otherwise guest MSR access to them with TDG.VP.VMCALL<MSR> on VE results in GP in guest.
Because LMCE is disabled with qemu by default, "-cpu lmce=on" to qemu command line is needed to reproduce it.
--- arch/x86/kvm/vmx/tdx.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
arch/x86/kvm/vmx/tdx.c +20 -0
@@ -1998,6 +1998,7 @@ bool tdx_has_emulated_msr(u32 index, bool write)default :return true; } +case MSR_IA32_FEAT_CTL :case MSR_IA32_APICBASE :case MSR_EFER :return !write;
@@ -2012,6 +2013,20 @@ bool tdx_has_emulated_msr(u32 index, bool write)int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {switch (msr->index) { +case MSR_IA32_FEAT_CTL : +/* +* MCE and MCA are advertised via cpuid. guest kernel could +* check if LMCE is enabled or not. +*/ + msr->data =FEAT_CTL_LOCKED ; +if (vcpu->arch.mcg_cap &MCG_LMCE_P ) + msr->data |=FEAT_CTL_LMCE_ENABLED ; +return 0 ; +case MSR_IA32_MCG_EXT_CTL : +if (!msr->host_initiated && !(vcpu->arch.mcg_cap &MCG_LMCE_P )) +return 1 ; + msr->data = vcpu->arch.mcg_ext_ctl; +return 0 ;case MSR_MTRRcap:/* * Override kvm_mtrr_get_msr() which hardcodes the value.
@@ -2030,6 +2045,11 @@ int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {switch (msr->index) { +case MSR_IA32_MCG_EXT_CTL : +if (!msr->host_initiated && !(vcpu->arch.mcg_cap &MCG_LMCE_P )) +return 1 ; + vcpu->arch.mcg_ext_ctl = msr->data; +return 0 ;case MSR_MTRRdefType:/* * Allow writeback only for all memory.
Signature · 1 line
2.25.1
PATCH v18 104/121 KVM: TDX: Handle TDG.VP.VMCALL<GetTdVmCallInfo> hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Implement TDG.VP.VMCALL<GetTdVmCallInfo> hypercall. If the input value is zero, return success code and zero in output registers.
TDG.VP.VMCALL<GetTdVmCallInfo> hypercall is a subleaf of TDG.VP.VMCALL to enumerate which TDG.VP.VMCALL sub leaves are supported. This hypercall is for future enhancement of the Guest-Host-Communication Interface (GHCI) specification. The GHCI version of 344426-001US defines it to require input R12 to be zero and to return zero in output registers, R11, R12, R13, and R14 so that guest TD enumerates no enhancement.
--- arch/x86/kvm/vmx/tdx.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
arch/x86/kvm/vmx/tdx.c +16 -0
@@ -1403,6 +1403,20 @@ static int tdx_emulate_wrmsr(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_get_td_vm_call_info (struct kvm_vcpu *vcpu) +{ +if (tdvmcall_a0_read (vcpu)) +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +else { +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +kvm_r11_write (vcpu,0 ); +tdvmcall_a0_write (vcpu,0 ); +tdvmcall_a1_write (vcpu,0 ); +tdvmcall_a2_write (vcpu,0 ); + } +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1421,6 +1435,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_rdmsr (vcpu);case EXIT_REASON_MSR_WRITE :return tdx_emulate_wrmsr (vcpu); +case TDG_VP_VMCALL_GET_TD_VM_CALL_INFO : +return tdx_get_td_vm_call_info (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v18 105/121 KVM: TDX: Silently discard SMI request
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX doesn't support system-management mode (SMM) and system-management interrupt (SMI) in guest TDs. Because guest state (vcpu state, memory state) is protected, it must go through the TDX module APIs to change guest state, injecting SMI and changing vcpu mode into SMM. The TDX module doesn't provide a way for VMM to inject SMI into guest TD and a way for VMM to switch guest vcpu mode into SMM.
We have two options in KVM when handling SMM or SMI in the guest TD or the device model (e.g. QEMU): 1) silently ignore the request or 2) return a meaningful error.
For simplicity, we implemented the option 1).
--- arch/x86/kvm/smm.h | 7 +++++- arch/x86/kvm/vmx/main.c | 45 ++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 29 ++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 12 ++++++++++ 4 files changed, 88 insertions(+), 5 deletions(-)
arch/x86/kvm/smm.h +6 -1
@@ -142,7 +142,12 @@ union kvm_smram {static inline int kvm_inject_smi (struct kvm_vcpu *vcpu) { -kvm_make_request (KVM_REQ_SMI , vcpu); +/* +* If SMM isn't supported (e.g. TDX), silently discard SMI request. +* Assume that SMM supported = MSR_IA32_SMBASE supported. +*/ +if (static_call (kvm_x86_has_emulated_msr)(vcpu->kvm,MSR_IA32_SMBASE )) +kvm_make_request (KVM_REQ_SMI , vcpu);return 0 ; }
arch/x86/kvm/vmx/main.c +41 -4
57 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.c +29 -0
@@ -2090,6 +2090,35 @@ int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) } } +#ifdef CONFIG_KVM_SMM +int tdx_smi_allowed (struct kvm_vcpu *vcpu,bool for_injection) +{ +/* SMI isn't supported for TDX. */ +WARN_ON_ONCE (1 ); +return false; +} + +int tdx_enter_smm (struct kvm_vcpu *vcpu,union kvm_smram *smram) +{ +/* smi_allowed() is always false for TDX as above. */ +WARN_ON_ONCE (1 ); +return 0 ; +} + +int tdx_leave_smm (struct kvm_vcpu *vcpu,const union kvm_smram *smram) +{ +WARN_ON_ONCE (1 ); +return 0 ; +} + +void tdx_enable_smi_window (struct kvm_vcpu *vcpu) +{ +/* SMI isn't supported for TDX. Silently discard SMI request. */ +WARN_ON_ONCE (1 ); + vcpu->arch.smi_pending = false; +} +#endif +static int tdx_get_capabilities (struct kvm_tdx_cmd *cmd) {struct kvm_tdx_capabilities __user *user_caps;
arch/x86/kvm/vmx/x86_ops.h +12 -0
@@ -222,4 +222,16 @@ static inline int tdx_sept_flush_remote_tlbs(struct kvm *kvm) { return 0; }static inline void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level) {}#endif +#if defined(CONFIG_INTEL_TDX_HOST ) && defined(CONFIG_KVM_SMM ) +int tdx_smi_allowed (struct kvm_vcpu *vcpu,bool for_injection); +int tdx_enter_smm (struct kvm_vcpu *vcpu,union kvm_smram *smram); +int tdx_leave_smm (struct kvm_vcpu *vcpu,const union kvm_smram *smram); +void tdx_enable_smi_window (struct kvm_vcpu *vcpu); +#else +static inline int tdx_smi_allowed (struct kvm_vcpu *vcpu,bool for_injection) {return false; } +static inline int tdx_enter_smm (struct kvm_vcpu *vcpu,union kvm_smram *smram) {return 0 ; } +static inline int tdx_leave_smm (struct kvm_vcpu *vcpu,const union kvm_smram *smram) {return 0 ; } +static inline void tdx_enable_smi_window (struct kvm_vcpu *vcpu) {} +#endif +#endif /* __KVM_X86_VMX_X86_OPS_H */
Signature · 1 line
2.25.1
PATCH v18 106/121 KVM: TDX: Silently ignore INIT/SIPI
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The TDX module API doesn't provide API for VMM to inject INIT IPI and SIPI. Instead it defines the different protocols to boot application processors. Ignore INIT and SIPI events for the TDX guest.
There are two options. 1) (silently) ignore INIT/SIPI request or 2) return error to guest TDs somehow. Given that TDX guest is paravirtualized to boot AP, the option 1 is chosen for simplicity.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/lapic.c | 19 +++++++++++------- arch/x86/kvm/svm/svm.c | 1 + arch/x86/kvm/vmx/main.c | 32 ++++++++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 4 ++-- 6 files changed, 48 insertions(+), 11 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -149,6 +149,7 @@ KVM_X86_OP_OPTIONAL(migrate_timers)KVM_X86_OP (msr_filter_changed) KVM_X86_OP(complete_emulated_msr) KVM_X86_OP(vcpu_deliver_sipi_vector) +KVM_X86_OP(vcpu_deliver_init) KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);KVM_X86_OP_OPTIONAL (get_untagged_addr)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1853,6 +1853,7 @@ struct kvm_x86_ops {int (*complete_emulated_msr)(struct kvm_vcpu *vcpu,int err);void (*vcpu_deliver_sipi_vector)(struct kvm_vcpu *vcpu,u8 vector); +void (*vcpu_deliver_init)(struct kvm_vcpu *vcpu);/* * Returns vCPU specific APICv inhibit reasons
@@ -2101,6 +2102,7 @@ void kvm_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg);void kvm_set_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var,int seg);int kvm_load_segment_descriptor (struct kvm_vcpu *vcpu,u16 selector,int seg);void kvm_vcpu_deliver_sipi_vector (struct kvm_vcpu *vcpu,u8 vector); +void kvm_vcpu_deliver_init (struct kvm_vcpu *vcpu);int kvm_task_switch (struct kvm_vcpu *vcpu,u16 tss_selector,int idt_index,int reason,bool has_error_code,u32 error_code);
arch/x86/kvm/lapic.c +12 -7
@@ -3268,6 +3268,16 @@ int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len)return 0 ; } +void kvm_vcpu_deliver_init (struct kvm_vcpu *vcpu) +{ +kvm_vcpu_reset (vcpu, true); +if (kvm_vcpu_is_bsp (vcpu)) + vcpu->arch.mp_state =KVM_MP_STATE_RUNNABLE ; +else + vcpu->arch.mp_state =KVM_MP_STATE_INIT_RECEIVED ; +} +EXPORT_SYMBOL_GPL (kvm_vcpu_deliver_init); +int kvm_apic_accept_events (struct kvm_vcpu *vcpu) {struct kvm_lapic *apic = vcpu->arch.apic;
@@ -3299,13 +3309,8 @@ int kvm_apic_accept_events(struct kvm_vcpu *vcpu)return 0 ; } -if (test_and_clear_bit (KVM_APIC_INIT , &apic->pending_events)) { -kvm_vcpu_reset (vcpu, true); -if (kvm_vcpu_is_bsp (apic->vcpu)) - vcpu->arch.mp_state =KVM_MP_STATE_RUNNABLE ; -else - vcpu->arch.mp_state =KVM_MP_STATE_INIT_RECEIVED ; - } +if (test_and_clear_bit (KVM_APIC_INIT , &apic->pending_events)) +static_call (kvm_x86_vcpu_deliver_init)(vcpu);if (test_and_clear_bit (KVM_APIC_SIPI , &apic->pending_events)) {if (vcpu->arch.mp_state ==KVM_MP_STATE_INIT_RECEIVED ) {/* evaluate pending_events before reading the vector */
arch/x86/kvm/svm/svm.c +1 -0
@@ -5037,6 +5037,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .complete_emulated_msr = svm_complete_emulated_msr, .vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector, + .vcpu_deliver_init = kvm_vcpu_deliver_init, .vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons, };
arch/x86/kvm/vmx/main.c +30 -2
@@ -335,6 +335,14 @@ static void vt_enable_smi_window(struct kvm_vcpu *vcpu) }#endif +static bool vt_apic_init_signal_blocked (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return true; + +return vmx_apic_init_signal_blocked (vcpu); +} +static void vt_apicv_pre_state_restore (struct kvm_vcpu *vcpu) {struct pi_desc *pi =vcpu_to_pi_desc (vcpu);
@@ -363,6 +371,25 @@ static void vt_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,vmx_deliver_interrupt (apic, delivery_mode, trig_mode, vector); } +static void vt_vcpu_deliver_sipi_vector (struct kvm_vcpu *vcpu,u8 vector) +{ +if (is_td_vcpu (vcpu)) +return ; + +kvm_vcpu_deliver_sipi_vector (vcpu, vector); +} + +static void vt_vcpu_deliver_init (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +/* TDX doesn't support INIT. Ignore INIT event */ + vcpu->arch.mp_state =KVM_MP_STATE_RUNNABLE ; +return ; + } + +kvm_vcpu_deliver_init (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) {
@@ -726,13 +753,14 @@ struct kvm_x86_ops vt_x86_ops __initdata = {#endif .check_emulate_instruction = vmx_check_emulate_instruction, - .apic_init_signal_blocked = vmx_apic_init_signal_blocked, + .apic_init_signal_blocked = vt_apic_init_signal_blocked, .migrate_timers = vmx_migrate_timers, .msr_filter_changed = vt_msr_filter_changed, .complete_emulated_msr = kvm_complete_insn_gp, - .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector, + .vcpu_deliver_sipi_vector = vt_vcpu_deliver_sipi_vector, + .vcpu_deliver_init = vt_vcpu_deliver_init, .get_untagged_addr = vmx_get_untagged_addr,
arch/x86/kvm/vmx/tdx.c +2 -2
@@ -819,8 +819,8 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu)void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) { -/* Ignore INIT silently because TDX doesn't support INIT event. */ -if (init_event) +/* vcpu_deliver_init method silently discards INIT event. */ +if (KVM_BUG_ON (init_event, vcpu->kvm))return ;if (KVM_BUG_ON (is_td_vcpu_created (to_tdx (vcpu)), vcpu->kvm))return ;
Signature · 1 line
2.25.1
PATCH v18 107/121 KVM: TDX: Add methods to ignore accesses to CPU state
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX protects TDX guest state from VMM. Implement access methods for TDX guest state to ignore them or return zero. Because those methods can be called by kvm ioctls to set/get cpu registers, they don't have KVM_BUG_ON except one method.
--- arch/x86/kvm/vmx/main.c | 289 +++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 48 +++++- arch/x86/kvm/vmx/x86_ops.h | 13 ++ 3 files changed, 321 insertions(+), 29 deletions(-)
arch/x86/kvm/vmx/main.c +262 -27
@@ -390,6 +390,200 @@ static void vt_vcpu_deliver_init(struct kvm_vcpu *vcpu)kvm_vcpu_deliver_init (vcpu); } +static void vt_vcpu_after_set_cpuid (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_vcpu_after_set_cpuid (vcpu); +} + +static void vt_update_exception_bitmap (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_update_exception_bitmap (vcpu); +} + +static u64 vt_get_segment_base (struct kvm_vcpu *vcpu,int seg) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_segment_base (vcpu, seg); + +return vmx_get_segment_base (vcpu, seg); +} + +static void vt_get_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var, +int seg) +{ +if (is_td_vcpu (vcpu)) { +tdx_get_segment (vcpu, var, seg); +return ; + } + +vmx_get_segment (vcpu, var, seg); +} + +static void vt_set_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var, +int seg) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_segment (vcpu, var, seg); +} + +static int vt_get_cpl (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_cpl (vcpu); + +return vmx_get_cpl (vcpu); +} + +static void vt_get_cs_db_l_bits (struct kvm_vcpu *vcpu,int *db,int *l) +{ +if (is_td_vcpu (vcpu)) { + *db =0 ; + *l =0 ; +return ; + } + +vmx_get_cs_db_l_bits (vcpu, db, l); +} + +static bool vt_is_valid_cr0 (struct kvm_vcpu *vcpu,unsigned long cr0) +{ +if (is_td_vcpu (vcpu)) +return true; + +return vmx_is_valid_cr0 (vcpu, cr0); +} + +static void vt_set_cr0 (struct kvm_vcpu *vcpu,unsigned long cr0) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_cr0 (vcpu, cr0); +} + +static bool vt_is_valid_cr4 (struct kvm_vcpu *vcpu,unsigned long cr4) +{ +if (is_td_vcpu (vcpu)) +return true; + +return vmx_is_valid_cr4 (vcpu, cr4); +} + +static void vt_set_cr4 (struct kvm_vcpu *vcpu,unsigned long cr4) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_cr4 (vcpu, cr4); +} + +static int vt_set_efer (struct kvm_vcpu *vcpu,u64 efer) +{ +if (is_td_vcpu (vcpu)) +return 0 ; + +return vmx_set_efer (vcpu, efer); +} + +static void vt_get_idt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (is_td_vcpu (vcpu)) { +memset (dt,0 ,sizeof (*dt)); +return ; + } + +vmx_get_idt (vcpu, dt); +} + +static void vt_set_idt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_idt (vcpu, dt); +} + +static void vt_get_gdt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (is_td_vcpu (vcpu)) { +memset (dt,0 ,sizeof (*dt)); +return ; + } + +vmx_get_gdt (vcpu, dt); +} + +static void vt_set_gdt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_gdt (vcpu, dt); +} + +static void vt_set_dr7 (struct kvm_vcpu *vcpu,unsigned long val) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_dr7 (vcpu, val); +} + +static void vt_sync_dirty_debug_regs (struct kvm_vcpu *vcpu) +{ +/* +* MOV-DR exiting is always cleared for TD guest, even in debug mode. +* Thus KVM_DEBUGREG_WONT_EXIT can never be set and it should never +* reach here for TD vcpu. +*/ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_sync_dirty_debug_regs (vcpu); +} + +static void vt_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg) +{ +if (is_td_vcpu (vcpu)) { +tdx_cache_reg (vcpu, reg); +return ; + } + +vmx_cache_reg (vcpu, reg); +} + +static unsigned long vt_get_rflags (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_rflags (vcpu); + +return vmx_get_rflags (vcpu); +} + +static void vt_set_rflags (struct kvm_vcpu *vcpu,unsigned long rflags) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_rflags (vcpu, rflags); +} + +static bool vt_get_if_flag (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return false; + +return vmx_get_if_flag (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) {
@@ -534,6 +728,14 @@ static void vt_inject_irq(struct kvm_vcpu *vcpu, bool reinjected)vmx_inject_irq (vcpu, reinjected); } +static void vt_inject_exception (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_inject_exception (vcpu); +} +static void vt_cancel_injection (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -580,6 +782,39 @@ static void vt_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason,vmx_get_exit_info (vcpu, reason, info1, info2, intr_info, error_code); } + +static void vt_update_cr8_intercept (struct kvm_vcpu *vcpu,int tpr,int irr) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_update_cr8_intercept (vcpu, tpr, irr); +} + +static void vt_load_eoi_exitmap (struct kvm_vcpu *vcpu,u64 *eoi_exit_bitmap) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_load_eoi_exitmap (vcpu, eoi_exit_bitmap); +} + +static int vt_set_tss_addr (struct kvm *kvm,unsigned int addr) +{ +if (is_td (kvm)) +return 0 ; + +return vmx_set_tss_addr (kvm, addr); +} + +static int vt_set_identity_map_addr (struct kvm *kvm,u64 ident_addr) +{ +if (is_td (kvm)) +return 0 ; + +return vmx_set_identity_map_addr (kvm, ident_addr); +} +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {if (is_td_vcpu (vcpu))
@@ -643,30 +878,30 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_load = vt_vcpu_load, .vcpu_put = vt_vcpu_put, - .update_exception_bitmap = vmx_update_exception_bitmap, + .update_exception_bitmap = vt_update_exception_bitmap, .get_msr_feature = vmx_get_msr_feature, .get_msr = vt_get_msr, .set_msr = vt_set_msr, - .get_segment_base = vmx_get_segment_base, - .get_segment = vmx_get_segment, - .set_segment = vmx_set_segment, - .get_cpl = vmx_get_cpl, - .get_cs_db_l_bits = vmx_get_cs_db_l_bits, - .is_valid_cr0 = vmx_is_valid_cr0, - .set_cr0 = vmx_set_cr0, - .is_valid_cr4 = vmx_is_valid_cr4, - .set_cr4 = vmx_set_cr4, - .set_efer = vmx_set_efer, - .get_idt = vmx_get_idt, - .set_idt = vmx_set_idt, - .get_gdt = vmx_get_gdt, - .set_gdt = vmx_set_gdt, - .set_dr7 = vmx_set_dr7, - .sync_dirty_debug_regs = vmx_sync_dirty_debug_regs, - .cache_reg = vmx_cache_reg, - .get_rflags = vmx_get_rflags, - .set_rflags = vmx_set_rflags, - .get_if_flag = vmx_get_if_flag, + .get_segment_base = vt_get_segment_base, + .get_segment = vt_get_segment, + .set_segment = vt_set_segment, + .get_cpl = vt_get_cpl, + .get_cs_db_l_bits = vt_get_cs_db_l_bits, + .is_valid_cr0 = vt_is_valid_cr0, + .set_cr0 = vt_set_cr0, + .is_valid_cr4 = vt_is_valid_cr4, + .set_cr4 = vt_set_cr4, + .set_efer = vt_set_efer, + .get_idt = vt_get_idt, + .set_idt = vt_set_idt, + .get_gdt = vt_get_gdt, + .set_gdt = vt_set_gdt, + .set_dr7 = vt_set_dr7, + .sync_dirty_debug_regs = vt_sync_dirty_debug_regs, + .cache_reg = vt_cache_reg, + .get_rflags = vt_get_rflags, + .set_rflags = vt_set_rflags, + .get_if_flag = vt_get_if_flag, .flush_tlb_all = vt_flush_tlb_all, .flush_tlb_current = vt_flush_tlb_current,
@@ -683,7 +918,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .patch_hypercall = vmx_patch_hypercall, .inject_irq = vt_inject_irq, .inject_nmi = vt_inject_nmi, - .inject_exception = vmx_inject_exception, + .inject_exception = vt_inject_exception, .cancel_injection = vt_cancel_injection, .interrupt_allowed = vt_interrupt_allowed, .nmi_allowed = vt_nmi_allowed,
@@ -691,11 +926,11 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_nmi_mask = vt_set_nmi_mask, .enable_nmi_window = vt_enable_nmi_window, .enable_irq_window = vt_enable_irq_window, - .update_cr8_intercept = vmx_update_cr8_intercept, + .update_cr8_intercept = vt_update_cr8_intercept, .set_virtual_apic_mode = vmx_set_virtual_apic_mode, .set_apic_access_page_addr = vmx_set_apic_access_page_addr, .refresh_apicv_exec_ctrl = vmx_refresh_apicv_exec_ctrl, - .load_eoi_exitmap = vmx_load_eoi_exitmap, + .load_eoi_exitmap = vt_load_eoi_exitmap, .apicv_pre_state_restore = vt_apicv_pre_state_restore, .required_apicv_inhibits = VMX_REQUIRED_APICV_INHIBITS, .hwapic_irr_update = vmx_hwapic_irr_update,
@@ -706,13 +941,13 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .dy_apicv_has_pending_interrupt = pi_has_pending_interrupt, .protected_apic_has_interrupt = vt_protected_apic_has_interrupt, - .set_tss_addr = vmx_set_tss_addr, - .set_identity_map_addr = vmx_set_identity_map_addr, + .set_tss_addr = vt_set_tss_addr, + .set_identity_map_addr = vt_set_identity_map_addr, .get_mt_mask = vt_get_mt_mask, .get_exit_info = vt_get_exit_info, - .vcpu_after_set_cpuid = vmx_vcpu_after_set_cpuid, + .vcpu_after_set_cpuid = vt_vcpu_after_set_cpuid, .has_wbinvd_exit = cpu_has_vmx_wbinvd_exit,
arch/x86/kvm/vmx/tdx.c +46 -2
60 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +13 -0
@@ -169,6 +169,12 @@ bool tdx_has_emulated_msr(u32 index, bool write);int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr);int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr); +int tdx_get_cpl (struct kvm_vcpu *vcpu); +void tdx_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg); +unsigned long tdx_get_rflags (struct kvm_vcpu *vcpu); +u64 tdx_get_segment_base (struct kvm_vcpu *vcpu,int seg); +void tdx_get_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var,int seg); +int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);void tdx_flush_tlb (struct kvm_vcpu *vcpu);
@@ -214,6 +220,13 @@ static inline bool tdx_has_emulated_msr(u32 index, bool write) { return false; }static inline int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; }static inline int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; } +static inline int tdx_get_cpl (struct kvm_vcpu *vcpu) {return 0 ; } +static inline void tdx_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg) {} +static inline unsigned long tdx_get_rflags (struct kvm_vcpu *vcpu) {return 0 ; } +static inline u64 tdx_get_segment_base (struct kvm_vcpu *vcpu,int seg) {return 0 ; } +static inline void tdx_get_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var, +int seg) {} +static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }static inline void tdx_flush_tlb (struct kvm_vcpu *vcpu) {}
Signature · 1 line
2.25.1
PATCH v18 108/121 KVM: TDX: Add methods to ignore guest instruction emulation
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because TDX protects TDX guest state from VMM, instructions in guest memory cannot be emulated. Implement methods to ignore guest instruction emulator.
--- arch/x86/kvm/vmx/main.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +26 -2
@@ -335,6 +335,30 @@ static void vt_enable_smi_window(struct kvm_vcpu *vcpu) }#endif +static int vt_check_emulate_instruction (struct kvm_vcpu *vcpu,int emul_type, +void *insn,int insn_len) +{ +if (is_td_vcpu (vcpu)) +return X86EMUL_RETRY_INSTR ; + +return vmx_check_emulate_instruction (vcpu, emul_type, insn, insn_len); +} + +static int vt_check_intercept (struct kvm_vcpu *vcpu, +struct x86_instruction_info *info, +enum x86_intercept_stage stage, +struct x86_exception *exception) +{ +/* +* This call back is triggered by the x86 instruction emulator. TDX +* doesn't allow guest memory inspection. +*/ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return X86EMUL_UNHANDLEABLE ; + +return vmx_check_intercept (vcpu, info, stage, exception); +} +static bool vt_apic_init_signal_blocked (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -958,7 +982,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .load_mmu_pgd = vt_load_mmu_pgd, - .check_intercept = vmx_check_intercept, + .check_intercept = vt_check_intercept, .handle_exit_irqoff = vt_handle_exit_irqoff, .request_immediate_exit = vt_request_immediate_exit,
@@ -987,7 +1011,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .enable_smi_window = vt_enable_smi_window, #endif - .check_emulate_instruction = vmx_check_emulate_instruction, + .check_emulate_instruction = vt_check_emulate_instruction, .apic_init_signal_blocked = vt_apic_init_signal_blocked, .migrate_timers = vmx_migrate_timers,
Signature · 1 line
2.25.1
PATCH v18 109/121 KVM: TDX: Add a method to ignore dirty logging
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Currently TDX KVM doesn't support tracking dirty pages (yet). Implement a method to ignore it. Because the flag for kvm memory slot to enable dirty logging isn't accepted for TDX, warn on the method is called for TDX.
--- arch/x86/kvm/vmx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/main.c +9 -1
@@ -847,6 +847,14 @@ static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)return vmx_get_mt_mask (vcpu, gfn, is_mmio); } +static void vt_update_cpu_dirty_logging (struct kvm_vcpu *vcpu) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_update_cpu_dirty_logging (vcpu); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -990,7 +998,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .sched_in = vt_sched_in, .cpu_dirty_log_size = PML_ENTITY_NUM, - .update_cpu_dirty_logging = vmx_update_cpu_dirty_logging, + .update_cpu_dirty_logging = vt_update_cpu_dirty_logging, .nested_ops = &vmx_nested_ops,
Signature · 1 line
2.25.1
PATCH v18 110/121 KVM: TDX: Add methods to ignore VMX preemption timer
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX doesn't support VMX preemption timer. Implement access methods for VMM to ignore VMX preemption timer.
--- arch/x86/kvm/vmx/main.c | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +23 -2
@@ -855,6 +855,27 @@ static void vt_update_cpu_dirty_logging(struct kvm_vcpu *vcpu)vmx_update_cpu_dirty_logging (vcpu); } +#ifdef CONFIG_X86_64 +static int vt_set_hv_timer (struct kvm_vcpu *vcpu,u64 guest_deadline_tsc, +bool *expired) +{ +/* VMX-preemption timer isn't available for TDX. */ +if (is_td_vcpu (vcpu)) +return -EINVAL ; + +return vmx_set_hv_timer (vcpu, guest_deadline_tsc, expired); +} + +static void vt_cancel_hv_timer (struct kvm_vcpu *vcpu) +{ +/* VMX-preemption timer can't be set. See vt_set_hv_timer(). */ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_cancel_hv_timer (vcpu); +} +#endif +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -1006,8 +1027,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .pi_start_assignment = vmx_pi_start_assignment, #ifdef CONFIG_X86_64 - .set_hv_timer = vmx_set_hv_timer, - .cancel_hv_timer = vmx_cancel_hv_timer, + .set_hv_timer = vt_set_hv_timer, + .cancel_hv_timer = vt_cancel_hv_timer, #endif .setup_mce = vmx_setup_mce,
Signature · 1 line
2.25.1
PATCH v18 111/121 KVM: TDX: Add methods to ignore accesses to TSC
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX protects TDX guest TSC state from VMM. Implement access methods to ignore guest TSC.
--- arch/x86/kvm/vmx/main.c | 44 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 40 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/main.c +40 -4
@@ -847,6 +847,42 @@ static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)return vmx_get_mt_mask (vcpu, gfn, is_mmio); } +static u64 vt_get_l2_tsc_offset (struct kvm_vcpu *vcpu) +{ +/* TDX doesn't support L2 guest at the moment. */ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return 0 ; + +return vmx_get_l2_tsc_offset (vcpu); +} + +static u64 vt_get_l2_tsc_multiplier (struct kvm_vcpu *vcpu) +{ +/* TDX doesn't support L2 guest at the moment. */ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return 0 ; + +return vmx_get_l2_tsc_multiplier (vcpu); +} + +static void vt_write_tsc_offset (struct kvm_vcpu *vcpu) +{ +/* In TDX, tsc offset can't be changed. */ +if (is_td_vcpu (vcpu)) +return ; + +vmx_write_tsc_offset (vcpu); +} + +static void vt_write_tsc_multiplier (struct kvm_vcpu *vcpu) +{ +/* In TDX, tsc multiplier can't be changed. */ +if (is_td_vcpu (vcpu)) +return ; + +vmx_write_tsc_multiplier (vcpu); +} +static void vt_update_cpu_dirty_logging (struct kvm_vcpu *vcpu) {if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm))
@@ -1004,10 +1040,10 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .has_wbinvd_exit = cpu_has_vmx_wbinvd_exit, - .get_l2_tsc_offset = vmx_get_l2_tsc_offset, - .get_l2_tsc_multiplier = vmx_get_l2_tsc_multiplier, - .write_tsc_offset = vmx_write_tsc_offset, - .write_tsc_multiplier = vmx_write_tsc_multiplier, + .get_l2_tsc_offset = vt_get_l2_tsc_offset, + .get_l2_tsc_multiplier = vt_get_l2_tsc_multiplier, + .write_tsc_offset = vt_write_tsc_offset, + .write_tsc_multiplier = vt_write_tsc_multiplier, .load_mmu_pgd = vt_load_mmu_pgd,
Signature · 1 line
2.25.1
PATCH v18 112/121 KVM: TDX: Ignore setting up mce
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because vmx_set_mce function is VMX specific and it cannot be used for TDX. Add vt stub to ignore setting up mce for TDX.
--- arch/x86/kvm/vmx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/main.c +9 -1
@@ -912,6 +912,14 @@ static void vt_cancel_hv_timer(struct kvm_vcpu *vcpu) }#endif +static void vt_setup_mce (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_setup_mce (vcpu); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -1067,7 +1075,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .cancel_hv_timer = vt_cancel_hv_timer, #endif - .setup_mce = vmx_setup_mce, + .setup_mce = vt_setup_mce, #ifdef CONFIG_KVM_SMM .smi_allowed = vt_smi_allowed,
Signature · 1 line
2.25.1
PATCH v18 113/121 KVM: TDX: Add a method to ignore for TDX to ignore hypercall patch
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because guest TD memory is protected, VMM patching guest binary for hypercall instruction isn't possible. Add a method to ignore hypercall patching with a warning. Note: guest TD kernel needs to be modified to use TDG.VP.VMCALL for hypercall.
--- arch/x86/kvm/vmx/main.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/main.c +14 -1
@@ -744,6 +744,19 @@ static u32 vt_get_interrupt_shadow(struct kvm_vcpu *vcpu)return vmx_get_interrupt_shadow (vcpu); } +static void vt_patch_hypercall (struct kvm_vcpu *vcpu, +unsigned char *hypercall) +{ +/* +* Because guest memory is protected, guest can't be patched. TD kernel +* is modified to use TDG.VP.VMCAL for hypercall. +*/ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_patch_hypercall (vcpu, hypercall); +} +static void vt_inject_irq (struct kvm_vcpu *vcpu,bool reinjected) {if (is_td_vcpu (vcpu))
@@ -1012,7 +1025,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .update_emulated_instruction = vmx_update_emulated_instruction, .set_interrupt_shadow = vt_set_interrupt_shadow, .get_interrupt_shadow = vt_get_interrupt_shadow, - .patch_hypercall = vmx_patch_hypercall, + .patch_hypercall = vt_patch_hypercall, .inject_irq = vt_inject_irq, .inject_nmi = vt_inject_nmi, .inject_exception = vt_inject_exception,
Signature · 1 line
2.25.1
PATCH v18 114/121 KVM: TDX: Add methods to ignore virtual apic related operation
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX protects TDX guest APIC state from VMM. Implement access methods of TDX guest vAPIC state to ignore them or return zero.
--- arch/x86/kvm/vmx/main.c | 61 ++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 6 ++++ arch/x86/kvm/vmx/x86_ops.h | 3 ++ 3 files changed, 64 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/main.c +55 -6
@@ -367,6 +367,14 @@ static bool vt_apic_init_signal_blocked(struct kvm_vcpu *vcpu)return vmx_apic_init_signal_blocked (vcpu); } +static void vt_set_virtual_apic_mode (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_set_virtual_apic_mode (vcpu); + +return vmx_set_virtual_apic_mode (vcpu); +} +static void vt_apicv_pre_state_restore (struct kvm_vcpu *vcpu) {struct pi_desc *pi =vcpu_to_pi_desc (vcpu);
@@ -375,6 +383,31 @@ static void vt_apicv_pre_state_restore(struct kvm_vcpu *vcpu)memset (pi->pir,0 ,sizeof (pi->pir)); } +static void vt_hwapic_irr_update (struct kvm_vcpu *vcpu,int max_irr) +{ +if (is_td_vcpu (vcpu)) +return ; + +return vmx_hwapic_irr_update (vcpu, max_irr); +} + +static void vt_hwapic_isr_update (int max_isr) +{ +if (is_td_vcpu (kvm_get_running_vcpu ())) +return ; + +return vmx_hwapic_isr_update (max_isr); +} + +static bool vt_guest_apic_has_interrupt (struct kvm_vcpu *vcpu) +{ +/* TDX doesn't support L2 at the moment. */ +if (WARN_ON_ONCE (is_td_vcpu (vcpu))) +return false; + +return vmx_guest_apic_has_interrupt (vcpu); +} +static int vt_sync_pir_to_irr (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -828,6 +861,22 @@ static void vt_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr)vmx_update_cr8_intercept (vcpu, tpr, irr); } +static void vt_set_apic_access_page_addr (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_apic_access_page_addr (vcpu); +} + +static void vt_refresh_apicv_exec_ctrl (struct kvm_vcpu *vcpu) +{ +if (WARN_ON_ONCE (is_td_vcpu (vcpu))) +return ; + +vmx_refresh_apicv_exec_ctrl (vcpu); +} +static void vt_load_eoi_exitmap (struct kvm_vcpu *vcpu,u64 *eoi_exit_bitmap) {if (is_td_vcpu (vcpu))
@@ -1037,15 +1086,15 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .enable_nmi_window = vt_enable_nmi_window, .enable_irq_window = vt_enable_irq_window, .update_cr8_intercept = vt_update_cr8_intercept, - .set_virtual_apic_mode = vmx_set_virtual_apic_mode, - .set_apic_access_page_addr = vmx_set_apic_access_page_addr, - .refresh_apicv_exec_ctrl = vmx_refresh_apicv_exec_ctrl, + .set_virtual_apic_mode = vt_set_virtual_apic_mode, + .set_apic_access_page_addr = vt_set_apic_access_page_addr, + .refresh_apicv_exec_ctrl = vt_refresh_apicv_exec_ctrl, .load_eoi_exitmap = vt_load_eoi_exitmap, .apicv_pre_state_restore = vt_apicv_pre_state_restore, .required_apicv_inhibits = VMX_REQUIRED_APICV_INHIBITS, - .hwapic_irr_update = vmx_hwapic_irr_update, - .hwapic_isr_update = vmx_hwapic_isr_update, - .guest_apic_has_interrupt = vmx_guest_apic_has_interrupt, + .hwapic_irr_update = vt_hwapic_irr_update, + .hwapic_isr_update = vt_hwapic_isr_update, + .guest_apic_has_interrupt = vt_guest_apic_has_interrupt, .sync_pir_to_irr = vt_sync_pir_to_irr, .deliver_interrupt = vt_deliver_interrupt, .dy_apicv_has_pending_interrupt = pi_has_pending_interrupt,
arch/x86/kvm/vmx/tdx.c +6 -0
@@ -2126,6 +2126,12 @@ void tdx_enable_smi_window(struct kvm_vcpu *vcpu) }#endif +void tdx_set_virtual_apic_mode (struct kvm_vcpu *vcpu) +{ +/* Only x2APIC mode is supported for TD. */ +WARN_ON_ONCE (kvm_get_apic_mode (vcpu) !=LAPIC_MODE_X2APIC ); +} +int tdx_get_cpl (struct kvm_vcpu *vcpu) {return 0 ;
arch/x86/kvm/vmx/x86_ops.h +3 -0
@@ -168,6 +168,7 @@ void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason,bool tdx_has_emulated_msr (u32 index,bool write);int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr);int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr); +void tdx_set_virtual_apic_mode (struct kvm_vcpu *vcpu);int tdx_get_cpl (struct kvm_vcpu *vcpu);void tdx_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg);
@@ -220,6 +221,8 @@ static inline bool tdx_has_emulated_msr(u32 index, bool write) { return false; }static inline int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; }static inline int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; } +static inline void tdx_set_virtual_apic_mode (struct kvm_vcpu *vcpu) {} +static inline int tdx_get_cpl (struct kvm_vcpu *vcpu) {return 0 ; }static inline void tdx_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg) {}static inline unsigned long tdx_get_rflags (struct kvm_vcpu *vcpu) {return 0 ; }
Signature · 1 line
2.25.1
PATCH v18 115/121 KVM: TDX: Inhibit APICv for TDX guest
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX doesn't support APICV, inhibit APICv for TDX guest. Follow how SEV does it. Define a new inhibit reason for TDX, set it on TD initialization, and add the flag to kvm_x86_ops.required_apicv_inhibits.
--- arch/x86/include/asm/kvm_host.h | 9 +++++++++ arch/x86/kvm/vmx/main.c | 3 ++- arch/x86/kvm/vmx/tdx.c | 4 ++++ 3 files changed, 15 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +9 -0
@@ -1314,6 +1314,15 @@ enum kvm_apicv_inhibit {* mapping between logical ID and vCPU. */ APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED , + +/*********************************************************/ +/* INHIBITs that are relevant only to the Intel's APICv. */ +/*********************************************************/ + +/* +* APICv is disabled because TDX doesn't support it. +*/ + APICV_INHIBIT_REASON_TDX, }; struct kvm_arch {
arch/x86/kvm/vmx/main.c +2 -1
@@ -1005,7 +1005,8 @@ static int vt_vcpu_mem_enc_ioctl(struct kvm_vcpu *vcpu, void __user *argp) BIT(APICV_INHIBIT_REASON_BLOCKIRQ) | \ BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED) | \ BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) | \ - BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED)) + BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED) | \ + BIT(APICV_INHIBIT_REASON_TDX))struct kvm_x86_ops vt_x86_ops__initdata = { .name =KBUILD_MODNAME ,
arch/x86/kvm/vmx/tdx.c +4 -0
@@ -2533,6 +2533,8 @@ static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params, goto teardown; } +kvm_set_apicv_inhibit (kvm,APICV_INHIBIT_REASON_TDX ); +return 0 ;/*
@@ -2895,6 +2897,8 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u64 vcpu_rcx)return -EIO ; } +WARN_ON_ONCE (kvm_apicv_activated (vcpu->kvm)); + vcpu->arch.apic->apicv_active = false; vcpu->arch.mp_state =KVM_MP_STATE_RUNNABLE ; tdx->td_vcpu_created = true;return 0 ;
Signature · 1 line
2.25.1
PATCH v18 116/121 Documentation/virt/kvm: Document on Trust Domain Extensions(TDX)
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add documentation to Intel Trusted Domain Extensions(TDX) support.
--- Documentation/virt/kvm/api.rst | 9 +- Documentation/virt/kvm/x86/index.rst | 1 + Documentation/virt/kvm/x86/intel-tdx.rst | 362 +++++++++++++++++++++++ 3 files changed, 371 insertions(+), 1 deletion(-) create mode 100644 Documentation/virt/kvm/x86/intel-tdx.rst
Documentation/virt/kvm/api.rst +8 -1
@@ -1394,6 +1394,9 @@ the memory region are automatically reflected into the guest. For example, an mmap() that affects the region will be made visible immediately. Another example is madvise(MADV_DROP). +For TDX guest, deleting/moving memory region loses guest memory contents. +Read only region isn't supported. Only as-id 0 is supported. + Note: On arm64, a write generated by the page-table walker (to update the Access and Dirty flags, for example) never results in a KVM_EXIT_MMIO exit when the slot has the KVM_MEM_READONLY flag. This
@@ -4734,7 +4737,7 @@ H_GET_CPU_CHARACTERISTICS hypercall. :Capability: basic :Architectures: x86 -:Type: vm +:Type: vm ioctl, vcpu ioctl :Parameters: an opaque platform specific structure (in/out) :Returns: 0 on success; -1 on error
@@ -4746,6 +4749,10 @@ Currently, this ioctl is used for issuing Secure Encrypted Virtualization (SEV) commands on AMD Processors. The SEV commands are defined in Documentation/virt/kvm/x86/amd-memory-encryption.rst. +Currently, this ioctl is used for issuing Trusted Domain Extensions +(TDX) commands on Intel Processors. The TDX commands are defined in +Documentation/virt/kvm/x86/intel-tdx.rst. + 4.111 KVM_MEMORY_ENCRYPT_REG_REGION -----------------------------------
Documentation/virt/kvm/x86/index.rst +1 -0
@@ -11,6 +11,7 @@ KVM for x86 systems cpuid errata hypercalls + intel-tdx mmu msr nested-vmx
Documentation/virt/kvm/x86/intel-tdx.rst new file +362 -0
362 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v18 117/121 KVM: x86: design documentation on TDX support of x86 KVM TDP MMU
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add a high level design document on TDX changes to TDP MMU.
--- Documentation/virt/kvm/x86/index.rst | 1 + Documentation/virt/kvm/x86/tdx-tdp-mmu.rst | 443 +++++++++++++++++++++ 2 files changed, 444 insertions(+) create mode 100644 Documentation/virt/kvm/x86/tdx-tdp-mmu.rst
Documentation/virt/kvm/x86/index.rst +1 -0
@@ -16,4 +16,5 @@ KVM for x86 systems msr nested-vmx running-nested-guests + tdx-tdp-mmu timekeeping
Documentation/virt/kvm/x86/tdx-tdp-mmu.rst new file +443 -0
443 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v18 118/121 KVM: TDX: Add hint TDX ioctl to release Secure-EPT
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add a new hint KVM TDX ioctl to release Secure-EPT as an optimization to reduce the time of the destruction of the guest.
It takes tens of minutes to destroy a guest with tens or hundreds of GB of guest memory. There are two cases to release pages used for the Secure-EPT and guest private memory. One case is runtime while the guest is still running. Another case is static when the TD won't run anymore.
In Runtime: Use this when the KVM memory slot is deleted or closes KVM file descriptors while the user process is live. Because the guest can still run, a TLB shoot-down is needed. The sequence is TLB shoot down, cache flush each page, releasing the page from the Secure-EPT tree, and zero-clear them. It requires four SEAMCALLs per page. TDH.MEM.RANGE.BLOCK() and TDH.MEM.TRACK() for TLB shoot down, TDH.PHYMEM.PAGE.WBINVD() for cache flush, and TDH.MEM.PAGE.REMOVE() to release a page.
In process existing: When we know the vcpu won't run further, KVM can free the host key ID (HKID) for memory encryption with cache flush. The vcpu can't run after that. It simplifies the sequence to release private pages by reclaiming and zeroing them to reduce the number of SEAMCALLs to one per private page, TDH.PHYMEM.PAGE.RECLAIM(). However, this is applicable only when the user process exits with the MMU notifier release callback.
Add a way for the user space to tell KVM a hint when it starts to destruct the guest for the efficient way in addition to the MMU notifier.
---
v16
- Newly added
--- arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/vmx/tdx.c | 9 +++++++++ 3 files changed, 11 insertions(+)
arch/x86/include/uapi/asm/kvm.h +1 -0
@@ -574,6 +574,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_INIT_VCPU ,KVM_TDX_INIT_MEM_REGION ,KVM_TDX_FINALIZE_VM , +KVM_TDX_RELEASE_VM , KVM_TDX_CMD_NR_MAX, };
arch/x86/kvm/mmu/mmu.c +1 -0
@@ -6908,6 +6908,7 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)static_call_cond (kvm_x86_flush_shadow_all_private)(kvm);kvm_mmu_zap_all (kvm); } +EXPORT_SYMBOL_GPL (kvm_arch_flush_shadow_all);static void kvm_mmu_zap_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) {
arch/x86/kvm/vmx/tdx.c +9 -0
@@ -2818,6 +2818,15 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)case KVM_TDX_FINALIZE_VM : r =tdx_td_finalizemr (kvm);break ; +case KVM_TDX_RELEASE_VM : { +int idx; + + idx =srcu_read_lock (&kvm->srcu); +kvm_arch_flush_shadow_all (kvm); +srcu_read_unlock (&kvm->srcu, idx); + r =0 ; +break ; + }default : r = -EINVAL ; goto out;
Signature · 1 line
2.25.1
PATCH v18 119/121 RFC: KVM: x86: Add x86 callback to check cpuid
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The x86 backend should check the consistency of KVM_SET_CPUID2 because it has its constraint. Add a callback for it. The backend code will come as another patch.
--- arch/x86/include/asm/kvm-x86-ops.h | 2 ++ arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/cpuid.c | 20 ++++++++++++-------- 3 files changed, 15 insertions(+), 8 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +2 -0
@@ -20,6 +20,8 @@ KVM_X86_OP(hardware_disable) KVM_X86_OP(hardware_unsetup) KVM_X86_OP_OPTIONAL_RET0(offline_cpu) KVM_X86_OP(has_emulated_msr) +/* TODO: Once all backend implemented this op, remove _OPTIONAL_RET0. */ +KVM_X86_OP_OPTIONAL_RET0(vcpu_check_cpuid) KVM_X86_OP(vcpu_after_set_cpuid) KVM_X86_OP(is_vm_type_supported) KVM_X86_OP_OPTIONAL(max_vcpus);
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1655,6 +1655,7 @@ struct kvm_x86_ops {void (*hardware_unsetup)(void );int (*offline_cpu)(void );bool (*has_emulated_msr)(struct kvm *kvm,u32 index); +int (*vcpu_check_cpuid)(struct kvm_vcpu *vcpu,struct kvm_cpuid_entry2 *e2,int nent);void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu);bool (*is_vm_type_supported)(unsigned long vm_type);
arch/x86/kvm/cpuid.c +12 -8
@@ -136,6 +136,7 @@ static int kvm_check_cpuid(struct kvm_vcpu *vcpu, {struct kvm_cpuid_entry2 *best;u64 xfeatures; +int r;/* * The existing code assumes virtual address is 48-bit or 57-bit in the
@@ -155,15 +156,18 @@ static int kvm_check_cpuid(struct kvm_vcpu *vcpu,* enabling in the FPU, e.g. to expand the guest XSAVE state size. */ best =cpuid_entry2_find (entries, nent,0xd ,0 ); -if (!best) -return 0 ; - - xfeatures = best->eax | ((u64 )best->edx <<32 ); - xfeatures &=XFEATURE_MASK_USER_DYNAMIC ; -if (!xfeatures) -return 0 ; +if (best) { + xfeatures = best->eax | ((u64 )best->edx <<32 ); + xfeatures &=XFEATURE_MASK_USER_DYNAMIC ; +if (xfeatures) { + r =fpu_enable_guest_xfd_features (&vcpu->arch.guest_fpu, + xfeatures); +if (r) +return r; + } + } -return fpu_enable_guest_xfd_features (&vcpu->arch.guest_fpu, xfeatures); +return static_call (kvm_x86_vcpu_check_cpuid)(vcpu, entries, nent); }/* Check whether the supplied CPUID data is equal to what is already set for the vCPU. */
Signature · 1 line
2.25.1
PATCH v18 120/121 RFC: KVM: x86, TDX: Add check for KVM_SET_CPUID2
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Implement a hook of KVM_SET_CPUID2 for additional consistency check.
Intel TDX or AMD SEV has a restriction on the value of cpuid. For example, some values must be the same between all vcpus. Check if the new values are consistent with the old values. The check is light because the cpuid consistency is very model specific and complicated. The user space VMM should set cpuid and MSRs consistently.
---
v18:
- Use TDH.SYS.RD() instead of struct tdsysinfo_struct
--- arch/x86/kvm/vmx/main.c | 10 ++++++ arch/x86/kvm/vmx/tdx.c | 66 ++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.h | 7 ++++ arch/x86/kvm/vmx/x86_ops.h | 4 +++ 4 files changed, 81 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/main.c +10 -0
@@ -447,6 +447,15 @@ static void vt_vcpu_deliver_init(struct kvm_vcpu *vcpu)kvm_vcpu_deliver_init (vcpu); } +static int vt_vcpu_check_cpuid (struct kvm_vcpu *vcpu, +struct kvm_cpuid_entry2 *e2,int nent) +{ +if (is_td_vcpu (vcpu)) +return tdx_vcpu_check_cpuid (vcpu, e2, nent); + +return 0 ; +} +static void vt_vcpu_after_set_cpuid (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -1107,6 +1116,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .get_exit_info = vt_get_exit_info, + .vcpu_check_cpuid = vt_vcpu_check_cpuid, .vcpu_after_set_cpuid = vt_vcpu_after_set_cpuid, .has_wbinvd_exit = cpu_has_vmx_wbinvd_exit,
arch/x86/kvm/vmx/tdx.c +60 -6
117 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +7 -0
@@ -32,6 +32,13 @@ struct kvm_tdx {atomic_t tdh_mem_track;u64 tsc_offset; + +/* +* For KVM_SET_CPUID to check consistency. Remember the one passed to +* TDH.MNG_INIT +*/ +int cpuid_nent; +struct kvm_cpuid_entry2 *cpuid; };union tdx_exit_reason {
arch/x86/kvm/vmx/x86_ops.h +4 -0
@@ -162,6 +162,8 @@ u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio);void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector); +int tdx_vcpu_check_cpuid (struct kvm_vcpu *vcpu,struct kvm_cpuid_entry2 *e2, +int nent);void tdx_inject_nmi (struct kvm_vcpu *vcpu);void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason,u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code);
@@ -214,6 +216,8 @@ static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)static inline void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector) {} +static inline int tdx_vcpu_check_cpuid (struct kvm_vcpu *vcpu,struct kvm_cpuid_entry2 *e2, +int nent) {return -EOPNOTSUPP ; }static inline void tdx_inject_nmi (struct kvm_vcpu *vcpu) {}static inline void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason,u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code) {}
Signature · 1 line
2.25.1
PATCH v18 121/121 the end of (the first phase of) TDX KVM patch series
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the end of (the first phase of) patch series of TDX KVM support.
--- Documentation/virt/kvm/index.rst | 1 - .../virt/kvm/intel-tdx-layer-status.rst | 33 ------------------- 2 files changed, 34 deletions(-) delete mode 100644 Documentation/virt/kvm/intel-tdx-layer-status.rst
Documentation/virt/kvm/index.rst +0 -1
@@ -20,4 +20,3 @@ KVM halt-polling review-checklist - intel-tdx-layer-status
Documentation/virt/kvm/intel-tdx-layer-status.rst deleted +0 -33
@@ -1,33 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0 - -=================================== -Intel Trust Dodmain Extensions(TDX) -=================================== - -Layer status -============ -What qemu can do ----------------- -- TDX VM TYPE is exposed to Qemu. -- Qemu can create/destroy guest of TDX vm type. -- Qemu can create/destroy vcpu of TDX vm type. -- Qemu can populate initial guest memory image. -- Qemu can finalize guest TD. -- Qemu can start to run vcpu. But vcpu can not make progress yet. - -Patch Layer status ------------------- - Patch layer Status - -* TDX, VMX coexistence: Applied -* TDX architectural definitions: Applied -* TD VM creation/destruction: Applied -* TD vcpu creation/destruction: Applied -* TDX EPT violation: Applied -* TD finalization: Applied -* TD vcpu enter/exit: Applied -* TD vcpu interrupts/exit/hypercall: Not yet - -* KVM MMU GPA shared bits: Applied -* KVM TDP refactoring for TDX: Applied -* KVM TDP MMU hooks: Applied
Signature · 1 line
2.25.1
On Mon, Jan 22, 2024 at 03:52:36PM -0800, [email protected] wrote:
...
Quoted 78 lines
The below layers are chosen so that the device model, for example, qemu can exercise each layering step by step. Check if TDX is supported, create TD VM, create TD vcpu, allow vcpu running, populate TD guest private memory, and handle vcpu exits/hypercalls/interrupts to run TD fully.
TDX vcpu
interrupt/exits/hypercall<------------\^ | | | TD finalization | ^ | | | TDX EPT violation<------------\ | ^ | | | | | TD vcpu enter/exit | | ^ | | | | | TD vcpu creation/destruction | \-------KVM TDP MMU MapGPA ^ | ^ | | | TD VM creation/destruction \---------------KVM TDP MMU hooks^ ^
| |
TDX architectural definitions KVM TDP refactoring for TDX
^ ^
| |
TDX, VMX <--------TDX host kernel KVM MMU GPA stolen bits
coexistence supportThe followings are explanations of each layer. Each layer has a dummy commit that starts with [MARKER] in subject. It is intended to help to identify where each layer starts.
TDX host kernel support:
https://lore.kernel.org/lkml/[email protected]/
The guts of system-wide initialization of TDX module. There is an independent patch series for host x86. TDX KVM patches call functions this patch series provides to initialize the TDX module.TDX, VMX coexistence:
Infrastructure to allow TDX to coexist with VMX and trigger the initialization of the TDX module.
This layer starts with
"KVM: VMX: Move outvmx_x86_opsto 'main.c' to wrap VMX and TDX"
TDX architectural definitions:
Add TDX architectural definitions and helper functions
This layer starts with
"[MARKER] The start of TDX KVM patch series: TDX architectural definitions".
TD VM creation/destruction:
Guest TD creation/destroy allocation and releasing of TDX specific vm and vcpu structure. Create an initial guest memory image with TDX measurement.
This layer starts with
"[MARKER] The start of TDX KVM patch series: TD VM creation/destruction".
TD vcpu creation/destruction:
guest TD creation/destroy Allocation and releasing of TDX specific vm and vcpu structure. Create an initial guest memory image with TDX measurement.
This layer starts with
"[MARKER] The start of TDX KVM patch series: TD vcpu creation/destruction"
TDX EPT violation:
Create an initial guest memory image with TDX measurement. Handle secure EPT violations to populate guest pages with TDX SEAMCALLs. This layer starts with
"[MARKER] The start of TDX KVM patch series: TDX EPT violation"
TD vcpu enter/exit:
Allow TDX vcpu to enter into TD and exit from TD. Save CPU state before entering into TD. Restore CPU state after exiting from TD.
This layer starts with
"[MARKER] The start of TDX KVM patch series: TD vcpu enter/exit"
TD vcpu interrupts/exit/hypercall:
Handle various exits/hypercalls and allow interrupts to be injected so that TD vcpu can continue running.
This layer starts with
"[MARKER] The start of TDX KVM patch series: TD vcpu exits/interrupts/hypercalls"
Quoted 3 lines
KVM MMU GPA shared bit:
Introduce framework to handle shared bit repurposed bit of GPA TDX repurposed a bit of GPA to indicate shared or private. If it's shared,
How about:
Introduce framework to handle shared bit which is repurposed GPA bit to indicate shared or private.
Quoted 17 lines
it's the same as the conventional VMX EPT case. VMM can access shared guest pages. If it's private, it's handled by Secure-EPT and the guest page is encrypted.
This layer starts with
"[MARKER] The start of TDX KVM patch series: KVM MMU GPA stolen bits"
KVM TDP refactoring for TDX:
TDX Secure EPT requires different constants. e.g. initial value EPT entry value etc. Various refactoring for those differences. This layer starts with
"[MARKER] The start of TDX KVM patch series: KVM TDP refactoring for TDX"
KVM TDP MMU hooks:
Introduce framework to TDP MMU to add hooks in addition to direct EPT access TDX added Secure EPT which is an enhancement to VMX EPT. Unlike conventional VMX EPT, CPU can't directly read/write Secure EPT. Instead, use TDX SEAMCALLs to operate on Secure EPT.
This layer starts with
"[MARKER] The start of TDX KVM patch series: KVM TDP MMU hooks"
...
Quoted 7 lines
Several VMX control structures (such as Shared EPT and Posted interrupt descriptor) are directly managed and accessed by the host VMM. These control structures are pointed to by fields in the TD VMCS.
The above means that 1) KVM needs to allocate different data structures for TDs,
2) KVM can reuse the existing code for TDs for some operations, 3) it needs to define TD-specific handling for others. 3) Redirect operations to . 3)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
unnecessary duplication.
Quoted 463 lines
Redirect operations to the TDX specific callbacks, like "if (
is_td_vcpu(vcpu))tdx_callback()elsevmx_callback();".*TD Private Memory
TD private memory is designed to hold TD private content, encrypted by the CPU using the TD ephemeral key. An encryption engine holds a table of encryption keys, and an encryption key is selected for each memory transaction based on a Host Key Identifier (HKID). By design, the host VMM does not have access to the encryption keys.In the first generation of MKTME, HKID is "stolen" from the physical address by allocating a configurable number of bits from the top of the physical address. The HKID space is partitioned into shared HKIDs for legacy MKTME accesses and private HKIDs for SEAM-mode-only accesses. We use 0 for the shared HKID on the host so that MKTME can be opaque or bypassed on the host.
During TDX non-root operation (i.e. guest TD), memory accesses can be qualified as either shared or private, based on the value of a new SHARED bit in the Guest Physical Address (GPA). The CPU translates shared GPAs using the usual VMX EPT (Extended Page Table) or "Shared EPT" (in this document), which resides in host VMM memory. The Shared EPT is directly managed by the host VMM - the same as with the current VMX. Since guest TDs usually require I/O, and the data exchange needs to be done via shared memory, thus KVM needs to use the current EPT functionality even for TDs.
* Secure EPT and Minoring using the TDP code
The CPU translates private GPAs using a separate Secure EPT. The Secure EPT pages are encrypted and integrity-protected with the TD's ephemeral private key. Secure EPT can be managed _indirectly_ by the host VMM, using the TDX interface functions, and thus conceptually Secure EPT is a subset of EPT (why "subset"). Since execution of such interface functions takes much longer time than accessing memory directly, in KVM we use the existing TDP code to minor the Secure EPT for the TD.This way, we can effectively walk Secure EPT without using the TDX interface functions.
* VM life cycle and TDX specific operations
The userspace VMM, such as QEMU, needs to build and treat TDs differently. For example, a TD needs to boot in private memory, and the host software cannot copy the initial image to private memory.* TSC Virtualization
The TDX module helps TDs maintain reliable TSC (Time Stamp Counter) values (e.g. consistent among the TD VCPUs) and the virtual TSC frequency is determined by TD configuration, i.e. when the TD is created, not per VCPU. The current KVM owns TSC virtualization for VMs, but the TDX module does for TDs.* MCE support for TDs
The TDX module doesn't allow VMM to inject MCE. Instead PV way is needed for TD to communicate with VMM. For now, KVM silently ignores MCE request by VMM. MSRs related to MCE (e.g, MCE bank registers) can be naturally emulated by paravirtualizing MSR access.[1] For details, the specifications, [2], [3], [4], [5], [6], [7], are available.
* Restrictions or future work
Some features are not included to reduce patch size. Those features are addressed as future independent patch series.
- large page (2M, 1G)
- qemu gdb stub
- guest PMU
- and more* Prerequisites
It's required to load the TDX module and initialize it. It's out of the scope of this patch series. Another independent patch for the common x86 code is planned. It definesCONFIG_INTEL_TDX_HOSTand this patch series usesCONFIG_INTEL_TDX_HOST. It's assumed that WithCONFIG_INTEL_TDX_HOST=y, the TDX module is initialized and ready for KVM to use the TDX module APIs for TDX guest life cycle like tdh.mng.init are ready to use.Concretely Global initialization, LP (Logical Processor) initialization, global configuration, the key configuration, and TDMR and PAMT initialization are done. The state of the TDX module is
SYS_READY. Please refer to the TDX module specification, the chapter Intel TDX Module Lifecycle State Machine** Detecting the TDX module readiness.
TDX host patch series implements the detection of the TDX module availability and its initialization so that KVM can use it. Also it manages Host KeyID (HKID) assigned to guest TD.
The assumed APIs the TDX host patch series provides are
- const structtdsysinfo_struct*tdx_get_sysinfo(void);
Return the system wide information about the TDX module. NULL if the TDX isn't initialized.
- inttdx_enable(void);
Initialization of TDX module so that the TDX module is ready for KVM to use.
- extern u32tdx_global_keyid__read_mostly;
global host key id that is used for the TDX module itself.
- u32tdx_get_num_keyid(void);
return the number of available TDX private host key id.
- inttdx_keyid_alloc(void);
Allocate HKID for guest TD.
- voidtdx_keyid_free(int keyid);
Free HKID for guest TD.(****)
* TDX KVM high-level design
- Host key ID management
Host Key ID (HKID) needs to be assigned to each TDX guest for memory encryption. It is assumed The TDX host patch series implements necessary functions, u32tdx_get_global_keyid(void), inttdx_keyid_alloc(void)and,
voidtdx_keyid_free(int keyid).- Data structures and VM type
Because TDX is different from VMX, define its own VM/VCPU structures, structkvm_tdxand structvcpu_tdxinstead of structkvm_vmxand structvcpu_vmx. To identify the VM, introduce VM-type to specify which VM type, VMX (default) or TDX, is used.- VM life cycle and TDX specific operations
Re-purpose the existingKVM_MEMORY_ENCRYPT_OPto add TDX specific operations. New commands are used to get the TDX system parameters, set TDX specific VM/VCPU parameters, set initial guest memory and measurement.The creation of TDX VM requires five additional operations in addition to the conventional VM creation.
- Get KVM system capability to check if TDX VM type is supported
- VM creation (KVM_CREATE_VM)
- New: Get the TDX specific system parameters.KVM_TDX_GET_CAPABILITY.
- New: Set TDX specific VM parameters.KVM_TDX_INIT_VM.
- VCPU creation (KVM_CREATE_VCPU)
- New: Set TDX specific VCPU parameters.KVM_TDX_INIT_VCPU.
- New: Initialize guest memory as boot state and extend the measurement with the memory.KVM_TDX_INIT_MEM_REGION.
- New: Finalize VM.KVM_TDX_FINALIZE. Complete measurement of the initial TDX VM contents.
- VCPU RUN (KVM_VCPU_RUN)- Protected guest state
Because the guest state (CPU state and guest memory) is protected, the KVM VMM can't operate on them. For example, accessing CPU registers, injecting exceptions, and accessing guest memory. Those operations are handled as silently ignored, returning zero or initial reset value when it's requested via KVM API ioctls.VM/VCPU state and callbacks for TDX specific operations. Define tdx specific VM state and VCPU state instead of VMX ones. Redirect operations to TDX specific callbacks. "if (tdx)
tdx_op()elsevmx_op()".Operations on the CPU state
silently ignore operations on the guest state. For example, the write to CPU registers is ignored and the read from CPU registers returns 0.. ignore access to CPU registers except for allowed ones.
. TSC: add a check if tsc is immutable and return an error. Because the KVM
implementation updates the internal tsc state and it's difficult to back out those changes. Instead, skip the logic.
. dirty logging: add check if dirty logging is supported.
. exceptions/SMI/MCE/SIPI/INIT: silently ignoreNote: virtual external interrupt and NMI can be injected into TDX guests.
- KVM MMU integration
One bit of the guest physical address (bit 51 or 47) is repurposed to indicate if the guest physical address is private (the bit is cleared) or shared (the bit is set). The bits are called stolen bits.- Stolen bits framework
systematically tracks which guest physical address, shared or private, is used.- Shared EPT and secure EPT
There are two EPTs. Shared EPT (the conventional one) and Secure EPT(the new one). Shared EPT is handled the same for the stolen bit set. Secure EPT points to private guest pages. To resolve EPT violation, KVM walks one of two EPTs based on faulted GPA. Because it's costly to access secure EPT during walking EPTs with SEAMCALLs for the private guest physical address, another private EPT is used as a shadow of Secure-EPT with the existing logic at the cost of extra memory.The following depicts the relationship.
KVM | TDX module | | | -------------+---------- | | | | | | V V | | shared GPA private GPA | | CPU shared EPT pointer KVM private EPT pointer | CPU secure EPT pointer | | | | | | | | V V | V shared EPT private EPT--------mirror----->Secure EPT | | | | | \--------------------+------\ | | | | | V | V V shared guest page | private guest page | | non-encrypted memory | encrypted memory |- Operating on Secure EPT
Use the TDX module APIs to operate on Secure EPT. To call the TDX API during resolving EPT violation, add hooks to additional operation and wiring it to TDX backend.* References
[1] TDX specification
https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html
[2] Intel Trust Domain Extensions (Intel TDX)
https://cdrdv2.intel.com/v1/dl/getContent/726790
[3] Intel CPU Architectural Extensions Specification
https://www.intel.com/content/dam/develop/external/us/en/documents-tps/intel-tdx-cpu-architectural-specification.pdf
[4] Intel TDX Module 1.0 Specification
https://www.intel.com/content/dam/develop/external/us/en/documents/tdx-module-1.0-public-spec-v0.931.pdf
[5] Intel TDX Loader Interface Specification
https://www.intel.com/content/dam/develop/external/us/en/documents-tps/intel-tdx-seamldr-interface-specification.pdf
[6] Intel TDX Guest-Hypervisor Communication Interface
https://cdrdv2.intel.com/v1/dl/getContent/726790
[7] Intel TDX Virtual Firmware Design Guide
https://www.intel.com/content/dam/develop/external/us/en/documents/tdx-virtual-firmware-design-guide-rev-1.01.pdf
[8] intel public github
kvm TDX branch: https://github.com/intel/tdx/tree/kvm
TDX guest branch: https://github.com/intel/tdx/tree/guest
qemu TDX https://github.com/intel/qemu-tdx
[9] TDVF
https://github.com/tianocore/edk2-staging/tree/TDVF
This was merged into EDK2 main branch. https://github.com/tianocore/edk2Chao Gao (2):
KVM: x86/mmu: Assume guest MMIOs are shared
KVM: x86: Allow to update cached values inkvm_user_return_msrsw/o
wrmsrIsaku Yamahata (96):
KVM: x86: Addis_vm_type_supportedcallback
KVM: x86/vmx: initializeloaded_vmcss_on_cpuinvmx_hardware_setup()
KVM: x86/vmx: Refactor KVM VMX module init/exit functions
KVM: VMX: Reorder vmx initialization with kvm vendor initialization
KVM: TDX: Initialize the TDX module when loading the KVM intel kernel
module
KVM: TDX: Add placeholders for TDX VM/vcpu structure
KVM: TDX: Make TDX VM type supported
[MARKER] The start of TDX KVM patch series: TDX architectural
definitions
KVM: TDX: Define TDX architectural definitions
KVM: TDX: Add C wrapper functions for SEAMCALLs to the TDX module
KVM: TDX: Retry SEAMCALL on the lack of entropy error
KVM: TDX: Add helper functions to print TDX SEAMCALL error [MARKER] The start of TDX KVM patch series: TD VM creation/destruction
KVM: TDX: Add helper functions to allocate/free TDX private host key
id
KVM: TDX: Add helper function to read TDX metadata in array x86/virt/tdx: Get system-wide info about TDX module on initialization
KVM: TDX: Add place holder for TDX VM specificmem_enc_opioctl
KVM: TDX: MakeKVM_CAP_MAX_VCPUSbackend specific
KVM: TDX: create/destroy VM structure
KVM: TDX: initialize VM with TDX specific parameters
KVM: TDX: Makepmu_intel.cignore guest TD case
KVM: TDX: Refuse to unplug the last cpu on the package [MARKER] The start of TDX KVM patch series: TD vcpu
creation/destruction
KVM: TDX: create/free TDX vcpu structure
KVM: TDX: Do TDX specific vcpu initialization
[MARKER] The start of TDX KVM patch series: KVM MMU GPA shared bits
KVM: x86/mmu: introduce config for PRIVATE KVM MMU
KVM: x86/mmu: Add address conversion functions for TDX shared bit of
GPA
[MARKER] The start of TDX KVM patch series: KVM TDP refactoring for
TDX
KVM: x86/mmu: Replace hardcoded value 0 for the initial value for SPTE
KVM: x86/mmu: Add Suppress VE bit to
shadow_mmio_mask/shadow_present_mask
KVM: x86/mmu: Track shadow MMIO value on a per-VM basis
KVM: x86/mmu: Disallow fast page fault on private GPA
KVM: VMX: Introduce test mode related to EPT violation VE [MARKER] The start of TDX KVM patch series: KVM TDP MMU hooks
KVM: x86/tdp_mmu: Init role member of structkvm_mmu_pageat
allocation
KVM: x86/mmu: Add a newis_privatemember for unionkvm_mmu_page_role
KVM: x86/mmu: Add a private pointer to structkvm_mmu_page
KVM: x86/tdp_mmu: Apply mmu notifier callback to only shared GPA
KVM: x86/tdp_mmu: Sprinkle__must_check
KVM: x86/tdp_mmu: Support TDX private mapping for TDP MMU [MARKER] The start of TDX KVM patch series: TDX EPT violation
KVM: TDX: Add accessors VMX VMCS helpers
KVM: TDX: Require TDP MMU and mmio caching for TDX
KVM: TDX: TDP MMU TDX support
KVM: TDX: MTRR: implementget_mt_mask()for TDX
[MARKER] The start of TDX KVM patch series: TD finalization
KVM: TDX: Create initial guest memory
KVM: TDX: Finalize VM initialization
[MARKER] The start of TDX KVM patch series: TD vcpu enter/exit
KVM: TDX: Implement TDX vcpu enter/exit path
KVM: TDX:vcpu_run: save/restore host state(host kernel gs)
KVM: TDX: restore host xsave state when exit from the guest TD
KVM: TDX: restore user ret MSRs
[MARKER] The start of TDX KVM patch series: TD vcpu
exits/interrupts/hypercalls
KVM: TDX: complete interrupts after tdexit
KVM: TDX: restore debug store when TD exit
KVM: TDX: handle vcpu migration over logical processor
KVM: x86: Add aswitch_db_regsflag to handle TDX's auto-switched
behavior
KVM: TDX: remove use of structvcpu_vmxfromposted_interrupt.c
KVM: TDX: Implement interrupt injection
KVM: TDX: Implements vcpurequest_immediate_exit
KVM: TDX: Implement methods to inject NMI
KVM: TDX: Add a place holder to handle TDX VM exit
KVM: TDX: handleEXIT_REASON_OTHER_SMI
KVM: TDX: handle ept violation/misconfig exit
KVM: TDX: handleEXCEPTION_NMIandEXTERNAL_INTERRUPT
KVM: TDX: HandleEXIT_REASON_OTHER_SMIwith MSMI
KVM: TDX: Add a place holder for handler of TDX hypercalls(TDG.VP.VMCALL)KVM: TDX: handle KVM hypercall with TDG.VP.VMCALL
KVM: TDX: Add KVM Exit for TDX TDG.VP.VMCALL
KVM: TDX: Handle TDX PV CPUID hypercall
KVM: TDX: Handle TDX PV HLT hypercall
KVM: TDX: Handle TDX PV port io hypercall
KVM: TDX: Implement callbacks for MSR operations for TDX
KVM: TDX: Handle TDX PV rdmsr/wrmsr hypercall
KVM: TDX: Handle MSR MTRRCap and MTRRDefType access
KVM: TDX: Handle MSRIA32_FEAT_CTLMSR andIA32_MCG_EXT_CTL
KVM: TDX: Handle TDG.VP.VMCALL<GetTdVmCallInfo> hypercall
KVM: TDX: Silently discard SMI request
KVM: TDX: Silently ignore INIT/SIPI
KVM: TDX: Add methods to ignore accesses to CPU state
KVM: TDX: Add methods to ignore guest instruction emulation
KVM: TDX: Add a method to ignore dirty logging
KVM: TDX: Add methods to ignore VMX preemption timer
KVM: TDX: Add methods to ignore accesses to TSC
KVM: TDX: Ignore setting up mce
KVM: TDX: Add a method to ignore for TDX to ignore hypercall patch
KVM: TDX: Add methods to ignore virtual apic related operation
KVM: TDX: Inhibit APICv for TDX guestDocumentation/virt/kvm: Document on Trust Domain Extensions(TDX)
KVM: x86: design documentation on TDX support of x86 KVM TDP MMU
KVM: TDX: Add hint TDX ioctl to release Secure-EPT
RFC: KVM: x86: Add x86 callback to check cpuid
RFC: KVM: x86, TDX: Add check forKVM_SET_CPUID2[MARKER] the end of (the first phase of) TDX KVM patch seriesKai Huang (2):
x86/virt/tdx: Export TDX KeyID information
x86/virt/tdx: Export SEAMCALL functionsSean Christopherson (17):
KVM: VMX: Move outvmx_x86_opsto 'main.c' to wrap VMX and TDX
KVM: TDX: Add TDX "architectural" error codes
KVM: TDX: x86: Add ioctl to get TDX systemwide parameters
KVM: Allow page-sized MMU caches to be initialized with custom 64-bit
values
KVM: x86/mmu: Allow non-zero value for non-present SPTE and removed
SPTE
KVM: x86/mmu: Allow per-VM override of the TDP max page level
KVM: x86/tdp_mmu: Don't zap private pages for unsupported cases
KVM: VMX: Split out guts of EPT violation to common/exposed function
KVM: VMX: Move setting of EPT MMU masks to common VT-x code
KVM: TDX: Addload_mmu_pgdmethod for TDX
KVM: x86/mmu: Introducekvm_mmu_map_tdp_page()for use by TDX
KVM: TDX: Add support for find pending IRQ in a protected local APIC
KVM: x86: Assume timer IRQ was injected if APIC state is proteced
KVM: VMX: Modify NMI and INTR handlers to takeintr_infoas function
argument
KVM: VMX: Move NMI/exception handler to common helper
KVM: x86: Split core of hypercall emulation to helper function
KVM: TDX: Handle TDX PV MMIO hypercallYan Zhao (1):
KVM: x86/mmu: TDX: Do not enable page track for TD guestYang Weijiang (1):
KVM: TDX: AddTSX_CTRLmsr intouret_msrslistYao Yuan (1):
KVM: TDX: Handle vmentry failure for INTEL TD guestYuan Yao (1):
KVM: TDX: Retry seamcall whenTDX_OPERAND_BUSYwith operand SEPTDocumentation/virt/kvm/api.rst | 9 +- Documentation/virt/kvm/index.rst | 1 + Documentation/virt/kvm/x86/index.rst | 2 + Documentation/virt/kvm/x86/intel-tdx.rst | 362 +++ Documentation/virt/kvm/x86/tdx-tdp-mmu.rst | 443 +++ arch/x86/events/intel/ds.c | 1 + arch/x86/include/asm/asm-prototypes.h | 1 + arch/x86/include/asm/kvm-x86-ops.h | 18 +- arch/x86/include/asm/kvm_host.h | 85 +- arch/x86/include/asm/tdx.h | 5 + arch/x86/include/asm/vmx.h | 14 + arch/x86/include/uapi/asm/kvm.h | 95 + arch/x86/include/uapi/asm/vmx.h | 5 +- arch/x86/kvm/Kconfig | 7 +- arch/x86/kvm/Makefile | 3 +- arch/x86/kvm/cpuid.c | 27 +- arch/x86/kvm/cpuid.h | 2 + arch/x86/kvm/irq.c | 3 + arch/x86/kvm/lapic.c | 33 +- arch/x86/kvm/lapic.h | 2 + arch/x86/kvm/mmu.h | 31 + arch/x86/kvm/mmu/mmu.c | 200 +- arch/x86/kvm/mmu/mmu_internal.h | 109 +- arch/x86/kvm/mmu/page_track.c | 3 + arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/mmu/spte.c | 17 +- arch/x86/kvm/mmu/spte.h | 27 +- arch/x86/kvm/mmu/tdp_iter.h | 14 +- arch/x86/kvm/mmu/tdp_mmu.c | 442 ++- arch/x86/kvm/mmu/tdp_mmu.h | 7 +- arch/x86/kvm/smm.h | 7 +- arch/x86/kvm/svm/svm.c | 8 + arch/x86/kvm/vmx/common.h | 166 + arch/x86/kvm/vmx/main.c | 1246 ++++++++ arch/x86/kvm/vmx/pmu_intel.c | 46 +- arch/x86/kvm/vmx/pmu_intel.h | 28 + arch/x86/kvm/vmx/posted_intr.c | 43 +- arch/x86/kvm/vmx/posted_intr.h | 13 + arch/x86/kvm/vmx/tdx.c | 3321 ++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 266 ++ arch/x86/kvm/vmx/tdx_arch.h | 277 ++ arch/x86/kvm/vmx/tdx_errno.h | 44 + arch/x86/kvm/vmx/tdx_error.c | 21 + arch/x86/kvm/vmx/tdx_ops.h | 408 +++ arch/x86/kvm/vmx/vmcs.h | 5 + arch/x86/kvm/vmx/vmx.c | 661 ++-- arch/x86/kvm/vmx/vmx.h | 52 +- arch/x86/kvm/vmx/x86_ops.h | 257 ++ arch/x86/kvm/x86.c | 129 +- arch/x86/kvm/x86.h | 4 + arch/x86/virt/vmx/tdx/seamcall.S | 4 + arch/x86/virt/vmx/tdx/tdx.c | 11 +- include/linux/kvm_host.h | 1 + include/linux/kvm_types.h | 1 + include/uapi/linux/kvm.h | 89 + virt/kvm/kvm_main.c | 31 +- 56 files changed, 8407 insertions(+), 702 deletions(-) create mode 100644 Documentation/virt/kvm/x86/intel-tdx.rst create mode 100644 Documentation/virt/kvm/x86/tdx-tdp-mmu.rst create mode 100644 arch/x86/kvm/vmx/common.h create mode 100644 arch/x86/kvm/vmx/main.c create mode 100644 arch/x86/kvm/vmx/pmu_intel.h create mode 100644 arch/x86/kvm/vmx/tdx.c create mode 100644 arch/x86/kvm/vmx/tdx.h create mode 100644 arch/x86/kvm/vmx/tdx_arch.h create mode 100644 arch/x86/kvm/vmx/tdx_errno.h create mode 100644 arch/x86/kvm/vmx/tdx_error.c create mode 100644 arch/x86/kvm/vmx/tdx_ops.h create mode 100644 arch/x86/kvm/vmx/x86_ops.hbase-commit:
6613476e225e090cc9aad49be7fa504e290dd33d
prerequisite-patch-id:5e29e9200c65fb7f6213b4aa85254f128a4fc49f
prerequisite-patch-id:39908082e873a3828568cc6c626e734d4ccb279a
prerequisite-patch-id:01d54029211a041370ee12d58825c42f2255d3f8
prerequisite-patch-id:ec2e5dc132d37f2ec76f56172fe82e0f30998a50
prerequisite-patch-id:b9438c767dbd45f4dda5287e104a54fe8f3c516f
prerequisite-patch-id:ef4b52b28c7459a217b5fc4bd4dc592354fb9a46
prerequisite-patch-id:bc425a71343a9fed0ad12e5b85a1c614f8eff934
prerequisite-patch-id:cb538a88ed4c5d7614d3e6fcd10b991cce605a0d
prerequisite-patch-id:6019835e0581c2749e8e353f2045a2e94040a2b0
prerequisite-patch-id:4dd00540050377ff852c0a939682d5894513444c--
2.25.1