KVM TDX: TDP MMU: large page support
Subsystems (4)
v3 cover KVM TDX: TDP MMU: large page support
From: Isaku Yamahata <[email protected]>
Changes from v2:
- implemented page merging path
- rebased to TDX KVM v11
Changes from v1:
- implemented page merging path
- rebased to UPM v10
- rebased to TDX KVM v10
- rebased to kvm.git queue + v6.1-rc8
---
This patch series is based on "v11 KVM TDX: basic feature support". It implements large page support for TDP MMU by allowing populating of the large page and splitting it when necessary.
Feedback for options to merge sub-pages into a large page are welcome.
Remaining TODOs
===============
* Make nx recovery thread useTDH.MEM.RANGE.BLOCK instead of zapping EPT entry.
* Record that the entry is blocked by introducing a bit in spte. On EPT violation, check if the entry is blocked or not. If the EPT violation is caused by a blocked Secure-EPT entry, trigger the page merge logic.
Splitting large pages when necessary
====================================
* It already tracking whether GFN is private or shared. When it's changed, update lpage_info to prevent a large page.
* TDX provides page level on Secure EPT violation. Pass around the page level that the lower level functions needs.
* Even if the page is the large page in the host, at the EPT level, only some sub-pages are mapped. In such cases abandon to map large pages and step into the sub-page level, unlike the conventional EPT.
* When zapping spte and the spte is for a large page, split and zap it unlike the conventional EPT because otherwise the protected page contents will be lost.
Merging small pages into a large page if possible
=================================================
On normal EPT violation, check whether pages can be merged into a large page after mapping it.
TDX operation
=============
The following describes what TDX operations procedures.
* EPT violation trick
Such track (zapping the EPT entry to trigger EPT violation) doesn't work for TDX. For TDX, it will lose the contents of the protected page to zap a page because the protected guest page is un-associated from the guest TD. Instead, TDX provides a different way to trigger EPT violation without losing the page contents so that VMM can detect guest TD activity by blocking/unblocking Secure-EPT entry. TDH.MEM.RANGE.BLOCK and TDH.MEM.RANGE.UNBLOCK. They correspond to clearing/setting a present bit in an EPT entry with page contents still kept. By TDH.MEM.RANGE.BLOCK and TLB shoot down, VMM can cause guest TD to trigger EPT violation. After that, VMM can unblock it by TDH.MEM.RANGE.UNBLOCK and resume guest TD execution. The procedure is as follows.
- Block Secure-EPT entry by TDH.MEM.RANGE.BLOCK.
- TLB shoot down.
- Wait for guest TD to trigger EPT violation.
- Unblock Secure-EPT entry by TDH.MEM.RANGE.UNBLOCK to resume the guest TD.
* merging sub-pages into a large page
The following steps are needed.
- Ensure that all sub-pages are mapped.
- TLB shoot down.
- Merge sub-pages into a large page (TDH.MEM.PAGE.PROMOTE). This requires all sub-pages are mapped.
- Cache flush Secure EPT page used to map subpages.
Thanks,
Isaku Yamahata (4):
KVM: x86/tdp_mmu: Allocate private page table for large page split
KVM: x86/tdp_mmu: Try to merge pages into a large page
KVM: x86/tdp_mmu: TDX: Implement merge pages into a large page
KVM: x86/mmu: Make kvm fault handelr aware of large page of private
memslot
Xiaoyao Li (12):
KVM: TDP_MMU: Go to next level if smaller private mapping exists
KVM: TDX: Pass page level to cache flush before TDX SEAMCALL
KVM: TDX: Pass KVM page level to tdh_mem_page_add() and
tdh_mem_page_aug()
KVM: TDX: Pass size to tdx_measure_page()
KVM: TDX: Pass size to reclaim_page()
KVM: TDX: Update tdx_sept_{set,drop}_private_spte() to support large
page
KVM: MMU: Introduce level info in PFERR code
KVM: TDX: Pin pages via get_page() right before ADD/AUG'ed to TDs
KVM: TDX: Pass desired page level in err code for page fault handler
KVM: x86/tdp_mmu: Split the large page when zap leaf
KVM: x86/tdp_mmu, TDX: Split a large page when 4KB page within it
converted to shared
KVM: TDX: Allow 2MB large page for TD GUEST
arch/x86/include/asm/kvm-x86-ops.h | 3 + arch/x86/include/asm/kvm_host.h | 10 ++ arch/x86/kvm/mmu/mmu.c | 50 ++++-- arch/x86/kvm/mmu/mmu_internal.h | 24 +++ arch/x86/kvm/mmu/tdp_iter.c | 37 +++-- arch/x86/kvm/mmu/tdp_iter.h | 2 + arch/x86/kvm/mmu/tdp_mmu.c | 252 ++++++++++++++++++++++++++--- arch/x86/kvm/vmx/common.h | 6 +- arch/x86/kvm/vmx/tdx.c | 226 ++++++++++++++++++++------ arch/x86/kvm/vmx/tdx_arch.h | 21 +++ arch/x86/kvm/vmx/tdx_errno.h | 2 + arch/x86/kvm/vmx/tdx_ops.h | 50 ++++-- arch/x86/kvm/vmx/vmx.c | 2 +- 13 files changed, 572 insertions(+), 113 deletions(-)
Signature · 1 line
2.25.1
RFC PATCH v3 1/16 KVM: TDP_MMU: Go to next level if smaller private mapping exists
From: Xiaoyao Li <[email protected]>
Cannot map a private page as large page if any smaller mapping exists.
It has to wait for all the not-mapped smaller page to be mapped and promote it to larger mapping.
--- arch/x86/kvm/mmu/tdp_mmu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/tdp_mmu.c +2 -1
@@ -1394,7 +1394,8 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)tdp_mmu_for_each_pte (iter, mmu, is_private, raw_gfn, raw_gfn +1 ) {int r; -if (fault->nx_huge_page_workaround_enabled) +if (fault->nx_huge_page_workaround_enabled || +kvm_gfn_shared_mask (vcpu->kvm))disallowed_hugepage_adjust (fault, iter.old_spte, iter.level);/*
Signature · 1 line
2.25.1
RFC PATCH v3 2/16 KVM: TDX: Pass page level to cache flush before TDX SEAMCALL
In reply to [email protected]From: Xiaoyao Li <[email protected]>
tdh_mem_page_aug() will support 2MB large page in the near future. Cache flush also needs to be 2MB instead of 4KB in such cases. Introduce a helper function to flush cache with page size info in preparation for large pages.
--- arch/x86/kvm/vmx/tdx_ops.h | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
arch/x86/kvm/vmx/tdx_ops.h +14 -8
@@ -6,6 +6,7 @@#include <linux/compiler.h> +#include <asm/pgtable_types.h> #include <asm/cacheflush.h> #include <asm/asm.h> #include <asm/kvm_host.h>
@@ -18,6 +19,11 @@void pr_tdx_error (u64 op,u64 error_code,const struct tdx_module_output *out); +static inline void tdx_clflush_page (hpa_t addr,enum pg_level level) +{ +clflush_cache_range (__va (addr),KVM_HPAGE_SIZE (level)); +} +/* * TDX module acquires its internal lock for resources. It doesn't spin to get * locks because of its restrictions of allowed execution time. Instead, it
@@ -50,21 +56,21 @@ static inline u64 seamcall_sept(u64 op, u64 rcx, u64 rdx, u64 r8, u64 r9,static inline u64 tdh_mng_addcx (hpa_t tdr,hpa_t addr) { -clflush_cache_range (__va (addr),PAGE_SIZE ); +tdx_clflush_page (addr,PG_LEVEL_4K );return __seamcall (TDH_MNG_ADDCX , addr, tdr,0 ,0 ,NULL ); }static inline u64 tdh_mem_page_add (hpa_t tdr,gpa_t gpa,hpa_t hpa,hpa_t source,struct tdx_module_output *out) { -clflush_cache_range (__va (hpa),PAGE_SIZE ); +tdx_clflush_page (hpa,PG_LEVEL_4K );return seamcall_sept (TDH_MEM_PAGE_ADD , gpa, tdr, hpa, source, out); }static inline u64 tdh_mem_sept_add (hpa_t tdr,gpa_t gpa,int level,hpa_t page,struct tdx_module_output *out) { -clflush_cache_range (__va (page),PAGE_SIZE ); +tdx_clflush_page (page,PG_LEVEL_4K );return seamcall_sept (TDH_MEM_SEPT_ADD , gpa | level, tdr, page,0 , out); }
@@ -76,21 +82,21 @@ static inline u64 tdh_mem_sept_remove(hpa_t tdr, gpa_t gpa, int level,static inline u64 tdh_vp_addcx (hpa_t tdvpr,hpa_t addr) { -clflush_cache_range (__va (addr),PAGE_SIZE ); +tdx_clflush_page (addr,PG_LEVEL_4K );return __seamcall (TDH_VP_ADDCX , addr, tdvpr,0 ,0 ,NULL ); }static inline u64 tdh_mem_page_relocate (hpa_t tdr,gpa_t gpa,hpa_t hpa,struct tdx_module_output *out) { -clflush_cache_range (__va (hpa),PAGE_SIZE ); +tdx_clflush_page (hpa,PG_LEVEL_4K );return __seamcall (TDH_MEM_PAGE_RELOCATE , gpa, tdr, hpa,0 , out); }static inline u64 tdh_mem_page_aug (hpa_t tdr,gpa_t gpa,hpa_t hpa,struct tdx_module_output *out) { -clflush_cache_range (__va (hpa),PAGE_SIZE ); +tdx_clflush_page (hpa,PG_LEVEL_4K );return seamcall_sept (TDH_MEM_PAGE_AUG , gpa, tdr, hpa,0 , out); }
@@ -107,13 +113,13 @@ static inline u64 tdh_mng_key_config(hpa_t tdr)static inline u64 tdh_mng_create (hpa_t tdr,int hkid) { -clflush_cache_range (__va (tdr),PAGE_SIZE ); +tdx_clflush_page (tdr,PG_LEVEL_4K );return __seamcall (TDH_MNG_CREATE , tdr, hkid,0 ,0 ,NULL ); }static inline u64 tdh_vp_create (hpa_t tdr,hpa_t tdvpr) { -clflush_cache_range (__va (tdvpr),PAGE_SIZE ); +tdx_clflush_page (tdvpr,PG_LEVEL_4K );return __seamcall (TDH_VP_CREATE , tdvpr, tdr,0 ,0 ,NULL ); }
Signature · 1 line
2.25.1
RFC PATCH v3 3/16 KVM: TDX: Pass KVM page level to tdh_mem_page_add() and tdh_mem_page_aug()
In reply to [email protected]From: Xiaoyao Li <[email protected]>
Level info is needed in tdh_clflush_page() to generate the correct page size.
Besides, explicitly pass level info to SEAMCALL instead of assuming it's zero. It works naturally when 2MB support lands.
--- arch/x86/kvm/vmx/tdx.c | 7 ++++--- arch/x86/kvm/vmx/tdx_ops.h | 19 ++++++++++++------- 2 files changed, 16 insertions(+), 10 deletions(-)
arch/x86/kvm/vmx/tdx.c +4 -3
@@ -1324,6 +1324,7 @@ static void tdx_unpin(struct kvm *kvm, kvm_pfn_t pfn)static int tdx_sept_set_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level,kvm_pfn_t pfn) { +int tdx_level =pg_level_to_tdx_sept_level (level);struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);hpa_t hpa =pfn_to_hpa (pfn);gpa_t gpa =gfn_to_gpa (gfn);
@@ -1348,7 +1349,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm))return -EINVAL ; - err =tdh_mem_page_aug (kvm_tdx->tdr_pa, gpa, hpa, &out); + err =tdh_mem_page_aug (kvm_tdx->tdr_pa, gpa, tdx_level, hpa, &out);if (err ==TDX_ERROR_SEPT_BUSY ) {tdx_unpin (kvm, pfn);return -EAGAIN ;
@@ -1387,8 +1388,8 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, kvm_tdx->source_pa =INVALID_PAGE ; do { - err = tdh_mem_page_add(kvm_tdx->tdr_pa, gpa, hpa, source_pa, - &out); + err = tdh_mem_page_add(kvm_tdx->tdr_pa, gpa, tdx_level, hpa, + source_pa, &out); /* * This path is executed during populating initial guest memory * image. i.e. before running any vcpu. Race is rare.
arch/x86/kvm/vmx/tdx_ops.h +12 -7
@@ -19,6 +19,11 @@void pr_tdx_error (u64 op,u64 error_code,const struct tdx_module_output *out); +static inline enum pg_level tdx_sept_level_to_pg_level (int tdx_level) +{ +return tdx_level +1 ; +} +static inline void tdx_clflush_page (hpa_t addr,enum pg_level level) {clflush_cache_range (__va (addr),KVM_HPAGE_SIZE (level));
@@ -60,11 +65,11 @@ static inline u64 tdh_mng_addcx(hpa_t tdr, hpa_t addr)return __seamcall (TDH_MNG_ADDCX , addr, tdr,0 ,0 ,NULL ); } -static inline u64 tdh_mem_page_add (hpa_t tdr,gpa_t gpa,hpa_t hpa,hpa_t source, -struct tdx_module_output *out) +static inline u64 tdh_mem_page_add (hpa_t tdr,gpa_t gpa,int level,hpa_t hpa, +hpa_t source,struct tdx_module_output *out) { -tdx_clflush_page (hpa,PG_LEVEL_4K ); -return seamcall_sept (TDH_MEM_PAGE_ADD , gpa, tdr, hpa, source, out); +tdx_clflush_page (hpa,tdx_sept_level_to_pg_level (level)); +return seamcall_sept (TDH_MEM_PAGE_ADD , gpa | level, tdr, hpa, source, out); } static inline u64 tdh_mem_sept_add(hpa_t tdr, gpa_t gpa, int level, hpa_t page,
@@ -93,11 +98,11 @@ static inline u64 tdh_mem_page_relocate(hpa_t tdr, gpa_t gpa, hpa_t hpa,return __seamcall (TDH_MEM_PAGE_RELOCATE , gpa, tdr, hpa,0 , out); } -static inline u64 tdh_mem_page_aug (hpa_t tdr,gpa_t gpa,hpa_t hpa, +static inline u64 tdh_mem_page_aug (hpa_t tdr,gpa_t gpa,int level,hpa_t hpa,struct tdx_module_output *out) { -tdx_clflush_page (hpa,PG_LEVEL_4K ); -return seamcall_sept (TDH_MEM_PAGE_AUG , gpa, tdr, hpa,0 , out); +tdx_clflush_page (hpa,tdx_sept_level_to_pg_level (level)); +return seamcall_sept (TDH_MEM_PAGE_AUG , gpa | level, tdr, hpa,0 , out); } static inline u64 tdh_mem_range_block(hpa_t tdr, gpa_t gpa, int level,
Signature · 1 line
2.25.1
RFC PATCH v3 4/16 KVM: TDX: Pass size to tdx_measure_page()
In reply to [email protected]From: Xiaoyao Li <[email protected]>
Extend tdx_measure_page() to pass size info so that it can measure large page as well.
--- arch/x86/kvm/vmx/tdx.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/tdx.c +5 -3
@@ -1299,13 +1299,15 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK ); } -static void tdx_measure_page (struct kvm_tdx *kvm_tdx,hpa_t gpa) +static void tdx_measure_page (struct kvm_tdx *kvm_tdx,hpa_t gpa,int size) {struct tdx_module_output out;u64 err;int i; -for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) { +WARN_ON_ONCE (size %TDX_EXTENDMR_CHUNKSIZE ); + +for (i =0 ; i < size; i +=TDX_EXTENDMR_CHUNKSIZE ) { err =tdh_mr_extend (kvm_tdx->tdr_pa, gpa + i, &out);if (KVM_BUG_ON (err, &kvm_tdx->kvm)) {pr_tdx_error (TDH_MR_EXTEND , err, &out);
@@ -1400,7 +1402,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,tdx_unpin (kvm, pfn);return -EIO ; }else if (measure) -tdx_measure_page (kvm_tdx, gpa); +tdx_measure_page (kvm_tdx, gpa,KVM_HPAGE_SIZE (level));return 0 ; }
Signature · 1 line
2.25.1
RFC PATCH v3 5/16 KVM: TDX: Pass size to reclaim_page()
In reply to [email protected]From: Xiaoyao Li <[email protected]>
A 2MB large page can be tdh_mem_page_aug()'ed to TD directly. In this case, it needs to reclaim and clear the page as 2MB size.
--- arch/x86/kvm/vmx/tdx.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)
arch/x86/kvm/vmx/tdx.c +16 -10
@@ -184,14 +184,17 @@ void tdx_hardware_disable(void)tdx_disassociate_vp (&tdx->vcpu); } -static void tdx_clear_page (unsigned long page_pa) +static void tdx_clear_page (unsigned long page_pa,int size) {const void *zero_page = (const void *)__va (page_to_phys (ZERO_PAGE (0 )));void *page =__va (page_pa);unsigned long i; +WARN_ON_ONCE (size %PAGE_SIZE ); +if (!static_cpu_has (X86_FEATURE_MOVDIR64B )) { -clear_page (page); +for (i =0 ; i < size; i +=PAGE_SIZE ) +clear_page (page + i);return ; }
@@ -205,7 +208,7 @@ static void tdx_clear_page(unsigned long page_pa)* The cache line could be poisoned (even without MKTME-i), clear the * poison bit. */ -for (i =0 ; i <PAGE_SIZE ; i +=64 ) +for (i =0 ; i < size; i +=64 )movdir64b (page + i, zero_page);/* * MOVDIR64B store uses WC buffer. Prevent following memory reads
@@ -214,7 +217,8 @@ static void tdx_clear_page(unsigned long page_pa)__mb (); } -static int tdx_reclaim_page (hpa_t pa,bool do_wb,u16 hkid) +static int tdx_reclaim_page (hpa_t pa,enum pg_level level, +bool do_wb,u16 hkid) {struct tdx_module_output out;u64 err;
@@ -232,8 +236,10 @@ static int tdx_reclaim_page(hpa_t pa, bool do_wb, u16 hkid)pr_tdx_error (TDH_PHYMEM_PAGE_RECLAIM , err, &out);return -EIO ; } +/* out.r8 == tdx sept page level */ +WARN_ON_ONCE (out.r8 !=pg_level_to_tdx_sept_level (level)); -if (do_wb) { +if (do_wb && level ==PG_LEVEL_4K ) {/* * Only TDR page gets into this path. No contention is expected * because of the last page of TD.
@@ -245,7 +251,7 @@ static int tdx_reclaim_page(hpa_t pa, bool do_wb, u16 hkid) } } -tdx_clear_page (pa); +tdx_clear_page (pa,KVM_HPAGE_SIZE (level));return 0 ; }
@@ -259,7 +265,7 @@ static void tdx_reclaim_td_page(unsigned long td_page_pa)* was already flushed by TDH.PHYMEM.CACHE.WB before here, So * cache doesn't need to be flushed again. */ -if (WARN_ON (tdx_reclaim_page (td_page_pa, false,0 ))) +if (WARN_ON (tdx_reclaim_page (td_page_pa,PG_LEVEL_4K , false,0 )))/* If reclaim failed, leak the page. */ return ;free_page ((unsigned long )__va (td_page_pa));
@@ -436,7 +442,7 @@ void tdx_vm_free(struct kvm *kvm)* while operating on TD (Especially reclaiming TDCS). Cache flush with * TDX global HKID is needed. */ -if (tdx_reclaim_page (kvm_tdx->tdr_pa, true, tdx_global_keyid)) +if (tdx_reclaim_page (kvm_tdx->tdr_pa,PG_LEVEL_4K , true, tdx_global_keyid))return ;free_page ((unsigned long )__va (kvm_tdx->tdr_pa));
@@ -1427,7 +1433,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,* The HKID assigned to this TD was already freed and cache * was already flushed. We don't have to flush again. */ - err =tdx_reclaim_page (hpa, false,0 ); + err =tdx_reclaim_page (hpa, level, false,0 );if (KVM_BUG_ON (err, kvm))return -EIO ;tdx_unpin (kvm, pfn);
@@ -1566,7 +1572,7 @@ static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn,* already flushed. We don't have to flush again. */ if (!is_hkid_assigned (kvm_tdx)) -return tdx_reclaim_page (__pa (private_spt), false,0 ); +return tdx_reclaim_page (__pa (private_spt),PG_LEVEL_4K , false,0 );/* * free_private_spt() is (obviously) called when a shadow page is being
Signature · 1 line
2.25.1
RFC PATCH v3 6/16 KVM: TDX: Update tdx_sept_{set,drop}_private_spte() to support large page
In reply to [email protected]From: Xiaoyao Li <[email protected]>
Allow large page level AUG and REMOVE for TDX pages.
--- arch/x86/kvm/vmx/tdx.c | 63 +++++++++++++++++++++--------------------- 1 file changed, 32 insertions(+), 31 deletions(-)
arch/x86/kvm/vmx/tdx.c +32 -31
@@ -1322,11 +1322,12 @@ static void tdx_measure_page(struct kvm_tdx *kvm_tdx, hpa_t gpa, int size) } } -static void tdx_unpin (struct kvm *kvm,kvm_pfn_t pfn) +static void tdx_unpin (struct kvm *kvm,kvm_pfn_t pfn,int level) { -struct page *page =pfn_to_page (pfn); +int i; -put_page (page); +for (i =0 ; i <KVM_PAGES_PER_HPAGE (level); i++) +put_page (pfn_to_page (pfn + i)); } static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,
@@ -1340,6 +1341,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,hpa_t source_pa;bool measure;u64 err; +int i;/* * Because restricted mem doesn't support page migration with
@@ -1349,22 +1351,19 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,* TODO: Once restricted mem introduces callback on page migration, * implement it and remove get_page/put_page(). */ -get_page (pfn_to_page (pfn)); +for (i =0 ; i <KVM_PAGES_PER_HPAGE (level); i++) +get_page (pfn_to_page (pfn + i));/* Build-time faults are induced and handled via TDH_MEM_PAGE_ADD. */ if (likely (is_td_finalized (kvm_tdx))) { -/* TODO: handle large pages. */ -if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) -return -EINVAL ; - err =tdh_mem_page_aug (kvm_tdx->tdr_pa, gpa, tdx_level, hpa, &out);if (err ==TDX_ERROR_SEPT_BUSY ) { -tdx_unpin (kvm, pfn); +tdx_unpin (kvm, pfn, level);return -EAGAIN ; }if (KVM_BUG_ON (err, kvm)) {pr_tdx_error (TDH_MEM_PAGE_AUG , err, &out); -tdx_unpin (kvm, pfn); +tdx_unpin (kvm, pfn, level);return -EIO ; }return 0 ;
@@ -1387,7 +1386,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,* always uses vcpu 0's page table and protected by vcpu->mutex). */ if (KVM_BUG_ON (kvm_tdx->source_pa ==INVALID_PAGE , kvm)) { -tdx_unpin (kvm, pfn); +tdx_unpin (kvm, pfn, level);return -EINVAL ; }
@@ -1405,7 +1404,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, }while (err ==TDX_ERROR_SEPT_BUSY );if (KVM_BUG_ON (err, kvm)) {pr_tdx_error (TDH_MEM_PAGE_ADD , err, &out); -tdx_unpin (kvm, pfn); +tdx_unpin (kvm, pfn, level);return -EIO ; }else if (measure)tdx_measure_page (kvm_tdx, gpa,KVM_HPAGE_SIZE (level));
@@ -1422,11 +1421,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,gpa_t gpa =gfn_to_gpa (gfn);hpa_t hpa =pfn_to_hpa (pfn);hpa_t hpa_with_hkid; +int r =0 ;u64 err; - -/* TODO: handle large pages. */ -if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) -return -EINVAL ; +int i;if (!is_hkid_assigned (kvm_tdx)) {/*
@@ -1436,7 +1433,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, err =tdx_reclaim_page (hpa, level, false,0 );if (KVM_BUG_ON (err, kvm))return -EIO ; -tdx_unpin (kvm, pfn); +tdx_unpin (kvm, pfn, level);return 0 ; }
@@ -1453,21 +1450,25 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ; } - hpa_with_hkid =set_hkid_to_hpa (hpa, (u16 )kvm_tdx->hkid); -do { -/* -* TDX_OPERAND_BUSY can happen on locking PAMT entry. Because -* this page was removed above, other thread shouldn't be -* repeatedly operating on this page. Just retry loop. -*/ - err =tdh_phymem_page_wbinvd (hpa_with_hkid); - }while (err == (TDX_OPERAND_BUSY |TDX_OPERAND_ID_RCX )); -if (KVM_BUG_ON (err, kvm)) { -pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err,NULL ); -return -EIO ; +for (i =0 ; i <KVM_PAGES_PER_HPAGE (level); i++) { + hpa_with_hkid =set_hkid_to_hpa (hpa, (u16 )kvm_tdx->hkid); +do { +/* +* TDX_OPERAND_BUSY can happen on locking PAMT entry. +* Because this page was removed above, other thread +* shouldn't be repeatedly operating on this page. +* Simple retry should work. +*/ + err =tdh_phymem_page_wbinvd (hpa_with_hkid); + }while (err == (TDX_OPERAND_BUSY |TDX_OPERAND_ID_RCX )); +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err,NULL ); + r = -EIO ; + }else +tdx_unpin (kvm, pfn + i,PG_LEVEL_4K ); + hpa +=PAGE_SIZE ; } -tdx_unpin (kvm, pfn); -return 0 ; +return r; } static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,
Signature · 1 line
2.25.1
RFC PATCH v3 7/16 KVM: MMU: Introduce level info in PFERR code
In reply to [email protected]From: Xiaoyao Li <[email protected]>
For TDX, EPT violation can happen when TDG.MEM.PAGE.ACCEPT.
And TDG.MEM.PAGE.ACCEPT contains the desired accept page level of TD guest.
1. KVM can map it with 4KB page while TD guest wants to accept 2MB page.
TD geust will get TDX_PAGE_SIZE_MISMATCH and it should try to accept 4KB size.
2. KVM can map it with 2MB page while TD guest wants to accept 4KB page.
KVM needs to honor it because
a) there is no way to tell guest KVM maps it as 2MB size. And
b) guest accepts it in 4KB size since guest knows some other 4KB page in the same 2MB range will be used as shared page.
For case 2, it need to pass desired page level to MMU's page_fault_handler. Use bit 29:31 of kvm PF error code for this purpose.
--- arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 5 +++++ arch/x86/kvm/vmx/common.h | 6 +++++- arch/x86/kvm/vmx/tdx.c | 15 ++++++++++++++- arch/x86/kvm/vmx/tdx.h | 19 +++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 2 +- 6 files changed, 47 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -257,6 +257,8 @@ enum x86_intercept_stage;#define PFERR_FETCH_BIT 4#define PFERR_PK_BIT 5#define PFERR_SGX_BIT 15 +#define PFERR_LEVEL_START_BIT 29 +#define PFERR_LEVEL_END_BIT 31#define PFERR_GUEST_FINAL_BIT 32#define PFERR_GUEST_PAGE_BIT 33#define PFERR_IMPLICIT_ACCESS_BIT 48
@@ -268,6 +270,7 @@ enum x86_intercept_stage;#define PFERR_FETCH_MASK BIT(PFERR_FETCH_BIT)#define PFERR_PK_MASK BIT(PFERR_PK_BIT)#define PFERR_SGX_MASK BIT(PFERR_SGX_BIT) +#define PFERR_LEVEL_MASK GENMASK_ULL(PFERR_LEVEL_END_BIT, PFERR_LEVEL_START_BIT)#define PFERR_GUEST_FINAL_MASK BIT_ULL(PFERR_GUEST_FINAL_BIT)#define PFERR_GUEST_PAGE_MASK BIT_ULL(PFERR_GUEST_PAGE_BIT)#define PFERR_IMPLICIT_ACCESS BIT_ULL(PFERR_IMPLICIT_ACCESS_BIT)
arch/x86/kvm/mmu/mmu.c +5 -0
@@ -4503,6 +4503,11 @@ EXPORT_SYMBOL_GPL(kvm_handle_page_fault);int kvm_tdp_page_fault (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) { +u8 err_level = (fault->error_code &PFERR_LEVEL_MASK ) >>PFERR_LEVEL_START_BIT ; + +if (err_level) + fault->max_level =min (fault->max_level, err_level); +/* * If the guest's MTRRs may be used to compute the "real" memtype, * restrict the mapping level to ensure KVM uses a consistent memtype
arch/x86/kvm/vmx/common.h +5 -1
@@ -78,7 +78,8 @@ static inline void vmx_handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu, }static inline int __vmx_handle_ept_violation (struct kvm_vcpu *vcpu,gpa_t gpa, -unsigned long exit_qualification) +unsigned long exit_qualification, +int err_page_level) {u64 error_code;
@@ -98,6 +99,9 @@ static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t gpa, error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ?PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; +if (err_page_level >0 ) + error_code |= (err_page_level <<PFERR_LEVEL_START_BIT ) &PFERR_LEVEL_MASK ; +return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); }
arch/x86/kvm/vmx/tdx.c +14 -1
@@ -1634,7 +1634,20 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,static int tdx_handle_ept_violation (struct kvm_vcpu *vcpu) { +union tdx_ext_exit_qualification ext_exit_qual;unsigned long exit_qual; +int err_page_level =0 ; + + ext_exit_qual.full =tdexit_ext_exit_qual (vcpu); + +if (ext_exit_qual.type >=NUM_EXT_EXIT_QUAL ) { +pr_err ("EPT violation at gpa 0x%lx, with invalid ext exit qualification type 0x%x\n" , +tdexit_gpa (vcpu), ext_exit_qual.type); +kvm_vm_bugged (vcpu->kvm); +return 0 ; + }else if (ext_exit_qual.type ==EXT_EXIT_QUAL_ACCEPT ) { + err_page_level = ext_exit_qual.req_sept_level +1 ; + }if (kvm_is_private_gpa (vcpu->kvm,tdexit_gpa (vcpu))) {/*
@@ -1661,7 +1674,7 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu) }trace_kvm_page_fault (vcpu,tdexit_gpa (vcpu), exit_qual); -return __vmx_handle_ept_violation (vcpu,tdexit_gpa (vcpu), exit_qual); +return __vmx_handle_ept_violation (vcpu,tdexit_gpa (vcpu), exit_qual, err_page_level); }static int tdx_handle_ept_misconfig (struct kvm_vcpu *vcpu)
arch/x86/kvm/vmx/tdx.h +19 -0
@@ -57,6 +57,25 @@ union tdx_exit_reason {u64 full; }; +union tdx_ext_exit_qualification { +struct { +u64 type :4 ; +u64 reserved0 :28 ; +u64 req_sept_level :3 ; +u64 err_sept_level :3 ; +u64 err_sept_state :8 ; +u64 err_sept_is_leaf :1 ; +u64 reserved1 :17 ; + }; +u64 full; +}; + +enum tdx_ext_exit_qualification_type { +EXT_EXIT_QUAL_NONE , +EXT_EXIT_QUAL_ACCEPT , +NUM_EXT_EXIT_QUAL , +}; +struct vcpu_tdx {struct kvm_vcpu vcpu;
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -5791,7 +5791,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)if (unlikely (allow_smaller_maxphyaddr &&kvm_vcpu_is_illegal_gpa (vcpu, gpa)))return kvm_emulate_instruction (vcpu,0 ); -return __vmx_handle_ept_violation (vcpu, gpa, exit_qualification); +return __vmx_handle_ept_violation (vcpu, gpa, exit_qualification,0 ); }static int handle_ept_misconfig (struct kvm_vcpu *vcpu)
Signature · 1 line
2.25.1
RFC PATCH v3 8/16 KVM: TDX: Pin pages via get_page() right before ADD/AUG'ed to TDs
In reply to [email protected]From: Xiaoyao Li <[email protected]>
When kvm_faultin_pfn(), it doesn't have the info regarding which page level will the gfn be mapped at. Hence it doesn't know to pin a 4K page or a 2M page.
Move the guest private pages pinning logic right before TDH_MEM_PAGE_ADD/AUG() since at that time it knows the page level info.
--- arch/x86/kvm/vmx/tdx.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/tdx.c +8 -7
@@ -1322,7 +1322,8 @@ static void tdx_measure_page(struct kvm_tdx *kvm_tdx, hpa_t gpa, int size) } } -static void tdx_unpin (struct kvm *kvm,kvm_pfn_t pfn,int level) +static void tdx_unpin (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +enum pg_level level) {int i;
@@ -1358,12 +1359,12 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (likely (is_td_finalized (kvm_tdx))) { err =tdh_mem_page_aug (kvm_tdx->tdr_pa, gpa, tdx_level, hpa, &out);if (err ==TDX_ERROR_SEPT_BUSY ) { -tdx_unpin (kvm, pfn, level); +tdx_unpin (kvm, gfn, pfn, level);return -EAGAIN ; }if (KVM_BUG_ON (err, kvm)) {pr_tdx_error (TDH_MEM_PAGE_AUG , err, &out); -tdx_unpin (kvm, pfn, level); +tdx_unpin (kvm, gfn, pfn, level);return -EIO ; }return 0 ;
@@ -1386,7 +1387,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,* always uses vcpu 0's page table and protected by vcpu->mutex). */ if (KVM_BUG_ON (kvm_tdx->source_pa ==INVALID_PAGE , kvm)) { -tdx_unpin (kvm, pfn, level); +tdx_unpin (kvm, gfn, pfn, level);return -EINVAL ; }
@@ -1404,7 +1405,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, }while (err ==TDX_ERROR_SEPT_BUSY );if (KVM_BUG_ON (err, kvm)) {pr_tdx_error (TDH_MEM_PAGE_ADD , err, &out); -tdx_unpin (kvm, pfn, level); +tdx_unpin (kvm, gfn, pfn, level);return -EIO ; }else if (measure)tdx_measure_page (kvm_tdx, gpa,KVM_HPAGE_SIZE (level));
@@ -1433,7 +1434,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, err =tdx_reclaim_page (hpa, level, false,0 );if (KVM_BUG_ON (err, kvm))return -EIO ; -tdx_unpin (kvm, pfn, level); +tdx_unpin (kvm, gfn, pfn, level);return 0 ; }
@@ -1465,7 +1466,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err,NULL ); r = -EIO ; }else -tdx_unpin (kvm, pfn + i,PG_LEVEL_4K ); +tdx_unpin (kvm, gfn + i, pfn + i,PG_LEVEL_4K ); hpa +=PAGE_SIZE ; }return r;
Signature · 1 line
2.25.1
RFC PATCH v3 9/16 KVM: TDX: Pass desired page level in err code for page fault handler
In reply to [email protected]From: Xiaoyao Li <[email protected]>
For TDX, EPT violation can happen when TDG.MEM.PAGE.ACCEPT.
And TDG.MEM.PAGE.ACCEPT contains the desired accept page level of TD guest.
1. KVM can map it with 4KB page while TD guest wants to accept 2MB page.
TD geust will get TDX_PAGE_SIZE_MISMATCH and it should try to accept 4KB size.
2. KVM can map it with 2MB page while TD guest wants to accept 4KB page.
KVM needs to honor it because
a) there is no way to tell guest KVM maps it as 2MB size. And
b) guest accepts it in 4KB size since guest knows some other 4KB page in the same 2MB range will be used as shared page.
For case 2, it need to pass desired page level to MMU's page_fault_handler. Use bit 29:31 of kvm PF error code for this purpose.
--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/vmx/common.h | 2 +- arch/x86/kvm/vmx/tdx.c | 7 ++++++- arch/x86/kvm/vmx/tdx.h | 19 ------------------- arch/x86/kvm/vmx/tdx_arch.h | 19 +++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 2 +- 6 files changed, 29 insertions(+), 22 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -279,6 +279,8 @@ enum x86_intercept_stage; PFERR_WRITE_MASK | \ PFERR_PRESENT_MASK) +#define PFERR_LEVEL (err_code) (((err_code) & PFERR_LEVEL_MASK) >> PFERR_LEVEL_START_BIT) +/* apic attention bits */ #define KVM_APIC_CHECK_VAPIC 0/*
arch/x86/kvm/vmx/common.h +1 -1
@@ -99,7 +99,7 @@ static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t gpa, error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ?PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; -if (err_page_level >0 ) +if (err_page_level >PG_LEVEL_NONE ) error_code |= (err_page_level <<PFERR_LEVEL_START_BIT ) &PFERR_LEVEL_MASK ;return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 );
arch/x86/kvm/vmx/tdx.c +6 -1
@@ -2360,6 +2360,7 @@ static int tdx_init_mem_region(struct kvm *kvm, struct kvm_tdx_cmd *cmd)struct kvm_tdx_init_mem_region region;struct kvm_vcpu *vcpu;struct page *page; +u64 error_code;kvm_pfn_t pfn;int idx, ret =0 ;
@@ -2412,7 +2413,11 @@ static int tdx_init_mem_region(struct kvm *kvm, struct kvm_tdx_cmd *cmd) kvm_tdx->source_pa =pfn_to_hpa (page_to_pfn (page)) | (cmd->flags &KVM_TDX_MEASURE_MEMORY_REGION ); - pfn =kvm_mmu_map_tdp_page (vcpu, region.gpa,TDX_SEPT_PFERR , +/* TODO: large page support. */ + error_code =TDX_SEPT_PFERR ; + error_code |= (PG_LEVEL_4K <<PFERR_LEVEL_START_BIT ) & +PFERR_LEVEL_MASK ; + pfn =kvm_mmu_map_tdp_page (vcpu, region.gpa, error_code,PG_LEVEL_4K );if (is_error_noslot_pfn (pfn) || kvm->vm_bugged) ret = -EFAULT ;
arch/x86/kvm/vmx/tdx.h +0 -19
@@ -57,25 +57,6 @@ union tdx_exit_reason {u64 full; }; -union tdx_ext_exit_qualification { -struct { -u64 type :4 ; -u64 reserved0 :28 ; -u64 req_sept_level :3 ; -u64 err_sept_level :3 ; -u64 err_sept_state :8 ; -u64 err_sept_is_leaf :1 ; -u64 reserved1 :17 ; - }; -u64 full; -}; - -enum tdx_ext_exit_qualification_type { -EXT_EXIT_QUAL_NONE , -EXT_EXIT_QUAL_ACCEPT , -NUM_EXT_EXIT_QUAL , -}; -struct vcpu_tdx {struct kvm_vcpu vcpu;
arch/x86/kvm/vmx/tdx_arch.h +19 -0
@@ -163,4 +163,23 @@ struct td_params {#define TDX_MIN_TSC_FREQUENCY_KHZ (100 * 1000)#define TDX_MAX_TSC_FREQUENCY_KHZ (10 * 1000 * 1000) +union tdx_ext_exit_qualification { +struct { +u64 type :4 ; +u64 reserved0 :28 ; +u64 req_sept_level :3 ; +u64 err_sept_level :3 ; +u64 err_sept_state :8 ; +u64 err_sept_is_leaf :1 ; +u64 reserved1 :17 ; + }; +u64 full; +}; + +enum tdx_ext_exit_qualification_type { +EXT_EXIT_QUAL_NONE =0 , +EXT_EXIT_QUAL_ACCEPT , +NUM_EXT_EXIT_QUAL , +}; +#endif /* __KVM_X86_TDX_ARCH_H */
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -5791,7 +5791,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)if (unlikely (allow_smaller_maxphyaddr &&kvm_vcpu_is_illegal_gpa (vcpu, gpa)))return kvm_emulate_instruction (vcpu,0 ); -return __vmx_handle_ept_violation (vcpu, gpa, exit_qualification,0 ); +return __vmx_handle_ept_violation (vcpu, gpa, exit_qualification,PG_LEVEL_NONE ); }static int handle_ept_misconfig (struct kvm_vcpu *vcpu)
Signature · 1 line
2.25.1
RFC PATCH v3 10/16 KVM: x86/tdp_mmu: Allocate private page table for large page split
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Make tdp_mmu_alloc_sp_split() aware of private page table.
--- arch/x86/kvm/mmu/mmu_internal.h | 14 ++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 8 ++++++-- 2 files changed, 20 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu_internal.h +14 -0
@@ -201,6 +201,15 @@ static inline void kvm_mmu_alloc_private_spt(struct kvm_vcpu *vcpu, struct kvm_m } } +static inline int kvm_alloc_private_spt_for_split (struct kvm_mmu_page *sp,gfp_t gfp) +{ + gfp &= ~__GFP_ZERO; + sp->private_spt = (void *)__get_free_page (gfp); +if (!sp->private_spt) +return -ENOMEM ; +return 0 ; +} +static inline void kvm_mmu_free_private_spt (struct kvm_mmu_page *sp) {if (sp->private_spt)
@@ -229,6 +238,11 @@ static inline void kvm_mmu_alloc_private_spt(struct kvm_vcpu *vcpu, struct kvm_m { } +static inline int kvm_alloc_private_spt_for_split (struct kvm_mmu_page *sp,gfp_t gfp) +{ +return -ENOMEM ; +} +static inline void kvm_mmu_free_private_spt (struct kvm_mmu_page *sp) { }
arch/x86/kvm/mmu/tdp_mmu.c +6 -2
@@ -1690,8 +1690,12 @@ static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split(gfp_t gfp, union kvm_mm sp->role = role; sp->spt = (void *)__get_free_page (gfp); -/* TODO: large page support for private GPA. */ -WARN_ON_ONCE (kvm_mmu_page_role_is_private (role)); +if (kvm_mmu_page_role_is_private (role)) { +if (kvm_alloc_private_spt_for_split (sp, gfp)) { +free_page ((unsigned long )sp->spt); + sp->spt =NULL ; + } + }if (!sp->spt) {kmem_cache_free (mmu_page_header_cache, sp);return NULL ;
Signature · 1 line
2.25.1
RFC PATCH v3 11/16 KVM: x86/tdp_mmu: Split the large page when zap leaf
In reply to [email protected]From: Xiaoyao Li <[email protected]>
When TDX enabled, a large page cannot be zapped if it contains mixed pages. In this case, it has to split the large page.
--- arch/x86/kvm/mmu/mmu.c | 9 +++++ arch/x86/kvm/mmu/mmu_internal.h | 2 + arch/x86/kvm/mmu/tdp_mmu.c | 68 +++++++++++++++++++++++++++++++-- 3 files changed, 76 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +9 -0
@@ -7250,6 +7250,15 @@ static bool linfo_is_mixed(struct kvm_lpage_info *linfo)return linfo->disallow_lpage &KVM_LPAGE_PRIVATE_SHARED_MIXED ; } +bool kvm_mem_attr_is_mixed (struct kvm_memory_slot *slot,gfn_t gfn,int level) +{ +struct kvm_lpage_info *linfo =lpage_info_slot (gfn &KVM_HPAGE_MASK (level), + slot, level); + +WARN_ON_ONCE (level ==PG_LEVEL_4K ); +return linfo_is_mixed (linfo); +} +static void linfo_set_mixed (gfn_t gfn,struct kvm_memory_slot *slot,int level,bool mixed) {
arch/x86/kvm/mmu/mmu_internal.h +2 -0
@@ -434,6 +434,8 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);void track_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp);void untrack_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp); +bool kvm_mem_attr_is_mixed (struct kvm_memory_slot *slot,gfn_t gfn,int level); +#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot,gfn_t gfn,kvm_pfn_t *pfn,int *order)
arch/x86/kvm/mmu/tdp_mmu.c +65 -3
104 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC PATCH v3 12/16 KVM: x86/tdp_mmu, TDX: Split a large page when 4KB page within it converted to shared
In reply to [email protected]From: Xiaoyao Li <[email protected]>
When mapping the shared page for TDX, it needs to zap private alias.
In the case that private page is mapped as large page (2MB), it can be removed directly only when the whole 2MB is converted to shared. Otherwise, it has to split 2MB page into 512 4KB page, and only remove the pages that converted to shared.
When a present large leaf spte switches to present non-leaf spte, TDX needs to split the corresponding SEPT page to reflect it.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/mmu/tdp_mmu.c | 24 +++++++++++++++--------- arch/x86/kvm/vmx/tdx.c | 25 +++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx_arch.h | 1 + arch/x86/kvm/vmx/tdx_ops.h | 7 +++++++ 6 files changed, 49 insertions(+), 11 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -97,6 +97,7 @@ KVM_X86_OP_OPTIONAL_RET0(get_mt_mask) KVM_X86_OP(load_mmu_pgd) KVM_X86_OP_OPTIONAL(link_private_spt) KVM_X86_OP_OPTIONAL(free_private_spt) +KVM_X86_OP_OPTIONAL(split_private_spt) KVM_X86_OP_OPTIONAL(set_private_spte) KVM_X86_OP_OPTIONAL(remove_private_spte) KVM_X86_OP_OPTIONAL(zap_private_spte)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1695,6 +1695,8 @@ struct kvm_x86_ops { void *private_spt);int (*free_private_spt)(struct kvm *kvm,gfn_t gfn,enum pg_level level,void *private_spt); +int (*split_private_spt)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +void *private_spt);int (*set_private_spte)(struct kvm *kvm,gfn_t gfn,enum pg_level level,kvm_pfn_t pfn); int (*remove_private_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
arch/x86/kvm/mmu/tdp_mmu.c +15 -9
@@ -585,18 +585,24 @@ static int __must_check handle_changed_private_spte(struct kvm *kvm, gfn_t gfn,lockdep_assert_held (&kvm->mmu_lock);if (is_present) { -/* TDP MMU doesn't change present -> present */ -KVM_BUG_ON (was_present, kvm); +void *private_spt; -/* -* Use different call to either set up middle level -* private page table, or leaf. -*/ -if (is_leaf) +if (level >PG_LEVEL_4K && was_leaf && !is_leaf) { +/* +* splitting large page into 4KB. +* tdp_mmu_split_huage_page() => tdp_mmu_link_sp() +*/ + private_spt =get_private_spt (gfn, new_spte, level); +KVM_BUG_ON (!private_spt, kvm); + ret =static_call (kvm_x86_zap_private_spte)(kvm, gfn, level); +kvm_flush_remote_tlbs (kvm); +if (!ret) + ret =static_call (kvm_x86_split_private_spt)(kvm, gfn, + level, private_spt); + }else if (is_leaf) ret =static_call (kvm_x86_set_private_spte)(kvm, gfn, level, new_pfn);else { -void *private_spt =get_private_spt (gfn, new_spte, level); - + private_spt =get_private_spt (gfn, new_spte, level);KVM_BUG_ON (!private_spt, kvm); ret =static_call (kvm_x86_link_private_spt)(kvm, gfn, level, private_spt); }
arch/x86/kvm/vmx/tdx.c +23 -2
@@ -1493,6 +1493,28 @@ static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,return 0 ; } +static int tdx_sept_split_private_spt (struct kvm *kvm,gfn_t gfn, +enum pg_level level,void *private_spt) +{ +int tdx_level =pg_level_to_tdx_sept_level (level); +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +gpa_t gpa =gfn_to_gpa (gfn); +hpa_t hpa =__pa (private_spt); +struct tdx_module_output out; +u64 err; + +/* See comment in tdx_sept_set_private_spte() */ + err =tdh_mem_page_demote (kvm_tdx->tdr_pa, gpa, tdx_level, hpa, &out); +if (err ==TDX_ERROR_SEPT_BUSY ) +return -EAGAIN ; +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_MEM_PAGE_DEMOTE , err, &out); +return -EIO ; + } + +return 0 ; +} +static int tdx_sept_zap_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level) {
@@ -1502,8 +1524,6 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,struct tdx_module_output out;u64 err; -/* For now large page isn't supported yet. */ -WARN_ON_ONCE (level !=PG_LEVEL_4K ); err =tdh_mem_range_block (kvm_tdx->tdr_pa, gpa, tdx_level, &out);if (err ==TDX_ERROR_SEPT_BUSY )return -EAGAIN ;
@@ -2725,6 +2745,7 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops) x86_ops->link_private_spt = tdx_sept_link_private_spt; x86_ops->free_private_spt = tdx_sept_free_private_spt; + x86_ops->split_private_spt = tdx_sept_split_private_spt; x86_ops->set_private_spte = tdx_sept_set_private_spte; x86_ops->remove_private_spte = tdx_sept_remove_private_spte; x86_ops->zap_private_spte = tdx_sept_zap_private_spte;
arch/x86/kvm/vmx/tdx_arch.h +1 -0
@@ -21,6 +21,7 @@#define TDH_MNG_CREATE 9#define TDH_VP_CREATE 10#define TDH_MNG_RD 11 +#define TDH_MEM_PAGE_DEMOTE 15#define TDH_MR_EXTEND 16#define TDH_MR_FINALIZE 17#define TDH_VP_FLUSH 18
arch/x86/kvm/vmx/tdx_ops.h +7 -0
@@ -133,6 +133,13 @@ static inline u64 tdh_mng_rd(hpa_t tdr, u64 field, struct tdx_module_output *outreturn __seamcall (TDH_MNG_RD , tdr, field,0 ,0 , out); } +static inline u64 tdh_mem_page_demote (hpa_t tdr,gpa_t gpa,int level,hpa_t page, +struct tdx_module_output *out) +{ +tdx_clflush_page (page,PG_LEVEL_4K ); +return seamcall_sept (TDH_MEM_PAGE_DEMOTE , gpa | level, tdr, page,0 , out); +} +static inline u64 tdh_mr_extend (hpa_t tdr,gpa_t gpa,struct tdx_module_output *out) {
Signature · 1 line
2.25.1
RFC PATCH v3 13/16 KVM: x86/tdp_mmu: Try to merge pages into a large page
In reply to [email protected]From: Isaku Yamahata <[email protected]>
When a large page is passed to the KVM page fault handler and some of sub pages are already populated, try to merge sub pages into a large page. This situation can happen when the guest converts small pages into shared and convert it back into private.
When a large page is passed to KVM mmu page fault handler and the spte corresponding to the page is non-leaf (one or more of sub pages are already populated at lower page level), the current kvm mmu zaps non-leaf spte at a large page level, and populate a leaf spte at that level. Thus small pages are converted into a large page. However, it doesn't work for TDX because zapping and re-populating results in zeroing page content. Instead, populate all small pages and merge them into a large page.
Merging pages into a large page can fail when some sub pages are accepted and some are not. In such case, with the assumption that guest tries to accept at large page size for performance when possible, don't try to be smart to identify which page is still pending, map all pages at lower page level, and let vcpu re-execute.
--- arch/x86/include/asm/kvm-x86-ops.h | 2 + arch/x86/include/asm/kvm_host.h | 3 + arch/x86/kvm/mmu/tdp_iter.c | 37 ++++++-- arch/x86/kvm/mmu/tdp_iter.h | 2 + arch/x86/kvm/mmu/tdp_mmu.c | 140 ++++++++++++++++++++++++++++- 5 files changed, 174 insertions(+), 10 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +2 -0
@@ -98,9 +98,11 @@ KVM_X86_OP(load_mmu_pgd) KVM_X86_OP_OPTIONAL(link_private_spt) KVM_X86_OP_OPTIONAL(free_private_spt) KVM_X86_OP_OPTIONAL(split_private_spt) +KVM_X86_OP_OPTIONAL(merge_private_spt) KVM_X86_OP_OPTIONAL(set_private_spte) KVM_X86_OP_OPTIONAL(remove_private_spte) KVM_X86_OP_OPTIONAL(zap_private_spte) +KVM_X86_OP_OPTIONAL(unzap_private_spte) KVM_X86_OP(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -1697,11 +1697,14 @@ struct kvm_x86_ops { void *private_spt);int (*split_private_spt)(struct kvm *kvm,gfn_t gfn,enum pg_level level,void *private_spt); +int (*merge_private_spt)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +void *private_spt);int (*set_private_spte)(struct kvm *kvm,gfn_t gfn,enum pg_level level,kvm_pfn_t pfn);int (*remove_private_spte)(struct kvm *kvm,gfn_t gfn,enum pg_level level,kvm_pfn_t pfn);int (*zap_private_spte)(struct kvm *kvm,gfn_t gfn,enum pg_level level); +int (*unzap_private_spte)(struct kvm *kvm,gfn_t gfn,enum pg_level level);bool (*has_wbinvd_exit)(void );
arch/x86/kvm/mmu/tdp_iter.c +28 -9
@@ -75,6 +75,14 @@ tdp_ptep_t spte_to_child_pt(u64 spte, int level)return (tdp_ptep_t )__va (spte_to_pfn (spte) <<PAGE_SHIFT ); } +static void step_down (struct tdp_iter *iter,tdp_ptep_t child_pt) +{ + iter->level--; + iter->pt_path[iter->level -1 ] = child_pt; + iter->gfn =round_gfn_for_level (iter->next_last_level_gfn, iter->level); +tdp_iter_refresh_sptep (iter); +} +/* * Steps down one level in the paging structure towards the goal GFN. Returns * true if the iterator was able to step down a level, false otherwise.
@@ -96,14 +104,28 @@ static bool try_step_down(struct tdp_iter *iter)if (!child_pt)return false; - iter->level--; - iter->pt_path[iter->level -1 ] = child_pt; - iter->gfn =round_gfn_for_level (iter->next_last_level_gfn, iter->level); -tdp_iter_refresh_sptep (iter); - +step_down (iter, child_pt);return true; } +/* Steps down for freezed spte. Don't re-read sptep because it was freezed. */ +void tdp_iter_step_down (struct tdp_iter *iter,tdp_ptep_t child_pt) +{ +WARN_ON_ONCE (!child_pt); +WARN_ON_ONCE (iter->yielded); +WARN_ON_ONCE (iter->level == iter->min_level); + +step_down (iter, child_pt); +} + +void tdp_iter_step_side (struct tdp_iter * iter) +{ + iter->gfn +=KVM_PAGES_PER_HPAGE (iter->level); + iter->next_last_level_gfn = iter->gfn; + iter->sptep++; + iter->old_spte =kvm_tdp_mmu_read_spte (iter->sptep); +} +/* * Steps to the next entry in the current page table, at the current page table * level. The next entry could point to a page backing guest memory or another
@@ -121,10 +143,7 @@ static bool try_step_side(struct tdp_iter *iter) (SPTE_ENT_PER_PAGE - 1))return false; - iter->gfn +=KVM_PAGES_PER_HPAGE (iter->level); - iter->next_last_level_gfn = iter->gfn; - iter->sptep++; - iter->old_spte =kvm_tdp_mmu_read_spte (iter->sptep); +tdp_iter_step_side (iter);return true; }
arch/x86/kvm/mmu/tdp_iter.h +2 -0
@@ -114,6 +114,8 @@ void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root, int min_level, gfn_t next_last_level_gfn);void tdp_iter_next (struct tdp_iter *iter);void tdp_iter_restart (struct tdp_iter *iter); +void tdp_iter_step_side (struct tdp_iter *iter); +void tdp_iter_step_down (struct tdp_iter *iter,tdp_ptep_t child_pt);static inline union kvm_mmu_page_role tdp_iter_child_role (struct tdp_iter *iter) {
arch/x86/kvm/mmu/tdp_mmu.c +139 -1
152 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC PATCH v3 14/16 KVM: x86/tdp_mmu: TDX: Implement merge pages into a large page
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Implement merge_private_stp callback.
--- arch/x86/kvm/vmx/tdx.c | 70 ++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx_arch.h | 1 + arch/x86/kvm/vmx/tdx_errno.h | 2 ++ arch/x86/kvm/vmx/tdx_ops.h | 6 ++++ 4 files changed, 79 insertions(+)
arch/x86/kvm/vmx/tdx.c +70 -0
@@ -1515,6 +1515,47 @@ static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn,return 0 ; } +static int tdx_sept_merge_private_spt (struct kvm *kvm,gfn_t gfn, +enum pg_level level,void *private_spt) +{ +int tdx_level =pg_level_to_tdx_sept_level (level); +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +struct tdx_module_output out; +gpa_t gpa =gfn_to_gpa (gfn); +u64 err; + +/* See comment in tdx_sept_set_private_spte() */ + err =tdh_mem_page_promote (kvm_tdx->tdr_pa, gpa, tdx_level, &out); +if (err ==TDX_ERROR_SEPT_BUSY ) +return -EAGAIN ; +if (err ==TDX_EPT_INVALID_PROMOTE_CONDITIONS ) +/* +* Some pages are accepted, some pending. Need to wait for TD +* to accept all pages. Tell it the caller. +*/ +return -EAGAIN ; +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_MEM_PAGE_PROMOTE , err, &out); +return -EIO ; + } +WARN_ON_ONCE (out.rcx !=__pa (private_spt)); + +/* +* TDH.MEM.PAGE.PROMOTE frees the Secure-EPT page for the lower level. +* Flush cache for reuse. +*/ +do { + err =tdh_phymem_page_wbinvd (set_hkid_to_hpa (__pa (private_spt), +to_kvm_tdx (kvm)->hkid)); + }while (err == (TDX_OPERAND_BUSY |TDX_OPERAND_ID_RCX )); +if (WARN_ON_ONCE (err)) { +pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err,NULL ); +return -EIO ; + } + +return 0 ; +} +static int tdx_sept_zap_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level) {
@@ -1584,6 +1625,33 @@ static void tdx_track(struct kvm_tdx *kvm_tdx) } +static int tdx_sept_unzap_private_spte (struct kvm *kvm,gfn_t gfn, +enum pg_level level) +{ +int tdx_level =pg_level_to_tdx_sept_level (level); +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +gpa_t gpa =gfn_to_gpa (gfn); +struct tdx_module_output out; +u64 err; + +do { + err =tdh_mem_range_unblock (kvm_tdx->tdr_pa, gpa, tdx_level, &out); + +/* +* tdh_mem_range_block() is accompanied with tdx_track() via kvm +* remote tlb flush. Wait for the caller of +* tdh_mem_range_block() to complete TDX track. +*/ + }while (err == (TDX_TLB_TRACKING_NOT_DONE |TDX_OPERAND_ID_SEPT )); +if (err ==TDX_ERROR_SEPT_BUSY ) +return -EAGAIN ; +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_MEM_RANGE_UNBLOCK , err, &out); +return -EIO ; + } +return 0 ; +} +static int tdx_sept_free_private_spt (struct kvm *kvm,gfn_t gfn,enum pg_level level,void *private_spt) {
@@ -2746,9 +2814,11 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops) x86_ops->link_private_spt = tdx_sept_link_private_spt; x86_ops->free_private_spt = tdx_sept_free_private_spt; x86_ops->split_private_spt = tdx_sept_split_private_spt; + x86_ops->merge_private_spt = tdx_sept_merge_private_spt; x86_ops->set_private_spte = tdx_sept_set_private_spte; x86_ops->remove_private_spte = tdx_sept_remove_private_spte; x86_ops->zap_private_spte = tdx_sept_zap_private_spte; + x86_ops->unzap_private_spte = tdx_sept_unzap_private_spte;return 0 ; }
arch/x86/kvm/vmx/tdx_arch.h +1 -0
@@ -29,6 +29,7 @@#define TDH_MNG_KEY_FREEID 20#define TDH_MNG_INIT 21#define TDH_VP_INIT 22 +#define TDH_MEM_PAGE_PROMOTE 23#define TDH_VP_RD 26#define TDH_MNG_KEY_RECLAIMID 27#define TDH_PHYMEM_PAGE_RECLAIM 28
arch/x86/kvm/vmx/tdx_errno.h +2 -0
@@ -19,6 +19,8 @@#define TDX_KEY_CONFIGURED 0x0000081500000000ULL#define TDX_NO_HKID_READY_TO_WBCACHE 0x0000082100000000ULL#define TDX_EPT_WALK_FAILED 0xC0000B0000000000ULL +#define TDX_TLB_TRACKING_NOT_DONE 0xC0000B0800000000ULL +#define TDX_EPT_INVALID_PROMOTE_CONDITIONS 0xC0000B0900000000ULL/* * TDG.VP.VMCALL Status Codes (returned in R10)
arch/x86/kvm/vmx/tdx_ops.h +6 -0
@@ -140,6 +140,12 @@ static inline u64 tdh_mem_page_demote(hpa_t tdr, gpa_t gpa, int level, hpa_t pagreturn seamcall_sept (TDH_MEM_PAGE_DEMOTE , gpa | level, tdr, page,0 , out); } +static inline u64 tdh_mem_page_promote (hpa_t tdr,gpa_t gpa,int level, +struct tdx_module_output *out) +{ +return seamcall_sept (TDH_MEM_PAGE_PROMOTE , gpa | level, tdr,0 ,0 , out); +} +static inline u64 tdh_mr_extend (hpa_t tdr,gpa_t gpa,struct tdx_module_output *out) {
Signature · 1 line
2.25.1
RFC PATCH v3 15/16 KVM: x86/mmu: Make kvm fault handelr aware of large page of private memslot
In reply to [email protected]From: Isaku Yamahata <[email protected]>
struct kvm_page_fault.req_level is the page level which takes care of the faulted-in page size. For now its calculation is only for the conventional kvm memslot by host_pfn_mapping_level() that traverses page table.
However, host_pfn_mapping_level() cannot be used for private kvm memslot because pages of private kvm memlost aren't mapped into user virtual address space. Instead page order is given when getting pfn. Remember it in struct kvm_page_fault and use it.
--- arch/x86/kvm/mmu/mmu.c | 36 +++++++++++++++++++++------------ arch/x86/kvm/mmu/mmu_internal.h | 8 ++++++++ 2 files changed, 31 insertions(+), 13 deletions(-)
arch/x86/kvm/mmu/mmu.c +23 -13
@@ -3118,12 +3118,12 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,return level; } -int kvm_mmu_max_mapping_level (struct kvm *kvm, -const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level,bool is_private) +static int __kvm_mmu_max_mapping_level (struct kvm *kvm, +const struct kvm_memory_slot *slot, +gfn_t gfn,int max_level,int host_level, +bool faultin_private) {struct kvm_lpage_info *linfo; -int host_level; max_level =min (max_level, max_huge_page_level);for ( ; max_level >PG_LEVEL_4K ; max_level--) {
@@ -3132,16 +3132,24 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,break ; } -if (is_private) -return max_level; -if (max_level ==PG_LEVEL_4K )return PG_LEVEL_4K ; - host_level =host_pfn_mapping_level (kvm, gfn, slot); +if (!faultin_private) { +WARN_ON_ONCE (host_level !=PG_LEVEL_NONE ); + host_level =host_pfn_mapping_level (kvm, gfn, slot); + } +WARN_ON_ONCE (host_level ==PG_LEVEL_NONE );return min (host_level, max_level); } +int kvm_mmu_max_mapping_level (struct kvm *kvm, +const struct kvm_memory_slot *slot,gfn_t gfn, +int max_level,bool faultin_private) +{ +return __kvm_mmu_max_mapping_level (kvm, slot, gfn, max_level,PG_LEVEL_NONE , faultin_private); +} +void kvm_mmu_hugepage_adjust (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) {struct kvm_memory_slot *slot = fault->slot;
@@ -3162,9 +3170,10 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault* Enforce the iTLB multihit workaround after capturing the requested * level, which will be used to do precise, accurate accounting. */ - fault->req_level =kvm_mmu_max_mapping_level (vcpu->kvm, slot, - fault->gfn, fault->max_level, - fault->is_private); + fault->req_level =__kvm_mmu_max_mapping_level (vcpu->kvm, slot, + fault->gfn, fault->max_level, + fault->host_level, +kvm_is_faultin_private (fault));if (fault->req_level ==PG_LEVEL_4K || fault->huge_page_disallowed)return ;
@@ -4294,7 +4303,8 @@ static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu,if (kvm_restricted_mem_get_pfn (slot, fault->gfn, &fault->pfn, &order))return RET_PF_RETRY ; - fault->max_level =min (order_to_level (order), fault->max_level); + fault->host_level =order_to_level (order); + fault->max_level =min ((u8 )fault->host_level, fault->max_level); fault->map_writable = !(slot->flags &KVM_MEM_READONLY );return RET_PF_CONTINUE ; }
@@ -4338,7 +4348,7 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)if (fault->is_private !=kvm_mem_is_private (vcpu->kvm, fault->gfn))return kvm_do_memory_fault_exit (vcpu, fault); -if (fault->is_private &&kvm_slot_can_be_private (slot)) +if (kvm_is_faultin_private (fault))return kvm_faultin_pfn_private (vcpu, fault); async = false;
arch/x86/kvm/mmu/mmu_internal.h +8 -0
@@ -337,6 +337,7 @@ struct kvm_page_fault {kvm_pfn_t pfn;hva_t hva;bool map_writable; +enum pg_level host_level;/* valid only for private memslot && private gfn */ };int kvm_tdp_page_fault (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);
@@ -445,4 +446,11 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot, }#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +static inline bool kvm_is_faultin_private (const struct kvm_page_fault *fault) +{ +if (IS_ENABLED (CONFIG_HAVE_KVM_RESTRICTED_MEM )) +return fault->is_private &&kvm_slot_can_be_private (fault->slot); +return false; +} +#endif /* __KVM_X86_MMU_INTERNAL_H */
Signature · 1 line
2.25.1
RFC PATCH v3 16/16 KVM: TDX: Allow 2MB large page for TD GUEST
In reply to [email protected]From: Xiaoyao Li <[email protected]>
Now that everything is there to support 2MB page for TD guest. Because TDX module TDH.MEM.PAGE.AUG supports 4KB page and 2MB page, set struct kvm_arch.tdp_max_page_level to 2MB page level.
--- arch/x86/kvm/mmu/tdp_mmu.c | 9 ++------- arch/x86/kvm/vmx/tdx.c | 4 ++-- 2 files changed, 4 insertions(+), 9 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +2 -7
@@ -1630,14 +1630,9 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) sp->nx_huge_page_disallowed = fault->huge_page_disallowed; -if (is_shadow_present_pte (iter.old_spte)) { -/* -* TODO: large page support. -* Doesn't support large page for TDX now -*/ -KVM_BUG_ON (is_private_sptep (iter.sptep), vcpu->kvm); +if (is_shadow_present_pte (iter.old_spte)) r =tdp_mmu_split_huge_page (kvm, &iter, sp, true); - }else +else r =tdp_mmu_link_sp (kvm, &iter, sp, true);/*
arch/x86/kvm/vmx/tdx.c +2 -2
@@ -487,8 +487,8 @@ int tdx_vm_init(struct kvm *kvm)*/ kvm_mmu_set_mmio_spte_value (kvm,0 ); -/* TODO: Enable 2mb and 1gb large page support. */ - kvm->arch.tdp_max_page_level =PG_LEVEL_4K ; +/* TDH.MEM.PAGE.AUG supports up to 2MB page. */ + kvm->arch.tdp_max_page_level =PG_LEVEL_2M ;/* * This function initializes only KVM software construct. It doesn't
Signature · 1 line
2.25.1