KVM: x86/mmu: Lots of bug fixes
Sean Christopherson
maintainerv2 cover KVM: x86/mmu: Lots of bug fixes
Fix nested NPT (nSVM) with 32-bit L1 and SME with shadow paging, which are completely broken. Opportunistically fix theoretical bugs related to prematurely reloading/unloading the MMU.
If nNPT is enabled, L1 can crash the host simply by using 32-bit NPT to trigger a null pointer dereference on pae_root.
SME with shadow paging (including nNPT) fails to set the C-bit in the shadow pages that don't go through standard MMU flows (PDPTPRs and the PML4 used by nNPT to shadow legacy NPT). It also failes to account for CR3[63:32], and thus the C-bit, being ignored outside of 64-bit mode.
Patches 01 and 02 fix the null pointer bugs.
Patches 03-09 fix mostly-benign related memory leaks.
Patches 10-12 fix the SME shadow paging bugs, which are also what led me to the nNPT null pointer bugs.
Patches 13 and 14 fix theoretical bugs with PTP_SWITCH and INVPCID that I found when auditing flows that touch the MMU context.
Patches 14-17 do additional clean up to hopefully make it harder to introduce bugs in the future.
On the plus side, I finally understand why KVM supports shadowing 2-level page tables with 4-level page tables...
Based on kvm/queue, commit fe5f0041c026 ("KVM/SVM: Move vmenter.S exception fixups out of line"). The null pointer fixes cherry-pick cleanly onto kvm/master, haven't tried the other bug fixes (I doubt they're worth backporting even though I tagged 'em with stable).
v2:
- Collect a review from Ben (did not include his review of patch 03 since the patch and its direct dependencies were changed).
- Move pae_root and lm_root allocation to a separate helper to avoid sleeping via get_zeroed_page() while holding mmu_lock.
- Add a patch to grab 'mmu' in a local variable.
- Remove the BUILD_BUG_ON() in make_mmu_pages_available() since the final check wouldn't actually guarnatee 4 pages were "available". Instead, add a comment about the limit being soft.
v1:
- https://lkml.kernel.org/r/[email protected]
Sean Christopherson (17):
KVM: nSVM: Set the shadow root level to the TDP level for nested NPT
KVM: x86/mmu: Alloc page for PDPTEs when shadowing 32-bit NPT with
64-bit
KVM: x86/mmu: Capture 'mmu' in a local variable when allocating roots
KVM: x86/mmu: Allocate the lm_root before allocating PAE roots
KVM: x86/mmu: Allocate pae_root and lm_root pages in dedicated helper
KVM: x86/mmu: Ensure MMU pages are available when allocating roots
KVM: x86/mmu: Check PDPTRs before allocating PAE roots
KVM: x86/mmu: Fix and unconditionally enable WARNs to detect PAE leaks
KVM: x86/mmu: Use '0' as the one and only value for an invalid PAE
root
KVM: x86/mmu: Set the C-bit in the PDPTRs and LM pseudo-PDPTRs
KVM: x86/mmu: Mark the PAE roots as decrypted for shadow paging
KVM: SVM: Don't strip the C-bit from CR2 on #PF interception
KVM: nVMX: Defer the MMU reload to the normal path on an EPTP switch
KVM: x86: Defer the MMU unload to the normal path on an global INVPCID
KVM: x86/mmu: Unexport MMU load/unload functions
KVM: x86/mmu: Sync roots after MMU load iff load as successful
KVM: x86/mmu: WARN on NULL pae_root or lm_root, or bad shadow root
level
arch/x86/include/asm/kvm_host.h | 3 - arch/x86/kvm/mmu.h | 4 + arch/x86/kvm/mmu/mmu.c | 273 ++++++++++++++++++++------------ arch/x86/kvm/mmu/tdp_mmu.c | 23 +-- arch/x86/kvm/svm/svm.c | 9 +- arch/x86/kvm/vmx/nested.c | 9 +- arch/x86/kvm/x86.c | 2 +- 7 files changed, 192 insertions(+), 131 deletions(-)
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 1/17 KVM: nSVM: Set the shadow root level to the TDP level for nested NPT
Override the shadow root level in the MMU context when configuring NPT for shadowing nested NPT. The level is always tied to the TDP level of the host, not whatever level the guest happens to be using.
--- arch/x86/kvm/mmu/mmu.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +8 -3
@@ -4618,12 +4618,17 @@ void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, u32 cr0, u32 cr4, u32 efer,struct kvm_mmu *context = &vcpu->arch.guest_mmu;union kvm_mmu_role new_role =kvm_calc_shadow_npt_root_page_role (vcpu); - context->shadow_root_level = new_role.base.level; -__kvm_mmu_new_pgd (vcpu, nested_cr3, new_role.base, false, false); -if (new_role.as_u64 != context->mmu_role.as_u64) +if (new_role.as_u64 != context->mmu_role.as_u64) {shadow_mmu_init_context (vcpu, context, cr0, cr4, efer, new_role); + +/* +* Override the level set by the common init helper, nested TDP +* always uses the host's TDP configuration. +*/ + context->shadow_root_level = new_role.base.level; + } }EXPORT_SYMBOL_GPL (kvm_init_shadow_npt_mmu);
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 2/17 KVM: x86/mmu: Alloc page for PDPTEs when shadowing 32-bit NPT with 64-bit
In reply to Sean ChristophersonAllocate the so called pae_root page on-demand, along with the lm_root page, when shadowing 32-bit NPT with 64-bit NPT, i.e. when running a 32-bit L1. KVM currently only allocates the page when NPT is disabled, or when L0 is 32-bit (using PAE paging).
Note, there is an existing memory leak involving the MMU roots, as KVM fails to free the PAE roots on failure. This will be addressed in a future commit.
--- arch/x86/kvm/mmu/mmu.c | 44 ++++++++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 15 deletions(-)
arch/x86/kvm/mmu/mmu.c +29 -15
@@ -3187,14 +3187,14 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,if (mmu->shadow_root_level >=PT64_ROOT_4LEVEL && (mmu->root_level >=PT64_ROOT_4LEVEL || mmu->direct_map)) {mmu_free_root_page (kvm, &mmu->root_hpa, &invalid_list); - }else { + }else if (mmu->pae_root) {for (i =0 ; i <4 ; ++i)if (mmu->pae_root[i] !=0 )mmu_free_root_page (kvm, &mmu->pae_root[i], &invalid_list); - mmu->root_hpa =INVALID_PAGE ; } + mmu->root_hpa =INVALID_PAGE ; mmu->root_pgd =0 ; }
@@ -3306,9 +3306,23 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)* the shadow page table may be a PAE or a long mode page table. */ pm_mask =PT_PRESENT_MASK ; -if (vcpu->arch.mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) +if (vcpu->arch.mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) { pm_mask |=PT_ACCESSED_MASK |PT_WRITABLE_MASK |PT_USER_MASK ; +/* +* Allocate the page for the PDPTEs when shadowing 32-bit NPT +* with 64-bit only when needed. Unlike 32-bit NPT, it doesn't +* need to be in low mem. See also lm_root below. +*/ +if (!vcpu->arch.mmu->pae_root) { +WARN_ON_ONCE (!tdp_enabled); + + vcpu->arch.mmu->pae_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); +if (!vcpu->arch.mmu->pae_root) +return -ENOMEM ; + } + } +for (i =0 ; i <4 ; ++i) {MMU_WARN_ON (VALID_PAGE (vcpu->arch.mmu->pae_root[i]));if (vcpu->arch.mmu->root_level ==PT32E_ROOT_LEVEL ) {
@@ -3331,21 +3345,19 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) vcpu->arch.mmu->root_hpa =__pa (vcpu->arch.mmu->pae_root);/* -* If we shadow a 32 bit page table with a long mode page -* table we enter this path. +* When shadowing 32-bit or PAE NPT with 64-bit NPT, the PML4 and PDP +* tables are allocated and initialized at MMU creation as there is no +* equivalent level in the guest's NPT to shadow. Allocate the tables +* on demand, as running a 32-bit L1 VMM is very rare. The PDP is +* handled above (to share logic with PAE), deal with the PML4 here. */ if (vcpu->arch.mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) {if (vcpu->arch.mmu->lm_root ==NULL ) { -/* -* The additional page necessary for this is only -* allocated on demand. -*/ -u64 *lm_root; lm_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); -if (lm_root ==NULL ) -return 1 ; +if (!lm_root) +return -ENOMEM ; lm_root[0 ] =__pa (vcpu->arch.mmu->pae_root) | pm_mask;
@@ -5248,9 +5260,11 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu)* while the PDP table is a per-vCPU construct that's allocated at MMU * creation. When emulating 32-bit mode, cr3 is only 32 bits even on * x86_64. Therefore we need to allocate the PDP table in the first -* 4GB of memory, which happens to fit the DMA32 zone. Except for -* SVM's 32-bit NPT support, TDP paging doesn't use PAE paging and can -* skip allocating the PDP table. +* 4GB of memory, which happens to fit the DMA32 zone. TDP paging +* generally doesn't use PAE paging and can skip allocating the PDP +* table. The main exception, handled here, is SVM's 32-bit NPT. The +* other exception is for shadowing L1's 32-bit or PAE NPT on 64-bit +* KVM; that horror is handled on-demand by mmu_alloc_shadow_roots(). */ if (tdp_enabled &&kvm_mmu_get_tdp_level (vcpu) >PT32E_ROOT_LEVEL )return 0 ;
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 3/17 KVM: x86/mmu: Capture 'mmu' in a local variable when allocating roots
In reply to Sean ChristophersonGrab 'mmu' and do s/vcpu->arch.mmu/mmu to shorten line lengths and yield smaller diffs when moving code around in future cleanup without forcing the new code to use the same ugly pattern.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 58 ++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 28 deletions(-)
arch/x86/kvm/mmu/mmu.c +30 -28
@@ -3235,7 +3235,8 @@ static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, gva_t gva,static int mmu_alloc_direct_roots (struct kvm_vcpu *vcpu) { -u8 shadow_root_level = vcpu->arch.mmu->shadow_root_level; +struct kvm_mmu *mmu = vcpu->arch.mmu; +u8 shadow_root_level = mmu->shadow_root_level;hpa_t root;unsigned i;
@@ -3244,42 +3245,43 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)if (!VALID_PAGE (root))return -ENOSPC ; - vcpu->arch.mmu->root_hpa = root; + mmu->root_hpa = root; }else if (shadow_root_level >=PT64_ROOT_4LEVEL ) { root =mmu_alloc_root (vcpu,0 ,0 , shadow_root_level, true);if (!VALID_PAGE (root))return -ENOSPC ; - vcpu->arch.mmu->root_hpa = root; + mmu->root_hpa = root; }else if (shadow_root_level ==PT32E_ROOT_LEVEL ) {for (i =0 ; i <4 ; ++i) { -MMU_WARN_ON (VALID_PAGE (vcpu->arch.mmu->pae_root[i])); +MMU_WARN_ON (VALID_PAGE (mmu->pae_root[i])); root =mmu_alloc_root (vcpu, i << (30 -PAGE_SHIFT ), i <<30 ,PT32_ROOT_LEVEL , true);if (!VALID_PAGE (root))return -ENOSPC ; - vcpu->arch.mmu->pae_root[i] = root |PT_PRESENT_MASK ; + mmu->pae_root[i] = root |PT_PRESENT_MASK ; } - vcpu->arch.mmu->root_hpa =__pa (vcpu->arch.mmu->pae_root); + mmu->root_hpa =__pa (mmu->pae_root); }else BUG ();/* root_pgd is ignored for direct MMUs. */ - vcpu->arch.mmu->root_pgd =0 ; + mmu->root_pgd =0 ;return 0 ; }static int mmu_alloc_shadow_roots (struct kvm_vcpu *vcpu) { +struct kvm_mmu *mmu = vcpu->arch.mmu;u64 pdptr, pm_mask;gfn_t root_gfn, root_pgd;hpa_t root;int i; - root_pgd = vcpu->arch.mmu->get_guest_pgd (vcpu); + root_pgd = mmu->get_guest_pgd (vcpu); root_gfn = root_pgd >>PAGE_SHIFT ;if (mmu_check_root (vcpu, root_gfn))
@@ -3289,14 +3291,14 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)* Do we shadow a long mode page table? If so we need to * write-protect the guests page table root. */ -if (vcpu->arch.mmu->root_level >=PT64_ROOT_4LEVEL ) { -MMU_WARN_ON (VALID_PAGE (vcpu->arch.mmu->root_hpa)); +if (mmu->root_level >=PT64_ROOT_4LEVEL ) { +MMU_WARN_ON (VALID_PAGE (mmu->root_hpa)); root =mmu_alloc_root (vcpu, root_gfn,0 , - vcpu->arch.mmu->shadow_root_level, false); + mmu->shadow_root_level, false);if (!VALID_PAGE (root))return -ENOSPC ; - vcpu->arch.mmu->root_hpa = root; + mmu->root_hpa = root; goto set_root_pgd; }
@@ -3306,7 +3308,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)* the shadow page table may be a PAE or a long mode page table. */ pm_mask =PT_PRESENT_MASK ; -if (vcpu->arch.mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) { +if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) { pm_mask |=PT_ACCESSED_MASK |PT_WRITABLE_MASK |PT_USER_MASK ;/*
@@ -3314,21 +3316,21 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)* with 64-bit only when needed. Unlike 32-bit NPT, it doesn't * need to be in low mem. See also lm_root below. */ -if (!vcpu->arch.mmu->pae_root) { +if (!mmu->pae_root) {WARN_ON_ONCE (!tdp_enabled); - vcpu->arch.mmu->pae_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); -if (!vcpu->arch.mmu->pae_root) + mmu->pae_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); +if (!mmu->pae_root)return -ENOMEM ; } }for (i =0 ; i <4 ; ++i) { -MMU_WARN_ON (VALID_PAGE (vcpu->arch.mmu->pae_root[i])); -if (vcpu->arch.mmu->root_level ==PT32E_ROOT_LEVEL ) { - pdptr = vcpu->arch.mmu->get_pdptr (vcpu, i); +MMU_WARN_ON (VALID_PAGE (mmu->pae_root[i])); +if (mmu->root_level ==PT32E_ROOT_LEVEL ) { + pdptr = mmu->get_pdptr (vcpu, i);if (!(pdptr &PT_PRESENT_MASK )) { - vcpu->arch.mmu->pae_root[i] =0 ; + mmu->pae_root[i] =0 ;continue ; } root_gfn = pdptr >>PAGE_SHIFT ;
@@ -3340,9 +3342,9 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) PT32_ROOT_LEVEL, false);if (!VALID_PAGE (root))return -ENOSPC ; - vcpu->arch.mmu->pae_root[i] = root | pm_mask; + mmu->pae_root[i] = root | pm_mask; } - vcpu->arch.mmu->root_hpa =__pa (vcpu->arch.mmu->pae_root); + mmu->root_hpa =__pa (mmu->pae_root);/* * When shadowing 32-bit or PAE NPT with 64-bit NPT, the PML4 and PDP
@@ -3351,24 +3353,24 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)* on demand, as running a 32-bit L1 VMM is very rare. The PDP is * handled above (to share logic with PAE), deal with the PML4 here. */ -if (vcpu->arch.mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) { -if (vcpu->arch.mmu->lm_root ==NULL ) { +if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) { +if (mmu->lm_root ==NULL ) {u64 *lm_root; lm_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT );if (!lm_root)return -ENOMEM ; - lm_root[0 ] =__pa (vcpu->arch.mmu->pae_root) | pm_mask; + lm_root[0 ] =__pa (mmu->pae_root) | pm_mask; - vcpu->arch.mmu->lm_root = lm_root; + mmu->lm_root = lm_root; } - vcpu->arch.mmu->root_hpa =__pa (vcpu->arch.mmu->lm_root); + mmu->root_hpa =__pa (mmu->lm_root); } set_root_pgd: - vcpu->arch.mmu->root_pgd = root_pgd; + mmu->root_pgd = root_pgd;return 0 ; }
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 4/17 KVM: x86/mmu: Allocate the lm_root before allocating PAE roots
In reply to Sean ChristophersonAllocate lm_root before the PAE roots so that the PAE roots aren't leaked if the memory allocation for the lm_root happens to fail.
Note, KVM can still leak PAE roots if mmu_check_root() fails on a guest's PDPTR, or if mmu_alloc_root() fails due to MMU pages not being available. Those issues will be fixed in future commits.
--- arch/x86/kvm/mmu/mmu.c | 64 ++++++++++++++++++++---------------------- 1 file changed, 31 insertions(+), 33 deletions(-)
arch/x86/kvm/mmu/mmu.c +31 -33
@@ -3308,21 +3308,38 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)* the shadow page table may be a PAE or a long mode page table. */ pm_mask =PT_PRESENT_MASK ; -if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) { +if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) pm_mask |=PT_ACCESSED_MASK |PT_WRITABLE_MASK |PT_USER_MASK ; -/* -* Allocate the page for the PDPTEs when shadowing 32-bit NPT -* with 64-bit only when needed. Unlike 32-bit NPT, it doesn't -* need to be in low mem. See also lm_root below. -*/ -if (!mmu->pae_root) { -WARN_ON_ONCE (!tdp_enabled); +/* +* When shadowing 32-bit or PAE NPT with 64-bit NPT, the PML4 and PDP +* tables are allocated and initialized at root creation as there is no +* equivalent level in the guest's NPT to shadow. Allocate the tables +* on demand, as running a 32-bit L1 VMM is very rare. Unlike 32-bit +* NPT, the PDP table doesn't need to be in low mem. Preallocate the +* pages so that the PAE roots aren't leaked on failure. +*/ +if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL && + (!mmu->pae_root || !mmu->lm_root)) { +u64 *lm_root, *pae_root; - mmu->pae_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); -if (!mmu->pae_root) -return -ENOMEM ; +if (WARN_ON_ONCE (!tdp_enabled || mmu->pae_root || mmu->lm_root)) +return -EIO ; + + pae_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); +if (!pae_root) +return -ENOMEM ; + + lm_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); +if (!lm_root) { +free_page ((unsigned long )pae_root); +return -ENOMEM ; } + + mmu->pae_root = pae_root; + mmu->lm_root = lm_root; + + lm_root[0 ] =__pa (mmu->pae_root) | pm_mask; }for (i =0 ; i <4 ; ++i) {
@@ -3344,30 +3361,11 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)return -ENOSPC ; mmu->pae_root[i] = root | pm_mask; } - mmu->root_hpa =__pa (mmu->pae_root); - -/* -* When shadowing 32-bit or PAE NPT with 64-bit NPT, the PML4 and PDP -* tables are allocated and initialized at MMU creation as there is no -* equivalent level in the guest's NPT to shadow. Allocate the tables -* on demand, as running a 32-bit L1 VMM is very rare. The PDP is -* handled above (to share logic with PAE), deal with the PML4 here. -*/ -if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) { -if (mmu->lm_root ==NULL ) { -u64 *lm_root; - - lm_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); -if (!lm_root) -return -ENOMEM ; - - lm_root[0 ] =__pa (mmu->pae_root) | pm_mask; - - mmu->lm_root = lm_root; - } +if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) mmu->root_hpa =__pa (mmu->lm_root); - } +else + mmu->root_hpa =__pa (mmu->pae_root); set_root_pgd: mmu->root_pgd = root_pgd;
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 5/17 KVM: x86/mmu: Allocate pae_root and lm_root pages in dedicated helper
In reply to Sean ChristophersonMove the on-demand allocation of the pae_root and lm_root pages, used by nested NPT for 32-bit L1s, into a separate helper. This will allow a future patch to hold mmu_lock while allocating the non-special roots so that make_mmu_pages_available() can be checked once at the start of root allocation, and thus avoid having to deal with failure in the middle of root allocation.
--- arch/x86/kvm/mmu/mmu.c | 84 +++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 30 deletions(-)
arch/x86/kvm/mmu/mmu.c +54 -30
@@ -3308,38 +3308,10 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)* the shadow page table may be a PAE or a long mode page table. */ pm_mask =PT_PRESENT_MASK ; -if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) +if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) { pm_mask |=PT_ACCESSED_MASK |PT_WRITABLE_MASK |PT_USER_MASK ; -/* -* When shadowing 32-bit or PAE NPT with 64-bit NPT, the PML4 and PDP -* tables are allocated and initialized at root creation as there is no -* equivalent level in the guest's NPT to shadow. Allocate the tables -* on demand, as running a 32-bit L1 VMM is very rare. Unlike 32-bit -* NPT, the PDP table doesn't need to be in low mem. Preallocate the -* pages so that the PAE roots aren't leaked on failure. -*/ -if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL && - (!mmu->pae_root || !mmu->lm_root)) { -u64 *lm_root, *pae_root; - -if (WARN_ON_ONCE (!tdp_enabled || mmu->pae_root || mmu->lm_root)) -return -EIO ; - - pae_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); -if (!pae_root) -return -ENOMEM ; - - lm_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); -if (!lm_root) { -free_page ((unsigned long )pae_root); -return -ENOMEM ; - } - - mmu->pae_root = pae_root; - mmu->lm_root = lm_root; - - lm_root[0 ] =__pa (mmu->pae_root) | pm_mask; + mmu->lm_root[0 ] =__pa (mmu->pae_root) | pm_mask; }for (i =0 ; i <4 ; ++i) {
@@ -3373,6 +3345,55 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)return 0 ; } +static int mmu_alloc_special_roots (struct kvm_vcpu *vcpu) +{ +struct kvm_mmu *mmu = vcpu->arch.mmu; +u64 *lm_root, *pae_root; + +/* +* When shadowing 32-bit or PAE NPT with 64-bit NPT, the PML4 and PDP +* tables are allocated and initialized at root creation as there is no +* equivalent level in the guest's NPT to shadow. Allocate the tables +* on demand, as running a 32-bit L1 VMM on 64-bit KVM is very rare. +*/ +if (mmu->direct_map || mmu->root_level >=PT64_ROOT_4LEVEL || + mmu->shadow_root_level <PT64_ROOT_4LEVEL ) +return 0 ; + +/* +* This mess only works with 4-level paging and needs to be updated to +* work with 5-level paging. +*/ +if (WARN_ON_ONCE (mmu->shadow_root_level !=PT64_ROOT_4LEVEL )) +return -EIO ; + +if (mmu->pae_root && mmu->lm_root) +return 0 ; + +/* +* The special roots should always be allocated in concert. Yell and +* bail if KVM ends up in a state where only one of the roots is valid. +*/ +if (WARN_ON_ONCE (!tdp_enabled || mmu->pae_root || mmu->lm_root)) +return -EIO ; + +/* Unlike 32-bit NPT, the PDP table doesn't need to be in low mem. */ + pae_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); +if (!pae_root) +return -ENOMEM ; + + lm_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); +if (!lm_root) { +free_page ((unsigned long )pae_root); +return -ENOMEM ; + } + + mmu->pae_root = pae_root; + mmu->lm_root = lm_root; + +return 0 ; +} +static int mmu_alloc_roots (struct kvm_vcpu *vcpu) {if (vcpu->arch.mmu->direct_map)
@@ -4820,6 +4841,9 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu)int r; r =mmu_topup_memory_caches (vcpu, !vcpu->arch.mmu->direct_map); +if (r) + goto out; + r =mmu_alloc_special_roots (vcpu);if (r) goto out; r =mmu_alloc_roots (vcpu);
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Paolo Bonzini
maintainerRe: [PATCH v2 05/17] KVM: x86/mmu: Allocate pae_root and lm_root pages in dedicated helper
arch/x86/kvm/mmu/mmu.c static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) · line 3366
Quoted 4 lines · Sean Christopherson
arch/x86/kvm/mmu/mmu.c +4
-0+/* +* This mess only works with 4-level paging and needs to be updated to +* work with 5-level paging. +*/
Planning for this, it's probably a good idea to rename lm_root to pml4_root. Can be done on top.
Paolo
Sean Christopherson
maintainerPATCH v2 6/17 KVM: x86/mmu: Ensure MMU pages are available when allocating roots
In reply to Sean ChristophersonHold the mmu_lock for write for the entire duration of allocating and initializing an MMU's roots. This ensures there are MMU pages available and thus prevents root allocations from failing. That in turn fixes a bug where KVM would fail to free valid PAE roots if a one of the later roots failed to allocate.
Add a comment to make_mmu_pages_available() to call out that the limit is a soft limit, e.g. KVM will temporarily exceed the threshold if a page fault allocates multiple shadow pages and there was only one page "available".
Note, KVM _still_ leaks the PAE roots if the guest PDPTR checks fail. This will be addressed in a future commit.
--- arch/x86/kvm/mmu/mmu.c | 50 +++++++++++++++----------------------- arch/x86/kvm/mmu/tdp_mmu.c | 23 ++++-------------- 2 files changed, 25 insertions(+), 48 deletions(-)
arch/x86/kvm/mmu/mmu.c +20 -30
@@ -2403,6 +2403,15 @@ static int make_mmu_pages_available(struct kvm_vcpu *vcpu)kvm_mmu_zap_oldest_mmu_pages (vcpu->kvm,KVM_REFILL_PAGES - avail); +/* +* Note, this check is intentionally soft, it only guarantees that one +* page is available, while the caller may end up allocating as many as +* four pages, e.g. for PAE roots or for 5-level paging. Temporarily +* exceeding the (arbitrary by default) limit will not harm the host, +* being too agressive may unnecessarily kill the guest, and getting an +* exact count is far more trouble than it's worth, especially in the +* page fault paths. +*/ if (!kvm_mmu_available_pages (vcpu->kvm))return -ENOSPC ;return 0 ;
@@ -3220,16 +3229,9 @@ static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, gva_t gva, {struct kvm_mmu_page *sp; -write_lock (&vcpu->kvm->mmu_lock); - -if (make_mmu_pages_available (vcpu)) { -write_unlock (&vcpu->kvm->mmu_lock); -return INVALID_PAGE ; - } sp =kvm_mmu_get_page (vcpu, gfn, gva, level, direct,ACC_ALL ); ++sp->root_count; -write_unlock (&vcpu->kvm->mmu_lock);return __pa (sp->spt); }
@@ -3242,16 +3244,9 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)if (is_tdp_mmu_enabled (vcpu->kvm)) { root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu); - -if (!VALID_PAGE (root)) -return -ENOSPC ; mmu->root_hpa = root; }else if (shadow_root_level >=PT64_ROOT_4LEVEL ) { - root =mmu_alloc_root (vcpu,0 ,0 , shadow_root_level, - true); - -if (!VALID_PAGE (root)) -return -ENOSPC ; + root =mmu_alloc_root (vcpu,0 ,0 , shadow_root_level, true); mmu->root_hpa = root; }else if (shadow_root_level ==PT32E_ROOT_LEVEL ) {for (i =0 ; i <4 ; ++i) {
@@ -3259,8 +3254,6 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) root =mmu_alloc_root (vcpu, i << (30 -PAGE_SHIFT ), i <<30 ,PT32_ROOT_LEVEL , true); -if (!VALID_PAGE (root)) -return -ENOSPC ; mmu->pae_root[i] = root |PT_PRESENT_MASK ; } mmu->root_hpa =__pa (mmu->pae_root);
@@ -3296,8 +3289,6 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) root =mmu_alloc_root (vcpu, root_gfn,0 , mmu->shadow_root_level, false); -if (!VALID_PAGE (root)) -return -ENOSPC ; mmu->root_hpa = root; goto set_root_pgd; }
@@ -3316,6 +3307,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)for (i =0 ; i <4 ; ++i) {MMU_WARN_ON (VALID_PAGE (mmu->pae_root[i])); +if (mmu->root_level ==PT32E_ROOT_LEVEL ) { pdptr = mmu->get_pdptr (vcpu, i);if (!(pdptr &PT_PRESENT_MASK )) {
@@ -3329,8 +3321,6 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) root =mmu_alloc_root (vcpu, root_gfn, i <<30 ,PT32_ROOT_LEVEL , false); -if (!VALID_PAGE (root)) -return -ENOSPC ; mmu->pae_root[i] = root | pm_mask; }
@@ -3394,14 +3384,6 @@ static int mmu_alloc_special_roots(struct kvm_vcpu *vcpu)return 0 ; } -static int mmu_alloc_roots (struct kvm_vcpu *vcpu) -{ -if (vcpu->arch.mmu->direct_map) -return mmu_alloc_direct_roots (vcpu); -else -return mmu_alloc_shadow_roots (vcpu); -} -void kvm_mmu_sync_roots (struct kvm_vcpu *vcpu) {int i;
@@ -4846,7 +4828,15 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) r =mmu_alloc_special_roots (vcpu);if (r) goto out; - r =mmu_alloc_roots (vcpu); +write_lock (&vcpu->kvm->mmu_lock); +if (make_mmu_pages_available (vcpu)) + r = -ENOSPC ; +else if (vcpu->arch.mmu->direct_map) + r =mmu_alloc_direct_roots (vcpu); +else + r =mmu_alloc_shadow_roots (vcpu); +write_unlock (&vcpu->kvm->mmu_lock); +kvm_mmu_sync_roots (vcpu);if (r) goto out;
arch/x86/kvm/mmu/tdp_mmu.c +5 -18
@@ -137,22 +137,21 @@ static struct kvm_mmu_page *alloc_tdp_mmu_page(struct kvm_vcpu *vcpu, gfn_t gfn,return sp; } -static struct kvm_mmu_page *get_tdp_mmu_vcpu_root (struct kvm_vcpu *vcpu) +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu) {union kvm_mmu_page_role role;struct kvm *kvm = vcpu->kvm;struct kvm_mmu_page *root; +lockdep_assert_held_write (&kvm->mmu_lock); + role =page_role_for_level (vcpu, vcpu->arch.mmu->shadow_root_level); -write_lock (&kvm->mmu_lock); -/* Check for an existing root before allocating a new one. */ for_each_tdp_mmu_root (kvm, root) {if (root->role.word == role.word) {kvm_mmu_get_root (kvm, root); -write_unlock (&kvm->mmu_lock); -return root; + goto out; } }
@@ -161,19 +160,7 @@ static struct kvm_mmu_page *get_tdp_mmu_vcpu_root(struct kvm_vcpu *vcpu)list_add (&root->link, &kvm->arch.tdp_mmu_roots); -write_unlock (&kvm->mmu_lock); - -return root; -} - -hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu) -{ -struct kvm_mmu_page *root; - - root =get_tdp_mmu_vcpu_root (vcpu); -if (!root) -return INVALID_PAGE ; - +out:return __pa (root->spt); }
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 7/17 KVM: x86/mmu: Check PDPTRs before allocating PAE roots
In reply to Sean ChristophersonCheck the validity of the PDPTRs before allocating any of the PAE roots, otherwise a bad PDPTR will cause KVM to leak any previously allocated roots.
--- arch/x86/kvm/mmu/mmu.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
arch/x86/kvm/mmu/mmu.c +14 -6
@@ -3269,7 +3269,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)static int mmu_alloc_shadow_roots (struct kvm_vcpu *vcpu) {struct kvm_mmu *mmu = vcpu->arch.mmu; -u64 pdptr, pm_mask; +u64 pdptrs[4 ], pm_mask;gfn_t root_gfn, root_pgd;hpa_t root;int i;
@@ -3280,6 +3280,17 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)if (mmu_check_root (vcpu, root_gfn))return 1 ; +if (mmu->root_level ==PT32E_ROOT_LEVEL ) { +for (i =0 ; i <4 ; ++i) { + pdptrs[i] = mmu->get_pdptr (vcpu, i); +if (!(pdptrs[i] &PT_PRESENT_MASK )) +continue ; + +if (mmu_check_root (vcpu, pdptrs[i] >>PAGE_SHIFT )) +return 1 ; + } + } +/* * Do we shadow a long mode page table? If so we need to * write-protect the guests page table root.
@@ -3309,14 +3320,11 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)MMU_WARN_ON (VALID_PAGE (mmu->pae_root[i]));if (mmu->root_level ==PT32E_ROOT_LEVEL ) { - pdptr = mmu->get_pdptr (vcpu, i); -if (!(pdptr &PT_PRESENT_MASK )) { +if (!(pdptrs[i] &PT_PRESENT_MASK )) { mmu->pae_root[i] =0 ;continue ; } - root_gfn = pdptr >>PAGE_SHIFT ; -if (mmu_check_root (vcpu, root_gfn)) -return 1 ; + root_gfn = pdptrs[i] >>PAGE_SHIFT ; } root = mmu_alloc_root(vcpu, root_gfn, i << 30,
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Re: [PATCH v2 07/17] KVM: x86/mmu: Check PDPTRs before allocating PAE roots
Quoted 28 lines · Sean Christopherson
Check the validity of the PDPTRs before allocating any of the PAE roots, otherwise a bad PDPTR will cause KVM to leak any previously allocated roots.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)arch/x86/kvm/mmu/mmu.c +2
-1@@ -3269,7 +3269,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)static int mmu_alloc_shadow_roots (struct kvm_vcpu *vcpu) {struct kvm_mmu *mmu = vcpu->arch.mmu; -u64 pdptr, pm_mask; +u64 pdptrs[4 ], pm_mask;gfn_t root_gfn, root_pgd;hpa_t root;int i;@@ -3280,6 +3280,17 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)if (mmu_check_root (vcpu, root_gfn))return 1 ; +if (mmu->root_level ==PT32E_ROOT_LEVEL ) {
arch/x86/kvm/mmu/mmu.c static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) · line 3293
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10
-0+for (i =0 ; i <4 ; ++i) { + pdptrs[i] = mmu->get_pdptr (vcpu, i); +if (!(pdptrs[i] &PT_PRESENT_MASK )) +continue ; + +if (mmu_check_root (vcpu, pdptrs[i] >>PAGE_SHIFT )) +return 1 ; + } + } +
I saw this splatting:
BUG: sleeping function called from invalid context at
arch/x86/kvm/kvm_cache_regs.h:115
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 3090, name:
qemu-system-x86
3 locks held by qemu-system-x86/3090:
#0: ffff8d499f8d00d0 (&vcpu->mutex){+.+.}-{3:3}, at:
kvm_vcpu_ioctl+0x8e/0x680 [kvm]
#1: ffffb1b540f873e8 (&kvm->srcu){....}-{0:0}, at:
vcpu_enter_guest+0xb30/0x1b10 [kvm]
#2: ffffb1b540f7d018 (&(kvm)->mmu_lock){+.+.}-{2:2}, at:
kvm_mmu_load+0xb5/0x540 [kvm]
Preemption disabled at:
[<ffffffffc0787365>] kvm_mmu_load+0xb5/0x540 [kvm]
CPU: 2 PID: 3090 Comm: qemu-system-x86 Tainted: G W OE
5.12.0-rc3+ #3
Call Trace:
dump_stack+0x87/0xb7
___might_sleep+0x202/0x250
__might_sleep+0x4a/0x80
kvm_pdptr_read+0x20/0x60 [kvm]
kvm_mmu_load+0x3bd/0x540 [kvm]
vcpu_enter_guest+0x1297/0x1b10 [kvm]
kvm_arch_vcpu_ioctl_run+0x372/0x690 [kvm]
kvm_vcpu_ioctl+0x3ca/0x680 [kvm]
__x64_sys_ioctl+0x27a/0x800
do_syscall_64+0x38/0x50
entry_SYSCALL_64_after_hwframe+0x44/0xaeThere is a might_sleep() in kvm_pdptr_read(), however, the original commit didn't explain more. I can send a formal one if the below fix is acceptable.
arch/x86/kvm/mmu/mmu.c +8 -1
@@ -3289,17 +3289,24 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)if (mmu_check_root (vcpu, root_gfn))return 1 ; +write_unlock (&vcpu->kvm->mmu_lock);if (mmu->root_level ==PT32E_ROOT_LEVEL ) {for (i =0 ; i <4 ; ++i) { pdptrs[i] = mmu->get_pdptr (vcpu, i);if (!(pdptrs[i] &PT_PRESENT_MASK ))continue ; -if (mmu_check_root (vcpu, pdptrs[i] >>PAGE_SHIFT )) +if (mmu_check_root (vcpu, pdptrs[i] >>PAGE_SHIFT )) { +write_lock (&vcpu->kvm->mmu_lock);return 1 ; + } } } +write_lock (&vcpu->kvm->mmu_lock); +if (make_mmu_pages_available (vcpu)) +return -ENOSPC ; +/* * Do we shadow a long mode page table? If so we need to * write-protect the guests page table root.
Paolo Bonzini
maintainerRe: [PATCH v2 07/17] KVM: x86/mmu: Check PDPTRs before allocating PAE roots
Quoted 10 lines · Wanpeng Li
I saw this splatting:
BUG: sleeping function called from invalid context at arch/x86/kvm/kvm_cache_regs.h:115 kvm_pdptr_read+0x20/0x60 [kvm] kvm_mmu_load+0x3bd/0x540 [kvm]There is a
might_sleep()inkvm_pdptr_read(), however, the original commit didn't explain more. I can send a formal one if the below fix is acceptable.
I think we can just push make_mmu_pages_available down into kvm_mmu_load's callees. This way it's not necessary to hold the lock until after the PDPTR check:
arch/x86/kvm/mmu/mmu.c +21 -9
@@ -3244,6 +3244,12 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)u8 shadow_root_level = mmu->shadow_root_level;hpa_t root;unsigned i; +int r; + +write_lock (&vcpu->kvm->mmu_lock); + r =make_mmu_pages_available (vcpu); +if (r <0 ) + goto out_unlock;if (is_tdp_mmu_enabled (vcpu->kvm)) { root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu);
@@ -3266,13 +3272,16 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) mmu->root_hpa =__pa (mmu->pae_root); }else {WARN_ONCE (1 ,"Bad TDP root level = %d\n" , shadow_root_level); -return -EIO ; + r = -EIO ; } +out_unlock: +write_unlock (&vcpu->kvm->mmu_lock); +/* root_pgd is ignored for direct MMUs. */ mmu->root_pgd =0 ; -return 0 ; +return r; }static int mmu_alloc_shadow_roots (struct kvm_vcpu *vcpu)
@@ -3282,6 +3291,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)gfn_t root_gfn, root_pgd;hpa_t root;int i; +int r; root_pgd = mmu->get_guest_pgd (vcpu); root_gfn = root_pgd >>PAGE_SHIFT ;
@@ -3300,6 +3310,11 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) } } +write_lock (&vcpu->kvm->mmu_lock); + r =make_mmu_pages_available (vcpu); +if (r <0 ) + goto out_unlock; +/* * Do we shadow a long mode page table? If so we need to * write-protect the guests page table root.
@@ -3308,7 +3323,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) root =mmu_alloc_root (vcpu, root_gfn,0 , mmu->shadow_root_level, false); mmu->root_hpa = root; - goto set_root_pgd; + goto out_unlock; }if (WARN_ON_ONCE (!mmu->pae_root))
@@ -3350,7 +3365,8 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)else mmu->root_hpa =__pa (mmu->pae_root); -set_root_pgd: +out_unlock: +write_unlock (&vcpu->kvm->mmu_lock); mmu->root_pgd = root_pgd;return 0 ;
@@ -4852,14 +4868,10 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) r =mmu_alloc_special_roots (vcpu);if (r) goto out; -write_lock (&vcpu->kvm->mmu_lock); -if (make_mmu_pages_available (vcpu)) - r = -ENOSPC ; -else if (vcpu->arch.mmu->direct_map) +if (vcpu->arch.mmu->direct_map) r =mmu_alloc_direct_roots (vcpu);else r =mmu_alloc_shadow_roots (vcpu); -write_unlock (&vcpu->kvm->mmu_lock);if (r) goto out;
Paolo
Sean Christopherson
maintainerRe: [PATCH v2 07/17] KVM: x86/mmu: Check PDPTRs before allocating PAE roots
Quoted 11 lines · Paolo Bonzini
Quoted 10 lines · Wanpeng Li
I saw this splatting:
BUG: sleeping function called from invalid context at arch/x86/kvm/kvm_cache_regs.h:115 kvm_pdptr_read+0x20/0x60 [kvm] kvm_mmu_load+0x3bd/0x540 [kvm]There is a
might_sleep()inkvm_pdptr_read(), however, the original commit didn't explain more. I can send a formal one if the below fix is acceptable.
We don't want to drop mmu_lock, even temporarily. The reason for holding it across the entire sequence is to ensure kvm_mmu_available_pages() isn't violated.
Quoted 3 lines
I think we can just push
make_mmu_pages_availabledown into kvm_mmu_load's callees. This way it's not necessary to hold the lock until after the PDPTR check:
...
Quoted 5 lines
(quoted diff) +0
-1@@ -4852,14 +4868,10 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) r = mmu_alloc_special_roots(vcpu); if (r) goto out; - write_lock(&vcpu->kvm->mmu_lock);
Review int kvm_mmu_load(struct kvm_vcpu *vcpu) · line 4876
Quoted 10 lines
(quoted diff) +1
-4- if (make_mmu_pages_available(vcpu)) - r = -ENOSPC; - else if (vcpu->arch.mmu->direct_map) + if (vcpu->arch.mmu->direct_map) r = mmu_alloc_direct_roots(vcpu); else r = mmu_alloc_shadow_roots(vcpu); - write_unlock(&vcpu->kvm->mmu_lock); if (r) goto out;
Freaking PDPTRs. I was really hoping we could keep the lock and pages_available() logic outside of the helpers. What if kvm_mmu_load() reads the PDPTRs and passes them into mmu_alloc_shadow_roots()? Or is that too ugly?
arch/x86/kvm/mmu/mmu.c +19 -10
@@ -3275,11 +3275,11 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)return 0 ; } -static int mmu_alloc_shadow_roots (struct kvm_vcpu *vcpu) +static int mmu_alloc_shadow_roots (struct kvm_vcpu *vcpu,u64 pdptrs[4 ]) {struct kvm_mmu *mmu = vcpu->arch.mmu; -u64 pdptrs[4 ], pm_mask;gfn_t root_gfn, root_pgd; +u64 pm_mask;hpa_t root;int i;
@@ -3291,11 +3291,8 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)if (mmu->root_level ==PT32E_ROOT_LEVEL ) {for (i =0 ; i <4 ; ++i) { - pdptrs[i] = mmu->get_pdptr (vcpu, i); -if (!(pdptrs[i] &PT_PRESENT_MASK )) -continue ; - -if (mmu_check_root (vcpu, pdptrs[i] >>PAGE_SHIFT )) +if ((pdptrs[i] &PT_PRESENT_MASK ) && +mmu_check_root (vcpu, pdptrs[i] >>PAGE_SHIFT ))return 1 ; } }
@@ -4844,21 +4841,33 @@ EXPORT_SYMBOL_GPL(kvm_mmu_reset_context);int kvm_mmu_load (struct kvm_vcpu *vcpu) { -int r; +struct kvm_mmu *mmu = vcpu->arch.mmu; +u64 pdptrs[4 ]; +int r, i; - r =mmu_topup_memory_caches (vcpu, !vcpu->arch.mmu->direct_map); + r =mmu_topup_memory_caches (vcpu, !mmu->direct_map);if (r) goto out; r =mmu_alloc_special_roots (vcpu);if (r) goto out; + +/* +* On SVM, reading PDPTRs might access guest memory, which might fault +* and thus might sleep. Grab the PDPTRs before acquiring mmu_lock. +*/ +if (!mmu->direct_map && mmu->root_level ==PT32E_ROOT_LEVEL ) { +for (i =0 ; i <4 ; ++i) + pdptrs[i] = mmu->get_pdptr (vcpu, i); + } +write_lock (&vcpu->kvm->mmu_lock);if (make_mmu_pages_available (vcpu)) r = -ENOSPC ;else if (vcpu->arch.mmu->direct_map) r =mmu_alloc_direct_roots (vcpu);else - r =mmu_alloc_shadow_roots (vcpu); + r =mmu_alloc_shadow_roots (vcpu, pdptrs);write_unlock (&vcpu->kvm->mmu_lock);if (r) goto out;
Paolo Bonzini
maintainerRe: [PATCH v2 07/17] KVM: x86/mmu: Check PDPTRs before allocating PAE roots
Quoted 3 lines · Sean Christopherson
Freaking PDPTRs. I was really hoping we could keep the lock and
pages_available()logic outside of the helpers. What ifkvm_mmu_load()reads the PDPTRs and passes them intommu_alloc_shadow_roots()? Or is that too ugly?
The patch I have posted (though untested) tries to do that in a slightly less ugly way by pushing make_mmu_pages_available down to mmu_alloc_*_roots.
Paolo
Quoted 70 lines
arch/x86/kvm/mmu/mmu.c +19
-10@@ -3275,11 +3275,11 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)return 0 ; } -static int mmu_alloc_shadow_roots (struct kvm_vcpu *vcpu) +static int mmu_alloc_shadow_roots (struct kvm_vcpu *vcpu,u64 pdptrs[4 ]) {struct kvm_mmu *mmu = vcpu->arch.mmu; -u64 pdptrs[4 ], pm_mask;gfn_t root_gfn, root_pgd; +u64 pm_mask;hpa_t root;int i;@@ -3291,11 +3291,8 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)if (mmu->root_level ==PT32E_ROOT_LEVEL ) {for (i =0 ; i <4 ; ++i) { - pdptrs[i] = mmu->get_pdptr (vcpu, i); -if (!(pdptrs[i] &PT_PRESENT_MASK )) -continue ; - -if (mmu_check_root (vcpu, pdptrs[i] >>PAGE_SHIFT )) +if ((pdptrs[i] &PT_PRESENT_MASK ) && +mmu_check_root (vcpu, pdptrs[i] >>PAGE_SHIFT ))return 1 ; } }@@ -4844,21 +4841,33 @@ EXPORT_SYMBOL_GPL(kvm_mmu_reset_context);int kvm_mmu_load (struct kvm_vcpu *vcpu) { -int r; +struct kvm_mmu *mmu = vcpu->arch.mmu; +u64 pdptrs[4 ]; +int r, i; - r =mmu_topup_memory_caches (vcpu, !vcpu->arch.mmu->direct_map); + r =mmu_topup_memory_caches (vcpu, !mmu->direct_map);if (r) goto out; r =mmu_alloc_special_roots (vcpu);if (r) goto out; + +/* +* On SVM, reading PDPTRs might access guest memory, which might fault +* and thus might sleep. Grab the PDPTRs before acquiring mmu_lock. +*/ +if (!mmu->direct_map && mmu->root_level ==PT32E_ROOT_LEVEL ) { +for (i =0 ; i <4 ; ++i) + pdptrs[i] = mmu->get_pdptr (vcpu, i); + } +write_lock (&vcpu->kvm->mmu_lock);if (make_mmu_pages_available (vcpu)) r = -ENOSPC ;else if (vcpu->arch.mmu->direct_map) r =mmu_alloc_direct_roots (vcpu);else - r =mmu_alloc_shadow_roots (vcpu); + r =mmu_alloc_shadow_roots (vcpu, pdptrs);write_unlock (&vcpu->kvm->mmu_lock);if (r) goto out;
Sean Christopherson
maintainerRe: [PATCH v2 07/17] KVM: x86/mmu: Check PDPTRs before allocating PAE roots
Quoted 7 lines · Paolo Bonzini
Quoted 3 lines · Sean Christopherson
Freaking PDPTRs. I was really hoping we could keep the lock and
pages_available()logic outside of the helpers. What ifkvm_mmu_load()reads the PDPTRs and passes them intommu_alloc_shadow_roots()? Or is that too ugly?The patch I have posted (though untested) tries to do that in a slightly less ugly way by pushing
make_mmu_pages_availabledown to mmu_alloc_*_roots.
Yeah, I agree it's less ugly. It would be nice to not duplicate that code, but it's probably not worth the ugliness. :-/
For your approach, can we put the out label after the success path? Setting mmu->root_pgd isn't wrong per se, but doing so might mislead future readers into thinking that it's functionally necessary.
arch/x86/kvm/mmu/mmu.c +39 -17
@@ -3244,6 +3244,13 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)u8 shadow_root_level = mmu->shadow_root_level;hpa_t root;unsigned i; +int r; + +write_lock (&vcpu->kvm->mmu_lock); + + r =make_mmu_pages_available (vcpu); +if (r) + goto out_unlock;if (is_tdp_mmu_enabled (vcpu->kvm)) { root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu);
@@ -3252,8 +3259,10 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) root =mmu_alloc_root (vcpu,0 ,0 , shadow_root_level, true); mmu->root_hpa = root; }else if (shadow_root_level ==PT32E_ROOT_LEVEL ) { -if (WARN_ON_ONCE (!mmu->pae_root)) -return -EIO ; +if (WARN_ON_ONCE (!mmu->pae_root)) { + r = -EIO ; + goto out_unlock; + }for (i =0 ; i <4 ; ++i) {WARN_ON_ONCE (IS_VALID_PAE_ROOT (mmu->pae_root[i]));
@@ -3266,13 +3275,15 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) mmu->root_hpa =__pa (mmu->pae_root); }else {WARN_ONCE (1 ,"Bad TDP root level = %d\n" , shadow_root_level); -return -EIO ; + r = -EIO ; + goto out_unlock; }/* root_pgd is ignored for direct MMUs. */ mmu->root_pgd =0 ; - -return 0 ; +out_unlock: +write_unlock (&vcpu->kvm->mmu_lock); +return r; }static int mmu_alloc_shadow_roots (struct kvm_vcpu *vcpu)
@@ -3281,7 +3292,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)u64 pdptrs[4 ], pm_mask;gfn_t root_gfn, root_pgd;hpa_t root; -int i; +int i, r; root_pgd = mmu->get_guest_pgd (vcpu); root_gfn = root_pgd >>PAGE_SHIFT ;
@@ -3289,6 +3300,10 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)if (mmu_check_root (vcpu, root_gfn))return 1 ; +/* +* On SVM, reading PDPTRs might access guest memory, which might fault +* and thus might sleep. Grab the PDPTRs before acquiring mmu_lock. +*/ if (mmu->root_level ==PT32E_ROOT_LEVEL ) {for (i =0 ; i <4 ; ++i) { pdptrs[i] = mmu->get_pdptr (vcpu, i);
@@ -3300,6 +3315,12 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) } } +write_lock (&vcpu->kvm->mmu_lock); + + r =make_mmu_pages_available (vcpu); +if (r) + goto out_unlock; +/* * Do we shadow a long mode page table? If so we need to * write-protect the guests page table root.
@@ -3311,8 +3332,10 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) goto set_root_pgd; } -if (WARN_ON_ONCE (!mmu->pae_root)) -return -EIO ; +if (WARN_ON_ONCE (!mmu->pae_root)) { + r = -EIO ; + goto out_unlock; + }/* * We shadow a 32 bit page table. This may be a legacy 2-level
@@ -3323,8 +3346,10 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) { pm_mask |=PT_ACCESSED_MASK |PT_WRITABLE_MASK |PT_USER_MASK ; -if (WARN_ON_ONCE (!mmu->lm_root)) -return -EIO ; +if (WARN_ON_ONCE (!mmu->lm_root)) { + r = -EIO ; + goto out_unlock; + } mmu->lm_root[0 ] =__pa (mmu->pae_root) | pm_mask; }
@@ -3352,8 +3377,9 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) set_root_pgd: mmu->root_pgd = root_pgd; - -return 0 ; +out_unlock: +write_unlock (&vcpu->kvm->mmu_lock); +return r; }static int mmu_alloc_special_roots (struct kvm_vcpu *vcpu)
@@ -4852,14 +4878,10 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) r =mmu_alloc_special_roots (vcpu);if (r) goto out; -write_lock (&vcpu->kvm->mmu_lock); -if (make_mmu_pages_available (vcpu)) - r = -ENOSPC ; -else if (vcpu->arch.mmu->direct_map) +if (vcpu->arch.mmu->direct_map) r =mmu_alloc_direct_roots (vcpu);else r =mmu_alloc_shadow_roots (vcpu); -write_unlock (&vcpu->kvm->mmu_lock);if (r) goto out;
Paolo Bonzini
maintainerRe: [PATCH v2 07/17] KVM: x86/mmu: Check PDPTRs before allocating PAE roots
Quoted 3 lines · Sean Christopherson
For your approach, can we put the out label after the success path? Setting
mmu->root_pgdisn't wrong per se, but doing so might mislead future readers into thinking that it's functionally necessary.
Indeed, thanks for the speedy review. I'll get it queued tomorrow.
Paolo
Sean Christopherson
maintainerPATCH v2 8/17 KVM: x86/mmu: Fix and unconditionally enable WARNs to detect PAE leaks
In reply to Sean ChristophersonExempt NULL PAE roots from the check to detect leaks, since kvm_mmu_free_roots() doesn't set them back to INVALID_PAGE. Stop hiding the WARNs to detect PAE root leaks behind MMU_WARN_ON, the hidden WARNs obviously didn't do their job given the hilarious number of bugs that could lead to PAE roots being leaked, not to mention the above false positive.
Opportunistically delete a warning on root_hpa being valid, there's nothing special about 4/5-level shadow pages that warrants a WARN.
--- arch/x86/kvm/mmu/mmu.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -4
@@ -3250,7 +3250,8 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) mmu->root_hpa = root; }else if (shadow_root_level ==PT32E_ROOT_LEVEL ) {for (i =0 ; i <4 ; ++i) { -MMU_WARN_ON (VALID_PAGE (mmu->pae_root[i])); +WARN_ON_ONCE (mmu->pae_root[i] && +VALID_PAGE (mmu->pae_root[i])); root =mmu_alloc_root (vcpu, i << (30 -PAGE_SHIFT ), i <<30 ,PT32_ROOT_LEVEL , true);
@@ -3296,8 +3297,6 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)* write-protect the guests page table root. */ if (mmu->root_level >=PT64_ROOT_4LEVEL ) { -MMU_WARN_ON (VALID_PAGE (mmu->root_hpa)); - root =mmu_alloc_root (vcpu, root_gfn,0 , mmu->shadow_root_level, false); mmu->root_hpa = root;
@@ -3317,7 +3316,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) }for (i =0 ; i <4 ; ++i) { -MMU_WARN_ON (VALID_PAGE (mmu->pae_root[i])); +WARN_ON_ONCE (mmu->pae_root[i] &&VALID_PAGE (mmu->pae_root[i]));if (mmu->root_level ==PT32E_ROOT_LEVEL ) {if (!(pdptrs[i] &PT_PRESENT_MASK )) {
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 9/17 KVM: x86/mmu: Use '0' as the one and only value for an invalid PAE root
In reply to Sean ChristophersonUse '0' to denote an invalid pae_root instead of '0' or INVALID_PAGE. Unlike root_hpa, the pae_roots hold permission bits and thus are guaranteed to be non-zero. Having to deal with both values leads to bugs, e.g. failing to set back to INVALID_PAGE, warning on the wrong value, etc...
--- arch/x86/kvm/mmu/mmu.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +12 -10
@@ -3197,11 +3197,14 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, (mmu->root_level >= PT64_ROOT_4LEVEL || mmu->direct_map)) {mmu_free_root_page (kvm, &mmu->root_hpa, &invalid_list); } else if (mmu->pae_root) { -for (i =0 ; i <4 ; ++i) -if (mmu->pae_root[i] !=0 ) -mmu_free_root_page (kvm, - &mmu->pae_root[i], - &invalid_list); +for (i =0 ; i <4 ; ++i) { +if (!mmu->pae_root[i]) +continue ; + +mmu_free_root_page (kvm, &mmu->pae_root[i], + &invalid_list); + mmu->pae_root[i] =0 ; + } } mmu->root_hpa =INVALID_PAGE ; mmu->root_pgd =0 ;
@@ -3250,8 +3253,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) mmu->root_hpa = root; }else if (shadow_root_level ==PT32E_ROOT_LEVEL ) {for (i =0 ; i <4 ; ++i) { -WARN_ON_ONCE (mmu->pae_root[i] && -VALID_PAGE (mmu->pae_root[i])); +WARN_ON_ONCE (mmu->pae_root[i]); root =mmu_alloc_root (vcpu, i << (30 -PAGE_SHIFT ), i <<30 ,PT32_ROOT_LEVEL , true);
@@ -3316,7 +3318,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) }for (i =0 ; i <4 ; ++i) { -WARN_ON_ONCE (mmu->pae_root[i] &&VALID_PAGE (mmu->pae_root[i])); +WARN_ON_ONCE (mmu->pae_root[i]);if (mmu->root_level ==PT32E_ROOT_LEVEL ) {if (!(pdptrs[i] &PT_PRESENT_MASK )) {
@@ -3438,7 +3440,7 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu)for (i =0 ; i <4 ; ++i) {hpa_t root = vcpu->arch.mmu->pae_root[i]; -if (root &&VALID_PAGE (root)) { +if (root && !WARN_ON_ONCE (!VALID_PAGE (root))) { root &=PT64_BASE_ADDR_MASK ; sp =to_shadow_page (root);mmu_sync_children (vcpu, sp);
@@ -5296,7 +5298,7 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) mmu->pae_root =page_address (page);for (i =0 ; i <4 ; ++i) - mmu->pae_root[i] =INVALID_PAGE ; + mmu->pae_root[i] =0 ;return 0 ; }
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Paolo Bonzini
maintainerRe: [PATCH v2 09/17] KVM: x86/mmu: Use '0' as the one and only value for an invalid PAE root
Quoted 5 lines · Sean Christopherson
Use '0' to denote an invalid
pae_rootinstead of '0' orINVALID_PAGE. Unlikeroot_hpa, thepae_rootshold permission bits and thus are guaranteed to be non-zero. Having to deal with both values leads to bugs, e.g. failing to set back toINVALID_PAGE, warning on the wrong value, etc...
I don't dispute this is a good idea, but it deserves one or more comments.
Paolo
Sean Christopherson
maintainerRe: [PATCH v2 09/17] KVM: x86/mmu: Use '0' as the one and only value for an invalid PAE root
Quoted 8 lines · Paolo Bonzini
Quoted 5 lines · Sean Christopherson
Use '0' to denote an invalid
pae_rootinstead of '0' orINVALID_PAGE. Unlikeroot_hpa, thepae_rootshold permission bits and thus are guaranteed to be non-zero. Having to deal with both values leads to bugs, e.g. failing to set back toINVALID_PAGE, warning on the wrong value, etc...I don't dispute this is a good idea, but it deserves one or more comments.
Agreed. What about adding macros?
/* Comment goes here. */
#define INVALID_PAE_ROOT 0
#define IS_VALID_PAE_ROOT(x) (!!(x))
Also, I missed this pattern in mmu_audit.c's mmu_spte_walk():
for (i = 0; i < 4; ++i) {
hpa_t root = vcpu->arch.mmu->pae_root[i];
if (root && VALID_PAGE(root)) {
root &= PT64_BASE_ADDR_MASK;
sp = to_shadow_page(root);
__mmu_spte_walk(vcpu, sp, fn, 2);
}
}Paolo Bonzini
maintainerRe: [PATCH v2 09/17] KVM: x86/mmu: Use '0' as the one and only value for an invalid PAE root
Quoted 8 lines · Sean Christopherson
Quoted 7 lines · Paolo Bonzini
Quoted 5 lines · Sean Christopherson
Use '0' to denote an invalid
pae_rootinstead of '0' orINVALID_PAGE. Unlikeroot_hpa, thepae_rootshold permission bits and thus are guaranteed to be non-zero. Having to deal with both values leads to bugs, e.g. failing to set back toINVALID_PAGE, warning on the wrong value, etc...
Quoted 7 lines
Quoted 1 line
I don't dispute this is a good idea, but it deserves one or more comments.
Agreed. What about adding macros?
/* Comment goes here. */
#defineINVALID_PAE_ROOT0
#defineIS_VALID_PAE_ROOT(x)(!!(x))
Yep, that's a nice solution.
Paolo
Quoted 14 lines
Also, I missed this pattern in mmu_audit.c's
mmu_spte_walk():for (i = 0; i < 4; ++i) { hpa_t root = vcpu->arch.mmu->pae_root[i]; if (root && VALID_PAGE(root)) { root &= PT64_BASE_ADDR_MASK; sp = to_shadow_page(root); __mmu_spte_walk(vcpu, sp, fn, 2); } }
Sean Christopherson
maintainerPATCH v2 10/17 KVM: x86/mmu: Set the C-bit in the PDPTRs and LM pseudo-PDPTRs
In reply to Sean ChristophersonSet the C-bit in SPTEs that are set outside of the normal MMU flows, specifically the PDPDTRs and the handful of special cased "LM root" entries, all of which are shadow paging only.
Note, the direct-mapped-root PDPTR handling is needed for the scenario where paging is disabled in the guest, in which case KVM uses a direct mapped MMU even though TDP is disabled.
--- arch/x86/kvm/mmu/mmu.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -2
@@ -3257,7 +3257,8 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) root =mmu_alloc_root (vcpu, i << (30 -PAGE_SHIFT ), i <<30 ,PT32_ROOT_LEVEL , true); - mmu->pae_root[i] = root |PT_PRESENT_MASK ; + mmu->pae_root[i] = root |PT_PRESENT_MASK | + shadow_me_mask; } mmu->root_hpa =__pa (mmu->pae_root); }else
@@ -3310,7 +3311,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)* or a PAE 3-level page table. In either case we need to be aware that * the shadow page table may be a PAE or a long mode page table. */ - pm_mask =PT_PRESENT_MASK ; + pm_mask =PT_PRESENT_MASK | shadow_me_mask;if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) { pm_mask |=PT_ACCESSED_MASK |PT_WRITABLE_MASK |PT_USER_MASK ;
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 11/17 KVM: x86/mmu: Mark the PAE roots as decrypted for shadow paging
In reply to Sean ChristophersonSet the PAE roots used as decrypted to play nice with SME when KVM is using shadow paging. Explicitly skip setting the C-bit when loading CR3 for PAE shadow paging, even though it's completely ignored by the CPU. The extra documentation is nice to have.
Note, there are several subtleties at play with NPT. In addition to legacy shadow paging, the PAE roots are used for SVM's NPT when either KVM is 32-bit (uses PAE paging) or KVM is 64-bit and shadowing 32-bit NPT. However, 32-bit Linux, and thus KVM, doesn't support SME. And 64-bit KVM can happily set the C-bit in CR3. This also means that keeping __sme_set(root) for 32-bit KVM when NPT is enabled is conceptually wrong, but functionally ok since SME is 64-bit only. Leave it as is to avoid unnecessary pollution.
--- arch/x86/kvm/mmu/mmu.c | 24 +++++++++++++++++++++++- arch/x86/kvm/svm/svm.c | 7 +++++-- 2 files changed, 28 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +23 -1
@@ -48,6 +48,7 @@#include <asm/memtype.h> #include <asm/cmpxchg.h> #include <asm/io.h> +#include <asm/set_memory.h> #include <asm/vmx.h> #include <asm/kvm_page_track.h> #include "trace.h"
@@ -3377,7 +3378,10 @@ static int mmu_alloc_special_roots(struct kvm_vcpu *vcpu)if (WARN_ON_ONCE (!tdp_enabled || mmu->pae_root || mmu->lm_root))return -EIO ; -/* Unlike 32-bit NPT, the PDP table doesn't need to be in low mem. */ +/* +* Unlike 32-bit NPT, the PDP table doesn't need to be in low mem, and +* doesn't need to be decrypted. +*/ pae_root = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT );if (!pae_root)return -ENOMEM ;
@@ -5264,6 +5268,8 @@ slot_handle_leaf(struct kvm *kvm, struct kvm_memory_slot *memslot,static void free_mmu_pages (struct kvm_mmu *mmu) { +if (!tdp_enabled && mmu->pae_root) +set_memory_encrypted ((unsigned long )mmu->pae_root,1 );free_page ((unsigned long )mmu->pae_root);free_page ((unsigned long )mmu->lm_root); }
@@ -5301,6 +5307,22 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu)for (i =0 ; i <4 ; ++i) mmu->pae_root[i] =0 ; +/* +* CR3 is only 32 bits when PAE paging is used, thus it's impossible to +* get the CPU to treat the PDPTEs as encrypted. Decrypt the page so +* that KVM's writes and the CPU's reads get along. Note, this is +* only necessary when using shadow paging, as 64-bit NPT can get at +* the C-bit even when shadowing 32-bit NPT, and SME isn't supported +* by 32-bit kernels (when KVM itself uses 32-bit NPT). +*/ +if (!tdp_enabled) +set_memory_decrypted ((unsigned long )mmu->pae_root,1 ); +else +WARN_ON_ONCE (shadow_me_mask); + +for (i =0 ; i <4 ; ++i) + mmu->pae_root[i] =0 ; +return 0 ; }
arch/x86/kvm/svm/svm.c +5 -2
@@ -3908,15 +3908,18 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long root,struct vcpu_svm *svm =to_svm (vcpu);unsigned long cr3; - cr3 =__sme_set (root);if (npt_enabled) { - svm->vmcb->control.nested_cr3 = cr3; + svm->vmcb->control.nested_cr3 =__sme_set (root);vmcb_mark_dirty (svm->vmcb,VMCB_NPT );/* Loading L2's CR3 is handled by enter_svm_guest_mode. */ if (!test_bit (VCPU_EXREG_CR3 , (ulong *)&vcpu->arch.regs_avail))return ; cr3 = vcpu->arch.cr3; + }else if (vcpu->arch.mmu->shadow_root_level >=PT64_ROOT_4LEVEL ) { + cr3 =__sme_set (root); + }else { + cr3 = root; } svm->vmcb->save.cr3 = cr3;
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Paolo Bonzini
maintainerRe: [PATCH v2 11/17] KVM: x86/mmu: Mark the PAE roots as decrypted for shadow paging
arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) · line 5308
Quoted 3 lines · Sean Christopherson
arch/x86/kvm/mmu/mmu.c +0
-0@@ -5301,6 +5307,22 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu)for (i =0 ; i <4 ; ++i) mmu->pae_root[i] =0 ;
I think this should be deleted, since you have another identical for loop below?
Paolo
Quoted 16 lines
arch/x86/kvm/mmu/mmu.c +16
-0+/* +* CR3 is only 32 bits when PAE paging is used, thus it's impossible to +* get the CPU to treat the PDPTEs as encrypted. Decrypt the page so +* that KVM's writes and the CPU's reads get along. Note, this is +* only necessary when using shadow paging, as 64-bit NPT can get at +* the C-bit even when shadowing 32-bit NPT, and SME isn't supported +* by 32-bit kernels (when KVM itself uses 32-bit NPT). +*/ +if (!tdp_enabled) +set_memory_decrypted ((unsigned long )mmu->pae_root,1 ); +else +WARN_ON_ONCE (shadow_me_mask); + +for (i =0 ; i <4 ; ++i) + mmu->pae_root[i] =0 ; +
Sean Christopherson
maintainerRe: [PATCH v2 11/17] KVM: x86/mmu: Mark the PAE roots as decrypted for shadow paging
Quoted 7 lines · Paolo Bonzini
arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) · line 5308
Quoted 3 lines · Sean Christopherson
arch/x86/kvm/mmu/mmu.c +0
-0@@ -5301,6 +5307,22 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu)for (i =0 ; i <4 ; ++i) mmu->pae_root[i] =0 ;I think this should be deleted, since you have another identical for loop below?
Yes, rebase gone awry. And the zeroing needs to be done after decryption. Good eyes!
Quoted 19 lines
Paolo
Quoted 16 lines
arch/x86/kvm/mmu/mmu.c +16
-0+/* +* CR3 is only 32 bits when PAE paging is used, thus it's impossible to +* get the CPU to treat the PDPTEs as encrypted. Decrypt the page so +* that KVM's writes and the CPU's reads get along. Note, this is +* only necessary when using shadow paging, as 64-bit NPT can get at +* the C-bit even when shadowing 32-bit NPT, and SME isn't supported +* by 32-bit kernels (when KVM itself uses 32-bit NPT). +*/ +if (!tdp_enabled) +set_memory_decrypted ((unsigned long )mmu->pae_root,1 ); +else +WARN_ON_ONCE (shadow_me_mask); + +for (i =0 ; i <4 ; ++i) + mmu->pae_root[i] =0 ; +
Sean Christopherson
maintainerPATCH v2 12/17 KVM: SVM: Don't strip the C-bit from CR2 on #PF interception
In reply to Sean ChristophersonDon't strip the C-bit from the faulting address on an intercepted #PF, the address is a virtual address, not a physical address.
--- arch/x86/kvm/svm/svm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/svm/svm.c +1 -1
@@ -1907,7 +1907,7 @@ static int pf_interception(struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu); -u64 fault_address =__sme_clr (svm->vmcb->control.exit_info_2); +u64 fault_address = svm->vmcb->control.exit_info_2;u64 error_code = svm->vmcb->control.exit_info_1; return kvm_handle_page_fault(vcpu, error_code, fault_address,
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 13/17 KVM: nVMX: Defer the MMU reload to the normal path on an EPTP switch
In reply to Sean ChristophersonDefer reloading the MMU after a EPTP successful EPTP switch. The VMFUNC instruction itself is executed in the previous EPTP context, any side effects, e.g. updating RIP, should occur in the old context. Practically speaking, this bug is benign as VMX doesn't touch the MMU when skipping an emulated instruction, nor does queuing a single-step #DB. No other post-switch side effects exist.
--- arch/x86/kvm/vmx/nested.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/nested.c +2 -7
@@ -5473,16 +5473,11 @@ static int nested_vmx_eptp_switching(struct kvm_vcpu *vcpu,if (!nested_vmx_check_eptp (vcpu, new_eptp))return 1 ; -kvm_mmu_unload (vcpu); mmu->ept_ad = accessed_dirty; mmu->mmu_role.base.ad_disabled = !accessed_dirty; vmcs12->ept_pointer = new_eptp; -/* -* TODO: Check what's the correct approach in case -* mmu reload fails. Currently, we just let the next -* reload potentially fail -*/ -kvm_mmu_reload (vcpu); + +kvm_make_request (KVM_REQ_MMU_RELOAD , vcpu); }return 0 ;
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 14/17 KVM: x86: Defer the MMU unload to the normal path on an global INVPCID
In reply to Sean ChristophersonDefer unloading the MMU after a INVPCID until the instruction emulation has completed, i.e. until after RIP has been updated.
On VMX, this is a benign bug as VMX doesn't touch the MMU when skipping an emulated instruction. However, on SVM, if nrip is disabled, the emulator is used to skip an instruction, which would lead to fireworks if the emulator were invoked without a valid MMU.
--- arch/x86/kvm/x86.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/x86.c +1 -1
@@ -11531,7 +11531,7 @@ int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva) fallthrough;case INVPCID_TYPE_ALL_INCL_GLOBAL : -kvm_mmu_unload (vcpu); +kvm_make_request (KVM_REQ_MMU_RELOAD , vcpu);return kvm_skip_emulated_instruction (vcpu);default :
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 15/17 KVM: x86/mmu: Unexport MMU load/unload functions
In reply to Sean ChristophersonUnexport the MMU load and unload helpers now that they are no longer used (incorrectly) in vendor code.
Opportunistically move the kvm_mmu_sync_roots() declaration into mmu.h, it should not be exposed to vendor code.
No functional change intended.
--- arch/x86/include/asm/kvm_host.h | 3 --- arch/x86/kvm/mmu.h | 4 ++++ arch/x86/kvm/mmu/mmu.c | 2 -- 3 files changed, 4 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -3
@@ -1592,9 +1592,6 @@ void kvm_update_dr7(struct kvm_vcpu *vcpu);int kvm_mmu_unprotect_page (struct kvm *kvm,gfn_t gfn);void __kvm_mmu_free_some_pages (struct kvm_vcpu *vcpu); -int kvm_mmu_load (struct kvm_vcpu *vcpu); -void kvm_mmu_unload (struct kvm_vcpu *vcpu); -void kvm_mmu_sync_roots (struct kvm_vcpu *vcpu);void kvm_mmu_free_roots (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu,ulong roots_to_free); gpa_t translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, u32 access,
arch/x86/kvm/mmu.h +4 -0
@@ -74,6 +74,10 @@ bool kvm_can_do_async_pf(struct kvm_vcpu *vcpu);int kvm_handle_page_fault (struct kvm_vcpu *vcpu,u64 error_code,u64 fault_address,char *insn,int insn_len); +int kvm_mmu_load (struct kvm_vcpu *vcpu); +void kvm_mmu_unload (struct kvm_vcpu *vcpu); +void kvm_mmu_sync_roots (struct kvm_vcpu *vcpu); +static inline int kvm_mmu_reload (struct kvm_vcpu *vcpu) {if (likely (vcpu->arch.mmu->root_hpa !=INVALID_PAGE ))
arch/x86/kvm/mmu/mmu.c +0 -2
@@ -4859,7 +4859,6 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) out:return r; } -EXPORT_SYMBOL_GPL (kvm_mmu_load);void kvm_mmu_unload (struct kvm_vcpu *vcpu) {
@@ -4868,7 +4867,6 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu)kvm_mmu_free_roots (vcpu, &vcpu->arch.guest_mmu,KVM_MMU_ROOTS_ALL );WARN_ON (VALID_PAGE (vcpu->arch.guest_mmu.root_hpa)); } -EXPORT_SYMBOL_GPL (kvm_mmu_unload);static bool need_remote_flush (u64 old,u64 new) {
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 16/17 KVM: x86/mmu: Sync roots after MMU load iff load as successful
In reply to Sean ChristophersonFor clarity, explicitly skip syncing roots if the MMU load failed instead of relying on the !VALID_PAGE check in kvm_mmu_sync_roots().
--- arch/x86/kvm/mmu/mmu.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -2
@@ -4850,10 +4850,11 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu)else r =mmu_alloc_shadow_roots (vcpu);write_unlock (&vcpu->kvm->mmu_lock); +if (r) + goto out;kvm_mmu_sync_roots (vcpu); -if (r) - goto out; +kvm_mmu_load_pgd (vcpu);static_call (kvm_x86_tlb_flush_current)(vcpu); out:
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Sean Christopherson
maintainerPATCH v2 17/17 KVM: x86/mmu: WARN on NULL pae_root or lm_root, or bad shadow root level
In reply to Sean ChristophersonWARN if KVM is about to dereference a NULL pae_root or lm_root when loading an MMU, and convert the BUG() on a bad shadow_root_level into a WARN (now that errors are handled cleanly). With nested NPT, botching the level and sending KVM down the wrong path is all too easy, and the on-demand allocation of pae_root and lm_root means bugs crash the host. Obviously, KVM could unconditionally allocate the roots, but that's arguably a worse failure mode as it would potentially corrupt the guest instead of crashing it.
--- arch/x86/kvm/mmu/mmu.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +13 -2
@@ -3253,6 +3253,9 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) root =mmu_alloc_root (vcpu,0 ,0 , shadow_root_level, true); mmu->root_hpa = root; }else if (shadow_root_level ==PT32E_ROOT_LEVEL ) { +if (WARN_ON_ONCE (!mmu->pae_root)) +return -EIO ; +for (i =0 ; i <4 ; ++i) {WARN_ON_ONCE (mmu->pae_root[i]);
@@ -3262,8 +3265,10 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) shadow_me_mask; } mmu->root_hpa =__pa (mmu->pae_root); - }else -BUG (); + }else { +WARN_ONCE (1 ,"Bad TDP root level = %d\n" , shadow_root_level); +return -EIO ; + }/* root_pgd is ignored for direct MMUs. */ mmu->root_pgd =0 ;
@@ -3307,6 +3312,9 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) goto set_root_pgd; } +if (WARN_ON_ONCE (!mmu->pae_root)) +return -EIO ; +/* * We shadow a 32 bit page table. This may be a legacy 2-level * or a PAE 3-level page table. In either case we need to be aware that
@@ -3316,6 +3324,9 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)if (mmu->shadow_root_level ==PT64_ROOT_4LEVEL ) { pm_mask |=PT_ACCESSED_MASK |PT_WRITABLE_MASK |PT_USER_MASK ; +if (WARN_ON_ONCE (!mmu->lm_root)) +return -EIO ; + mmu->lm_root[0 ] =__pa (mmu->pae_root) | pm_mask; }
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
Quoted 68 lines · Sean Christopherson
Fix nested NPT (nSVM) with 32-bit L1 and SME with shadow paging, which are completely broken. Opportunistically fix theoretical bugs related to prematurely reloading/unloading the MMU.
If nNPT is enabled, L1 can crash the host simply by using 32-bit NPT to trigger a null pointer dereference on
pae_root.SME with shadow paging (including nNPT) fails to set the C-bit in the shadow pages that don't go through standard MMU flows (PDPTPRs and the PML4 used by nNPT to shadow legacy NPT). It also failes to account for CR3[63:32], and thus the C-bit, being ignored outside of 64-bit mode.
Patches 01 and 02 fix the null pointer bugs.
Patches 03-09 fix mostly-benign related memory leaks.
Patches 10-12 fix the SME shadow paging bugs, which are also what led me to the nNPT null pointer bugs.
Patches 13 and 14 fix theoretical bugs with
PTP_SWITCHand INVPCID that I found when auditing flows that touch the MMU context.Patches 14-17 do additional clean up to hopefully make it harder to introduce bugs in the future.
On the plus side, I finally understand why KVM supports shadowing 2-level page tables with 4-level page tables...
Based on kvm/queue, commit
fe5f0041c026("KVM/SVM: Movevmenter.Sexception fixups out of line"). The null pointer fixes cherry-pick cleanly onto kvm/master, haven't tried the other bug fixes (I doubt they're worth backporting even though I tagged 'em with stable).v2:
- Collect a review from Ben (did not include his review of patch 03 since the patch and its direct dependencies were changed).
- Movepae_rootandlm_rootallocation to a separate helper to avoid sleeping viaget_zeroed_page()while holdingmmu_lock.
- Add a patch to grab 'mmu' in a local variable.
- Remove theBUILD_BUG_ON()inmake_mmu_pages_available()since the final check wouldn't actually guarnatee 4 pages were "available". Instead, add a comment about the limit being soft.v1:
- https://lkml.kernel.org/r/[email protected]Sean Christopherson (17):
KVM: nSVM: Set the shadow root level to the TDP level for nested NPT
KVM: x86/mmu: Alloc page for PDPTEs when shadowing 32-bit NPT with64-bitKVM: x86/mmu: Capture 'mmu' in a local variable when allocating roots
KVM: x86/mmu: Allocate thelm_rootbefore allocating PAE roots
KVM: x86/mmu: Allocatepae_rootandlm_rootpages in dedicated helper
KVM: x86/mmu: Ensure MMU pages are available when allocating roots
KVM: x86/mmu: Check PDPTRs before allocating PAE roots
KVM: x86/mmu: Fix and unconditionally enable WARNs to detect PAE leaks
KVM: x86/mmu: Use '0' as the one and only value for an invalid PAE
root
KVM: x86/mmu: Set the C-bit in the PDPTRs and LM pseudo-PDPTRs
KVM: x86/mmu: Mark the PAE roots as decrypted for shadow paging
KVM: SVM: Don't strip the C-bit from CR2 on #PF interception
KVM: nVMX: Defer the MMU reload to the normal path on an EPTP switch
KVM: x86: Defer the MMU unload to the normal path on an global INVPCID
KVM: x86/mmu: Unexport MMU load/unload functions
KVM: x86/mmu: Sync roots after MMU load iff load as successful
KVM: x86/mmu: WARN on NULLpae_rootorlm_root, or bad shadow root
level
Quoted 9 lines
arch/x86/include/asm/kvm_host.h | 3 - arch/x86/kvm/mmu.h | 4 + arch/x86/kvm/mmu/mmu.c | 273 ++++++++++++++++++++------------ arch/x86/kvm/mmu/tdp_mmu.c | 23 +-- arch/x86/kvm/svm/svm.c | 9 +- arch/x86/kvm/vmx/nested.c | 9 +- arch/x86/kvm/x86.c | 2 +- 7 files changed, 192 insertions(+), 131 deletions(-)
Queued all except 9 and 11, thanks.
Paolo