KVM: x86/mmu: Fast CR3 switch improvements
cover KVM: x86/mmu: Fast CR3 switch improvements
Two improvements for fast CR3 switch, implemented with nested VMX in mind, but they should be helpful in general.
Sean Christopherson (2):
KVM: x86/mmu: Ignore guest CR3 on fast root switch for direct MMU
KVM: x86/mmu: Reuse the current root if possible for fast switch
arch/x86/kvm/mmu/mmu.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
Signature · 1 line
2.24.1
PATCH 1/2 KVM: x86/mmu: Ignore guest CR3 on fast root switch for direct MMU
Ignore the guest's CR3 when looking for a cached root for a direct MMU, the guest's CR3 has no impact on the direct MMU's shadow pages (the role check ensures compatibility with CR0.WP, etc...).
Zero out root_cr3 when allocating the direct roots to make it clear that it's ignored.
--- arch/x86/kvm/mmu/mmu.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +5 -3
@@ -3730,7 +3730,9 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) vcpu->arch.mmu->root_hpa =__pa (vcpu->arch.mmu->pae_root); }else BUG (); - vcpu->arch.mmu->root_cr3 = vcpu->arch.mmu->get_cr3 (vcpu); + +/* root_cr3 is ignored for direct MMUs. */ + vcpu->arch.mmu->root_cr3 =0 ;return 0 ; }
@@ -4272,8 +4274,8 @@ static bool cached_root_available(struct kvm_vcpu *vcpu, gpa_t new_cr3,for (i =0 ; i <KVM_MMU_NUM_PREV_ROOTS ; i++) {swap (root, mmu->prev_roots[i]); -if (new_cr3 == root.cr3 &&VALID_PAGE (root.hpa) && -page_header (root.hpa) !=NULL && +if ((new_role.direct || new_cr3 == root.cr3) && +VALID_PAGE (root.hpa) &&page_header (root.hpa) && new_role.word ==page_header (root.hpa)->role.word)break ; }
Signature · 1 line
2.24.1
PATCH 2/2 KVM: x86/mmu: Reuse the current root if possible for fast switch
In reply to Sean ChristophersonReuse the current root when possible instead of grabbing a different root from the array of cached roots. Doing so avoids unnecessary MMU switches and also fixes a quirk where KVM can't reuse roots without creating multiple roots since the cache is a victim cache, i.e. roots are added to the cache when they're "evicted", not when they are created. The quirk could be fixed by adding roots to the cache on creation, but that would reduce the effective size of the cache as one of its entries would be burned to track the current root.
Reusing the current root is especially helpful for nested virt as the current root is almost always usable for the "new" MMU on nested VM-entry/VM-exit.
--- arch/x86/kvm/mmu/mmu.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +12 -3
@@ -4253,6 +4253,14 @@ static void nonpaging_init_context(struct kvm_vcpu *vcpu, context->nx = false; } +static inline bool is_root_usable (struct kvm_mmu_root_info *root,gpa_t cr3, +union kvm_mmu_page_role role) +{ +return (role.direct || cr3 == root->cr3) && +VALID_PAGE (root->hpa) &&page_header (root->hpa) && + role.word ==page_header (root->hpa)->role.word; +} +/* * Find out if a previously cached root matching the new CR3/role is available. * The current root is also inserted into the cache.
@@ -4271,12 +4279,13 @@ static bool cached_root_available(struct kvm_vcpu *vcpu, gpa_t new_cr3, root.cr3 = mmu->root_cr3; root.hpa = mmu->root_hpa; +if (is_root_usable (&root, new_cr3, new_role)) +return true; +for (i =0 ; i <KVM_MMU_NUM_PREV_ROOTS ; i++) {swap (root, mmu->prev_roots[i]); -if ((new_role.direct || new_cr3 == root.cr3) && -VALID_PAGE (root.hpa) &&page_header (root.hpa) && - new_role.word ==page_header (root.hpa)->role.word) +if (is_root_usable (&root, new_cr3, new_role))break ; }
Signature · 1 line
2.24.1
Quoted 10 lines · Sean Christopherson
Two improvements for fast CR3 switch, implemented with nested VMX in mind, but they should be helpful in general.
Sean Christopherson (2):
KVM: x86/mmu: Ignore guest CR3 on fast root switch for direct MMU
KVM: x86/mmu: Reuse the current root if possible for fast switcharch/x86/kvm/mmu/mmu.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
Queued, thanks.
Paolo