KVM: x86/mmu: Zap only obsolete roots on "reload"
Subsystems (6)
Sean Christopherson
maintainerv2 cover KVM: x86/mmu: Zap only obsolete roots on "reload"
For all intents and purposes, this is an x86/mmu series, but it touches s390 and common KVM code because KVM_REQ_MMU_RELOAD is currently a generic request despite its use being encapsulated entirely within arch code.
The meat of the series is to zap only obsolete (a.k.a. invalid) roots in response to KVM marking a root obsolete/invalid due to it being zapped. KVM currently drops/zaps all roots, which, aside from being a performance hit if the guest is using multiple roots, complicates x86 KVM paths that load a new root because it raises the question of what should be done if there's a pending KVM_REQ_MMU_RELOAD, i.e. if the path _knows_ that any root it loads will be obliterated.
Paolo, I'm hoping you can squash patch 01 with your patch it "fixes".
I'm also speculating that this will be applied after my patch to remove KVM_REQ_GPC_INVALIDATE, otherwise the changelog in patch 06 will be wrong.
v2:
- Collect reviews. [Claudio, Janosch]
- Rebase to latest kvm/queue.
v1: https://lore.kernel.org/all/[email protected]
Sean Christopherson (7):
KVM: x86: Remove spurious whitespaces from kvm_post_set_cr4()
KVM: x86: Invoke kvm_mmu_unload() directly on CR4.PCIDE change
KVM: Drop kvm_reload_remote_mmus(), open code request in x86 users
KVM: x86/mmu: Zap only obsolete roots if a root shadow page is zapped
KVM: s390: Replace KVM_REQ_MMU_RELOAD usage with arch specific request
KVM: Drop KVM_REQ_MMU_RELOAD and update vcpu-requests.rst
documentation
KVM: WARN if is_unsync_root() is called on a root without a shadow
page
Documentation/virt/kvm/vcpu-requests.rst | 7 +- arch/s390/include/asm/kvm_host.h | 2 + arch/s390/kvm/kvm-s390.c | 8 +-- arch/s390/kvm/kvm-s390.h | 2 +- arch/x86/include/asm/kvm_host.h | 2 + arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/mmu/mmu.c | 83 ++++++++++++++++++++---- arch/x86/kvm/x86.c | 10 +-- include/linux/kvm_host.h | 4 +- virt/kvm/kvm_main.c | 5 -- 10 files changed, 90 insertions(+), 34 deletions(-)
base-commit: f4bc051fc91ab9f1d5225d94e52d369ef58bec58
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Sean Christopherson
maintainerPATCH v2 1/7 KVM: x86: Remove spurious whitespaces from kvm_post_set_cr4()
Remove leading spaces that snuck into kvm_post_set_cr4(), fixing the KVM_REQ_TLB_FLUSH_CURRENT request in particular is helpful as it unaligns the body of the if-statement from the condition check.
--- arch/x86/kvm/x86.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +2 -2
@@ -1089,7 +1089,7 @@ void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned lon*/ if (((cr4 ^ old_cr4) &X86_CR4_PGE ) || (!(cr4 &X86_CR4_PCIDE ) && (old_cr4 &X86_CR4_PCIDE ))) -kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu); +kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu);/* * The TLB has to be flushed for the current PCID if any of the
@@ -1099,7 +1099,7 @@ void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned lon*/ else if (((cr4 ^ old_cr4) & X86_CR4_PAE) || ((cr4 & X86_CR4_SMEP) && !(old_cr4 & X86_CR4_SMEP))) -kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); +kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); }EXPORT_SYMBOL_GPL (kvm_post_set_cr4);
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Sean Christopherson
maintainerPATCH v2 2/7 KVM: x86: Invoke kvm_mmu_unload() directly on CR4.PCIDE change
In reply to Sean ChristophersonReplace a KVM_REQ_MMU_RELOAD request with a direct kvm_mmu_unload() call when the guest's CR4.PCIDE changes. This will allow tweaking the logic of KVM_REQ_MMU_RELOAD to free only obsolete/invalid roots, which is the historical intent of KVM_REQ_MMU_RELOAD. The recent PCIDE behavior is the only user of KVM_REQ_MMU_RELOAD that doesn't mark affected roots as obsolete, needs to unconditionally unload the entire MMU, _and_ affects only the current vCPU.
--- arch/x86/kvm/x86.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/x86.c +1 -1
@@ -1077,7 +1077,7 @@ void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned lon*/ if (!tdp_enabled && (cr4 &X86_CR4_PCIDE ) && !(old_cr4 &X86_CR4_PCIDE )) -kvm_make_request (KVM_REQ_MMU_RELOAD , vcpu); +kvm_mmu_unload (vcpu);/* * The TLB has to be flushed for all PCIDs if any of the following
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Sean Christopherson
maintainerPATCH v2 3/7 KVM: Drop kvm_reload_remote_mmus(), open code request in x86 users
In reply to Sean ChristophersonRemove the generic kvm_reload_remote_mmus() and open code its functionality into the two x86 callers. x86 is (obviously) the only architecture that uses the hook, and is also the only architecture that uses KVM_REQ_MMU_RELOAD in a way that's consistent with the name. That will change in a future patch, as x86's usage when zapping a single shadow page x86 doesn't actually _need_ to reload all vCPUs' MMUs, only MMUs whose root is being zapped actually need to be reloaded.
s390 also uses KVM_REQ_MMU_RELOAD, but for a slightly different purpose.
Drop the generic code in anticipation of implementing s390 and x86 arch specific requests, which will allow dropping KVM_REQ_MMU_RELOAD entirely.
Opportunistically reword the x86 TDP MMU comment to avoid making references to functions (and requests!) when possible, and to remove the rather ambiguous "this".
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 14 +++++++------- include/linux/kvm_host.h | 1 - virt/kvm/kvm_main.c | 5 ----- 3 files changed, 7 insertions(+), 13 deletions(-)
arch/x86/kvm/mmu/mmu.c +7 -7
@@ -2353,7 +2353,7 @@ static bool __kvm_mmu_prepare_zap_page(struct kvm *kvm,* treats invalid shadow pages as being obsolete. */ if (!is_obsolete_sp (kvm, sp)) -kvm_reload_remote_mmus (kvm); +kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_RELOAD ); }if (sp->lpage_disallowed)
@@ -5639,11 +5639,11 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)*/ kvm->arch.mmu_valid_gen = kvm->arch.mmu_valid_gen ?0 :1 ; -/* In order to ensure all threads see this change when -* handling the MMU reload signal, this must happen in the -* same critical section as kvm_reload_remote_mmus, and -* before kvm_zap_obsolete_pages as kvm_zap_obsolete_pages -* could drop the MMU lock and yield. +/* +* In order to ensure all vCPUs drop their soon-to-be invalid roots, +* invalidating TDP MMU roots must be done while holding mmu_lock for +* write and in the same critical section as making the reload request, +* e.g. before kvm_zap_obsolete_pages() could drop mmu_lock and yield. */ if (is_tdp_mmu_enabled (kvm))kvm_tdp_mmu_invalidate_all_roots (kvm);
@@ -5656,7 +5656,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)* Note: we need to do this under the protection of mmu_lock, * otherwise, vcpu would purge shadow page but miss tlb flush. */ -kvm_reload_remote_mmus (kvm); +kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_RELOAD );kvm_zap_obsolete_pages (kvm);
include/linux/kvm_host.h +0 -1
@@ -1325,7 +1325,6 @@ int kvm_vcpu_yield_to(struct kvm_vcpu *target);void kvm_vcpu_on_spin (struct kvm_vcpu *vcpu,bool usermode_vcpu_not_eligible);void kvm_flush_remote_tlbs (struct kvm *kvm); -void kvm_reload_remote_mmus (struct kvm *kvm);#ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE int kvm_mmu_topup_memory_cache (struct kvm_mmu_memory_cache *mc,int min);
virt/kvm/kvm_main.c +0 -5
@@ -354,11 +354,6 @@ void kvm_flush_remote_tlbs(struct kvm *kvm)EXPORT_SYMBOL_GPL (kvm_flush_remote_tlbs);#endif -void kvm_reload_remote_mmus (struct kvm *kvm) -{ -kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_RELOAD ); -} -#ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE static inline void *mmu_memory_cache_alloc_obj (struct kvm_mmu_memory_cache *mc,gfp_t gfp_flags)
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Re: [PATCH v2 3/7] KVM: Drop kvm_reload_remote_mmus(), open code request in x86 users
Quoted 14 lines · Sean Christopherson
Remove the generic
kvm_reload_remote_mmus()and open code its functionality into the two x86 callers. x86 is (obviously) the only architecture that uses the hook, and is also the only architecture that usesKVM_REQ_MMU_RELOADin a way that's consistent with the name. That will change in a future patch, as x86's usage when zapping a single shadow page x86 doesn't actually _need_ to reload all vCPUs' MMUs, only MMUs whose root is being zapped actually need to be reloaded.s390 also uses
KVM_REQ_MMU_RELOAD, but for a slightly different purpose.Drop the generic code in anticipation of implementing s390 and x86 arch specific requests, which will allow dropping
KVM_REQ_MMU_RELOADentirely.
Quoted 7 lines
Opportunistically reword the x86 TDP MMU comment to avoid making references to functions (and requests!) when possible, and to remove the rather ambiguous "this".
No functional change intended.
Cc Ben Gardon <[email protected]>
Quoted 77 lines
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 14 +++++++------- include/linux/kvm_host.h | 1 - virt/kvm/kvm_main.c | 5 ----- 3 files changed, 7 insertions(+), 13 deletions(-)arch/x86/kvm/mmu/mmu.c +7
-7@@ -2353,7 +2353,7 @@ static bool __kvm_mmu_prepare_zap_page(struct kvm *kvm,* treats invalid shadow pages as being obsolete. */ if (!is_obsolete_sp (kvm, sp)) -kvm_reload_remote_mmus (kvm); +kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_RELOAD ); }if (sp->lpage_disallowed)@@ -5639,11 +5639,11 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)*/ kvm->arch.mmu_valid_gen = kvm->arch.mmu_valid_gen ?0 :1 ; -/* In order to ensure all threads see this change when -* handling the MMU reload signal, this must happen in the -* same critical section as kvm_reload_remote_mmus, and -* before kvm_zap_obsolete_pages as kvm_zap_obsolete_pages -* could drop the MMU lock and yield. +/* +* In order to ensure all vCPUs drop their soon-to-be invalid roots, +* invalidating TDP MMU roots must be done while holding mmu_lock for +* write and in the same critical section as making the reload request, +* e.g. before kvm_zap_obsolete_pages() could drop mmu_lock and yield. */ if (is_tdp_mmu_enabled (kvm))kvm_tdp_mmu_invalidate_all_roots (kvm);@@ -5656,7 +5656,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)* Note: we need to do this under the protection of mmu_lock, * otherwise, vcpu would purge shadow page but miss tlb flush. */ -kvm_reload_remote_mmus (kvm); +kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_RELOAD );kvm_zap_obsolete_pages (kvm);include/linux/kvm_host.h +0
-1@@ -1325,7 +1325,6 @@ int kvm_vcpu_yield_to(struct kvm_vcpu *target);void kvm_vcpu_on_spin (struct kvm_vcpu *vcpu,bool usermode_vcpu_not_eligible);void kvm_flush_remote_tlbs (struct kvm *kvm); -void kvm_reload_remote_mmus (struct kvm *kvm);#ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE int kvm_mmu_topup_memory_cache (struct kvm_mmu_memory_cache *mc,int min);virt/kvm/kvm_main.c +0
-6@@ -354,11 +354,6 @@ void kvm_flush_remote_tlbs(struct kvm *kvm)EXPORT_SYMBOL_GPL (kvm_flush_remote_tlbs);#endif -void kvm_reload_remote_mmus (struct kvm *kvm) -{ -kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_RELOAD ); -} -#ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE static inline void *mmu_memory_cache_alloc_obj (struct kvm_mmu_memory_cache *mc,gfp_t gfp_flags) --2.35.1.574.g5d30c73bfb-goog
Sean Christopherson
maintainerPATCH v2 4/7 KVM: x86/mmu: Zap only obsolete roots if a root shadow page is zapped
In reply to Sean ChristophersonZap only obsolete roots when responding to zapping a single root shadow page. Because KVM keeps root_count elevated when stuffing a previous root into its PGD cache, shadowing a 64-bit guest means that zapping any root causes all vCPUs to reload all roots, even if their current root is not affected by the zap.
For many kernels, zapping a single root is a frequent operation, e.g. in Linux it happens whenever an mm is dropped, e.g. process exits, etc...
--- arch/x86/include/asm/kvm_host.h | 2 + arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/mmu/mmu.c | 65 +++++++++++++++++++++++++++++---- arch/x86/kvm/x86.c | 4 +- 4 files changed, 63 insertions(+), 9 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -102,6 +102,8 @@#define KVM_REQ_MSR_FILTER_CHANGED KVM_ARCH_REQ(29)#define KVM_REQ_UPDATE_CPU_DIRTY_LOGGING \ KVM_ARCH_REQ_FLAGS(30, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_MMU_FREE_OBSOLETE_ROOTS \ + KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define CR0_RESERVED_BITS \ (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \
arch/x86/kvm/mmu.h +1 -0
@@ -80,6 +80,7 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code,int kvm_mmu_load (struct kvm_vcpu *vcpu);void kvm_mmu_unload (struct kvm_vcpu *vcpu); +void kvm_mmu_free_obsolete_roots (struct kvm_vcpu *vcpu);void kvm_mmu_sync_roots (struct kvm_vcpu *vcpu);void kvm_mmu_sync_prev_roots (struct kvm_vcpu *vcpu);
arch/x86/kvm/mmu/mmu.c +58 -7
107 lines not on this page · raw message
arch/x86/kvm/x86.c +2 -2
@@ -9856,8 +9856,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) goto out; } } -if (kvm_check_request (KVM_REQ_MMU_RELOAD , vcpu)) -kvm_mmu_unload (vcpu); +if (kvm_check_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)) +kvm_mmu_free_obsolete_roots (vcpu);if (kvm_check_request (KVM_REQ_MIGRATE_TIMER , vcpu))__kvm_migrate_timers (vcpu);if (kvm_check_request (KVM_REQ_MASTERCLOCK_UPDATE , vcpu))
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Re: [PATCH v2 4/7] KVM: x86/mmu: Zap only obsolete roots if a root shadow page is zapped
Quoted 10 lines · Sean Christopherson
Zap only obsolete roots when responding to zapping a single root shadow page. Because KVM keeps
root_countelevated when stuffing a previous root into its PGD cache, shadowing a 64-bit guest means that zapping any root causes all vCPUs to reload all roots, even if their current root is not affected by the zap.For many kernels, zapping a single root is a frequent operation, e.g. in Linux it happens whenever an mm is dropped, e.g. process exits, etc...
Quoted 168 lines
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/include/asm/kvm_host.h | 2 + arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/mmu/mmu.c | 65 +++++++++++++++++++++++++++++---- arch/x86/kvm/x86.c | 4 +- 4 files changed, 63 insertions(+), 9 deletions(-)arch/x86/include/asm/kvm_host.h +2
-0@@ -102,6 +102,8 @@#define KVM_REQ_MSR_FILTER_CHANGED KVM_ARCH_REQ(29)#define KVM_REQ_UPDATE_CPU_DIRTY_LOGGING \ KVM_ARCH_REQ_FLAGS(30, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_MMU_FREE_OBSOLETE_ROOTS \ + KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define CR0_RESERVED_BITS \ (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \arch/x86/kvm/mmu.h +1
-0@@ -80,6 +80,7 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code,int kvm_mmu_load (struct kvm_vcpu *vcpu);void kvm_mmu_unload (struct kvm_vcpu *vcpu); +void kvm_mmu_free_obsolete_roots (struct kvm_vcpu *vcpu);void kvm_mmu_sync_roots (struct kvm_vcpu *vcpu);void kvm_mmu_sync_prev_roots (struct kvm_vcpu *vcpu);arch/x86/kvm/mmu/mmu.c +58
-7@@ -2310,7 +2310,7 @@ static bool __kvm_mmu_prepare_zap_page(struct kvm *kvm,struct list_head *invalid_list, int *nr_zapped) { - bool list_unstable; + bool list_unstable, zapped_root = false;trace_kvm_mmu_prepare_zap_page (sp); ++kvm->stat.mmu_shadow_zapped;@@ -2352,14 +2352,20 @@ static bool __kvm_mmu_prepare_zap_page(struct kvm *kvm,* in kvm_mmu_zap_all_fast(). Note, is_obsolete_sp() also * treats invalid shadow pages as being obsolete. */ -if (!is_obsolete_sp (kvm, sp)) -kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_RELOAD ); + zapped_root = !is_obsolete_sp (kvm, sp); }if (sp->lpage_disallowed)unaccount_huge_nx_page (kvm, sp); sp->role.invalid =1 ; + +/* +* Make the request to free obsolete roots after marking the root +* invalid, otherwise other vCPUs may not see it as invalid. +*/ +if (zapped_root) +kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_FREE_OBSOLETE_ROOTS );return list_unstable; }@@ -3947,7 +3953,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,* previous root, then __kvm_mmu_prepare_zap_page() signals all vCPUs * to reload even if no vCPU is actively using the root. */ -if (!sp &&kvm_test_request (KVM_REQ_MMU_RELOAD , vcpu)) +if (!sp &&kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu))return true; return fault->slot &&@@ -4180,8 +4186,8 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd)/* * It's possible that the cached previous root page is obsolete because * of a change in the MMU generation number. However, changing the -* generation number is accompanied by KVM_REQ_MMU_RELOAD, which will -* free the root set here and allocate a new one. +* generation number is accompanied by KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, +* which will free the root set here and allocate a new one. */ kvm_make_request (KVM_REQ_LOAD_MMU_PGD , vcpu);@@ -5085,6 +5091,51 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu)vcpu_clear_mmio_info (vcpu,MMIO_GVA_ANY ); } +static bool is_obsolete_root (struct kvm *kvm,hpa_t root_hpa) +{ +struct kvm_mmu_page *sp; + +if (!VALID_PAGE (root_hpa)) +return false; + +/* +* When freeing obsolete roots, treat roots as obsolete if they don't +* have an associated shadow page. This does mean KVM will get false +* positives and free roots that don't strictly need to be freed, but +* such false positives are relatively rare: +* +* (a) only PAE paging and nested NPT has roots without shadow pages +* (b) remote reloads due to a memslot update obsoletes _all_ roots +* (c) KVM doesn't track previous roots for PAE paging, and the guest +* is unlikely to zap an in-use PGD. +*/ + sp =to_shadow_page (root_hpa); +return !sp ||is_obsolete_sp (kvm, sp); +} + +static void __kvm_mmu_free_obsolete_roots (struct kvm *kvm,struct kvm_mmu *mmu) +{ +unsigned long roots_to_free =0 ; +int i; + +if (is_obsolete_root (kvm, mmu->root.hpa)) + roots_to_free |=KVM_MMU_ROOT_CURRENT ; + +for (i =0 ; i <KVM_MMU_NUM_PREV_ROOTS ; i++) { +if (is_obsolete_root (kvm, mmu->root.hpa)) + roots_to_free |=KVM_MMU_ROOT_PREVIOUS (i); + } + +if (roots_to_free) +kvm_mmu_free_roots (kvm, mmu, roots_to_free); +} + +void kvm_mmu_free_obsolete_roots (struct kvm_vcpu *vcpu) +{ +__kvm_mmu_free_obsolete_roots (vcpu->kvm, &vcpu->arch.root_mmu); +__kvm_mmu_free_obsolete_roots (vcpu->kvm, &vcpu->arch.guest_mmu); +} +static bool need_remote_flush (u64 old,u64 new) {if (!is_shadow_present_pte (old))@@ -5656,7 +5707,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)* Note: we need to do this under the protection of mmu_lock, * otherwise, vcpu would purge shadow page but miss tlb flush. */ -kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_RELOAD ); +kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_FREE_OBSOLETE_ROOTS );kvm_zap_obsolete_pages (kvm);arch/x86/kvm/x86.c +2
-3@@ -9856,8 +9856,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) goto out; } } -if (kvm_check_request (KVM_REQ_MMU_RELOAD , vcpu)) -kvm_mmu_unload (vcpu); +if (kvm_check_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)) +kvm_mmu_free_obsolete_roots (vcpu);if (kvm_check_request (KVM_REQ_MIGRATE_TIMER , vcpu))__kvm_migrate_timers (vcpu);if (kvm_check_request (KVM_REQ_MASTERCLOCK_UPDATE , vcpu)) --2.35.1.574.g5d30c73bfb-goog
Paolo Bonzini
maintainerRe: [PATCH v2 4/7] KVM: x86/mmu: Zap only obsolete roots if a root shadow page is zapped
In reply to Sean Christophersonarch/x86/kvm/mmu/mmu.c static void kvm_mmu_zap_all_fast(struct kvm *kvm) · line 5711
Quoted 7 lines · Sean Christopherson
arch/x86/kvm/mmu/mmu.c +1
-1@@ -5656,7 +5707,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)* Note: we need to do this under the protection of mmu_lock, * otherwise, vcpu would purge shadow page but miss tlb flush. */ -kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_RELOAD ); +kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_FREE_OBSOLETE_ROOTS );
I was going to squash in this:
* invalidating TDP MMU roots must be done while holding mmu_lock for
- * write and in the same critical section as making the reload request,
+ * write and in the same critical section as making the free request,
* e.g. before kvm_zap_obsolete_pages() could drop mmu_lock and yield.
But then I realized that this needs better comments and that my knowledge of this has serious holes. Regarding this comment, this is my proposal:
/*
* Invalidated TDP MMU roots are zapped within MMU read_lock to be
* able to walk the list of roots, but with the expectation of no
* concurrent change to the pages themselves. There cannot be
* any yield between kvm_tdp_mmu_invalidate_all_roots and the free
* request, otherwise somebody could grab a reference to the root
* and break that assumption.
*/
if (is_tdp_mmu_enabled(kvm))
kvm_tdp_mmu_invalidate_all_roots(kvm);
However, for the second comment (the one in the context above), there's much more. From easier to harder:
1) I'm basically clueless about the TLB flush "note" above.
2) It's not clear to me what needs to use for_each_tdp_mmu_root; for example, why would anything but the MMU notifiers use for_each_tdp_mmu_root? It is used in kvm_tdp_mmu_write_protect_gfn, kvm_tdp_mmu_try_split_huge_pages and kvm_tdp_mmu_clear_dirty_pt_masked.
3) Does it make sense that yielding users of for_each_tdp_mmu_root must either look at valid roots only, or take MMU lock for write? If so, can this be enforced in tdp_mmu_next_root?
4) If the previous point is correct, _who_ could grab a reference and not release it before kvm_tdp_mmu_zap_invalidated_roots runs? That is, is "somebody could grab a reference" an accurate explanation in the first comment above?
Thanks,
Paolo
Paolo Bonzini
maintainerRe: [PATCH v2 4/7] KVM: x86/mmu: Zap only obsolete roots if a root shadow page is zapped
Quoted 36 lines · Paolo Bonzini
Review static void kvm_mmu_zap_all_fast(struct kvm *kvm) · line 5710
Quoted 6 lines · Sean Christopherson
(quoted diff) +1
-1@@ -5656,7 +5707,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) * Note: we need to do this under the protection of mmu_lock, * otherwise, vcpu would purge shadow page but miss tlb flush. */ - kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD); + kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_FREE_OBSOLETE_ROOTS);I was going to squash in this:
* invalidating TDP MMU roots must be done while holding
mmu_lockfor
- * write and in the same critical section as making the reload request,
+ * write and in the same critical section as making the free request,
* e.g. beforekvm_zap_obsolete_pages()could dropmmu_lockand
yield.But then I realized that this needs better comments and that my knowledge of this has serious holes. Regarding this comment, this is my proposal:
/*
* Invalidated TDP MMU roots are zapped within MMUread_lockto be
* able to walk the list of roots, but with the expectation of no
* concurrent change to the pages themselves. There cannot be
* any yield betweenkvm_tdp_mmu_invalidate_all_rootsand the free
* request, otherwise somebody could grab a reference to the root
* and break that assumption.
*/
if (is_tdp_mmu_enabled(kvm))kvm_tdp_mmu_invalidate_all_roots(kvm);However, for the second comment (the one in the context above), there's much more. From easier to harder:
1) I'm basically clueless about the TLB flush "note" above.
Quoted 8 lines
2) It's not clear to me what needs to use
for_each_tdp_mmu_root; for example, why would anything but the MMU notifiers usefor_each_tdp_mmu_root? It is used inkvm_tdp_mmu_write_protect_gfn,kvm_tdp_mmu_try_split_huge_pagesandkvm_tdp_mmu_clear_dirty_pt_masked.3) Does it make sense that yielding users of
for_each_tdp_mmu_rootmust either look at valid roots only, or take MMU lock for write? If so, can this be enforced intdp_mmu_next_root?
Ok, I could understand this a little better now, but please correct me if this is incorrect:
2) if I'm not wrong, kvm_tdp_mmu_try_split_huge_pages indeed does not need to walk invalid roots. The others do because the TDP MMU does not necessarily kick vCPUs after marking roots as invalid. But because TDP MMU roots are gone for good once their refcount hits 0, I wonder if we could do something like
arch/x86/kvm/mmu/tdp_mmu.c +6 -1
@@ -164,6 +164,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,*/ if (!kvm_tdp_root_mark_invalid (root)) {refcount_set (&root->tdp_mmu_root_count,1 ); +kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_FREE_OBSOLETE_ROOTS );/* * If the struct kvm is alive, we might as well zap the root
@@ -1099,12 +1100,16 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm)void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm) {struct kvm_mmu_page *root; + bool invalidated_root = falselockdep_assert_held_write (&kvm->mmu_lock);list_for_each_entry (root, &kvm->arch.tdp_mmu_roots, link) {if (!WARN_ON_ONCE (!kvm_tdp_mmu_get_root (root))) - root->role.invalid = true; + invalidated_root |= !kvm_tdp_root_mark_invalid (root); } + +if (invalidated_root) +kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_FREE_OBSOLETE_ROOTS ); }/*
(based on my own version of Sean's patches) and stop walking invalid roots in kvm_tdp_mmu_write_protect_gfn and kvm_tdp_mmu_clear_dirty_pt_masked.
3) Yes, it makes sense that yielding users of for_each_tdp_mmu_root must either look at valid roots only, or take MMU lock for write. The only exception is kvm_tdp_mmu_try_split_huge_pages, which does not need to walk invalid roots. And kvm_tdp_mmu_zap_invalidated_pages(), but that one is basically an asynchronous worker [and this is where I had the inspiration to get rid of the function altogether]
Paolo
Sean Christopherson
maintainerRe: [PATCH v2 4/7] KVM: x86/mmu: Zap only obsolete roots if a root shadow page is zapped
Quoted 33 lines · Paolo Bonzini
Quoted 32 lines · Paolo Bonzini
Review static void kvm_mmu_zap_all_fast(struct kvm *kvm) · line 5710
Quoted 6 lines · Sean Christopherson
(quoted diff) +1
-1@@ -5656,7 +5707,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) * Note: we need to do this under the protection of mmu_lock, * otherwise, vcpu would purge shadow page but miss tlb flush. */ - kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD); + kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_FREE_OBSOLETE_ROOTS);I was going to squash in this:
* invalidating TDP MMU roots must be done while holding
mmu_lockfor
- * write and in the same critical section as making the reload request,
+ * write and in the same critical section as making the free request,
* e.g. beforekvm_zap_obsolete_pages()could dropmmu_lockand
yield.But then I realized that this needs better comments and that my knowledge of
this has serious holes. Regarding this comment, this is my proposal:/*
* Invalidated TDP MMU roots are zapped within MMUread_lockto be
* able to walk the list of roots, but with the expectation of no
* concurrent change to the pages themselves. There cannot be
* any yield betweenkvm_tdp_mmu_invalidate_all_rootsand the free
* request, otherwise somebody could grab a reference to the root
* and break that assumption.
*/
if (is_tdp_mmu_enabled(kvm))kvm_tdp_mmu_invalidate_all_roots(kvm);
Quoted 4 lines
Quoted 4 lines
However, for the second comment (the one in the context above), there's much more. From easier to harder:
1) I'm basically clueless about the TLB flush "note" above.
I assume you're referring to this ancient thing?
* Note: we need to do this under the protection of mmu_lock,
* otherwise, vcpu would purge shadow page but miss tlb flush.
The "vcpu" part should be "KVM", or more precisely kvm_zap_obsolete_pages(). The fast zap (not a vCPU) will drop mmu_lock() if it yields when "preparing" the zap, so the remote TLB flush via the kvm_mmu_commit_zap_page() is too late.
Quoted 11 lines
Quoted 10 lines
2) It's not clear to me what needs to use
for_each_tdp_mmu_root; for example, why would anything but the MMU notifiers usefor_each_tdp_mmu_root?
It is used inkvm_tdp_mmu_write_protect_gfn,kvm_tdp_mmu_try_split_huge_pages
andkvm_tdp_mmu_clear_dirty_pt_masked.3) Does it make sense that yielding users of
for_each_tdp_mmu_rootmust either look at valid roots only, or take MMU lock for write? If so, can this be enforced intdp_mmu_next_root?
Quoted 5 lines
Ok, I could understand this a little better now, but please correct me if this is incorrect:
2) if I'm not wrong,
kvm_tdp_mmu_try_split_huge_pagesindeed does not need to walk invalid roots.
Correct, it doesn't need to walk invalid roots. The only flows that need to walk invalid roots are the mmu_notifiers (or kvm_arch_flush_shadow_all() if KVM x86 were somehow able to survive without notifiers).
Quoted 2 lines
The others do because the TDP MMU does not necessarily kick vCPUs after marking roots as invalid.
Fudge. I'm pretty sure AMD/SVM TLB management is broken for the TDP MMU (though I would argue that KVM's ASID management is broken regardless of the TDP MMU...).
The notifiers need to walk all roots because they need to guarantee any metadata accounting, e.g. propagation of dirty bits, for the associated (host) pfn occurs before the notifier returns. It's not an issue of vCPUs having stale references, or at least it shouldn't be, it's an issue of the "writeback" occurring after the pfn is full released.
In the "fast zap", the KVM always kicks vCPUs after marking them invalid, before dropping mmu_lock (which is held for write). This is mandatory because the memslot is being deleted/moved, so KVM must guarantee the old slot can't be accessed by the guest.
In the put_root() path, there _shouldn't_ be a need to kick because the vCPU doesn't have a reference to the root, and the last vCPU to drop a reference to the root _should_ ensure it's unreachable.
Intel EPT is fine, because the EPT4A ensures a unique ASID, i.e. KVM can defer any TLB flush until the same physical root page is reused.
Shadow paging is fine because kvm_mmu_free_roots()'s call to kvm_mmu_commit_zap_page() will flush TLBs for all vCPUs when the last reference is put.
AMD NPT is hosed because KVM's awful ASID scheme doesn't assign an ASID per root and doesn't force a new ASID. IMO, this is an SVM mess and not a TDP MMU bug. In the short term, I think something like the following would suffice. Long term, we really need to redo SVM ASID management so that ASIDs are tied to a KVM root.
arch/x86/kvm/mmu/tdp_mmu.h +7 -4
@@ -70,11 +70,8 @@ bool kvm_mmu_init_tdp_mmu(struct kvm *kvm);void kvm_mmu_uninit_tdp_mmu (struct kvm *kvm);static inline bool is_tdp_mmu_page (struct kvm_mmu_page *sp) {return sp->tdp_mmu_page; } -static inline bool is_tdp_mmu (struct kvm_mmu *mmu) +static inline bool is_tdp_mmu_root (hpa_t hpa) { -struct kvm_mmu_page *sp; -hpa_t hpa = mmu->root.hpa; -if (WARN_ON (!VALID_PAGE (hpa)))return false;
@@ -86,10 +83,16 @@ static inline bool is_tdp_mmu(struct kvm_mmu *mmu) sp =to_shadow_page (hpa); return sp && is_tdp_mmu_page(sp) && sp->root_count; } + +static inline bool is_tdp_mmu (struct kvm_mmu *mmu) +{ +return is_tdp_mmu_root (mmu->root.hpa); +}#else static inline bool kvm_mmu_init_tdp_mmu (struct kvm *kvm) {return false; }static inline void kvm_mmu_uninit_tdp_mmu (struct kvm *kvm) {}static inline bool is_tdp_mmu_page (struct kvm_mmu_page *sp) {return false; } +static inline bool is_tdp_mmu_root (hpa_t hpa) {return false; }static inline bool is_tdp_mmu (struct kvm_mmu *mmu) {return false; }#endif
arch/x86/kvm/svm/svm.c +3 -0
@@ -3857,6 +3857,9 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,unsigned long cr3;if (npt_enabled) { +if (is_tdp_mmu_root (root_hpa)) + svm->current_vmcb->asid_generation =0 ; + svm->vmcb->control.nested_cr3 =__sme_set (root_hpa);vmcb_mark_dirty (svm->vmcb,VMCB_NPT );
Quoted 19 lines
But because TDP MMU roots are gone for good once their refcount hits 0, I wonder if we could do something like
arch/x86/kvm/mmu/tdp_mmu.c +2
-0@@ -164,6 +164,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,*/ if (!kvm_tdp_root_mark_invalid (root)) {refcount_set (&root->tdp_mmu_root_count,1 ); +kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_FREE_OBSOLETE_ROOTS );/* * If the struct kvm is alive, we might as well zap the root @@ -1099,12 +1100,16 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm)void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm) {struct kvm_mmu_page *root; +bool invalidated_root = false
arch/x86/kvm/mmu/tdp_mmu.c void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm) · line 1112
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-1lockdep_assert_held_write (&kvm->mmu_lock);list_for_each_entry (root, &kvm->arch.tdp_mmu_roots, link) {if (!WARN_ON_ONCE (!kvm_tdp_mmu_get_root (root))) - root->role.invalid = true; + invalidated_root |= !kvm_tdp_root_mark_invalid (root); } + +if (invalidated_root) +kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_FREE_OBSOLETE_ROOTS ); }
This won't work, see my other response about not being able to use a worker for this path (my brain is finally getting up to speed today...).
Quoted 10 lines
3) Yes, it makes sense that yielding users of
for_each_tdp_mmu_rootmust either look at valid roots only, or take MMU lock for write. The only exception iskvm_tdp_mmu_try_split_huge_pages, which does not need to walk invalid roots. Andkvm_tdp_mmu_zap_invalidated_pages(), but that one is basically an asynchronous worker [and this is where I had the inspiration to get rid of the function altogether]Paolo
Paolo Bonzini
maintainerRe: [PATCH v2 4/7] KVM: x86/mmu: Zap only obsolete roots if a root shadow page is zapped
Quoted 2 lines · Sean Christopherson
AMD NPT is hosed because KVM's awful ASID scheme doesn't assign an ASID per root and doesn't force a new ASID. IMO, this is an SVM mess and not a TDP MMU bug.
I agree.
Quoted 2 lines
In the short term, I think something like the following would suffice. Long term, we really need to redo SVM ASID management so that ASIDs are tied to a KVM root.
arch/x86/kvm/svm/svm.c +3 -0
@@ -3857,6 +3857,9 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,unsigned long cr3;if (npt_enabled) { +if (is_tdp_mmu_root (root_hpa)) + svm->current_vmcb->asid_generation =0 ; + svm->vmcb->control.nested_cr3 =__sme_set (root_hpa);vmcb_mark_dirty (svm->vmcb,VMCB_NPT );
Why not just new_asid (even unconditionally, who cares)?
BTW yeah, the smoke test worked but the actual one failed horribly.
Paolo
Sean Christopherson
maintainerRe: [PATCH v2 4/7] KVM: x86/mmu: Zap only obsolete roots if a root shadow page is zapped
Quoted 25 lines · Paolo Bonzini
Quoted 2 lines · Sean Christopherson
AMD NPT is hosed because KVM's awful ASID scheme doesn't assign an ASID per root and doesn't force a new ASID. IMO, this is an SVM mess and not a TDP MMU bug.
I agree.
Quoted 2 lines
In the short term, I think something like the following would suffice. Long term, we really need to redo SVM ASID management so that ASIDs are tied to a KVM root.
arch/x86/kvm/svm/svm.c +3
-0@@ -3857,6 +3857,9 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu,hpa_t root_hpa, unsigned long cr3;if (npt_enabled) { +if (is_tdp_mmu_root (root_hpa)) + svm->current_vmcb->asid_generation =0 ; + svm->vmcb->control.nested_cr3 =__sme_set (root_hpa);vmcb_mark_dirty (svm->vmcb,VMCB_NPT );
Quoted 1 line
Why not just
new_asid
My mental coin flip came up tails? new_asid() is definitely more intuitive.
Quoted 1 line
(even unconditionally, who cares)?
Heh, I was going to say we do care to some extent for nested transitions, then I remembered we flush on every nested transition anyways, in no small part because the ASID handling is a mess.
Paolo Bonzini
maintainerRe: [PATCH v2 4/7] KVM: x86/mmu: Zap only obsolete roots if a root shadow page is zapped
Quoted 16 lines · Sean Christopherson
Quoted 16 lines
arch/x86/kvm/svm/svm.c +3
-0@@ -3857,6 +3857,9 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu,hpa_t root_hpa, unsigned long cr3;if (npt_enabled) { +if (is_tdp_mmu_root (root_hpa)) + svm->current_vmcb->asid_generation =0 ; + svm->vmcb->control.nested_cr3 =__sme_set (root_hpa);vmcb_mark_dirty (svm->vmcb,VMCB_NPT );
Quoted 3 lines
Quoted 1 line
Why not just
new_asidMy mental coin flip came up tails?
new_asid()is definitely more intuitive.
Can you submit a patch (seems like 5.17+stable material)?
Paolo
Sean Christopherson
maintainerRe: [PATCH v2 4/7] KVM: x86/mmu: Zap only obsolete roots if a root shadow page is zapped
Quoted 17 lines · Paolo Bonzini
Quoted 16 lines · Sean Christopherson
Quoted 16 lines
arch/x86/kvm/svm/svm.c +3
-0@@ -3857,6 +3857,9 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu,hpa_t root_hpa, unsigned long cr3;if (npt_enabled) { +if (is_tdp_mmu_root (root_hpa)) + svm->current_vmcb->asid_generation =0 ; + svm->vmcb->control.nested_cr3 =__sme_set (root_hpa);vmcb_mark_dirty (svm->vmcb,VMCB_NPT );
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Why not just
new_asidMy mental coin flip came up tails?
new_asid()is definitely more intuitive.Can you submit a patch (seems like 5.17+stable material)?
After a lot more thinking, there's no bug. If KVM unloads all roots, e.g. fast zap, then all vCPUs are guaranteed to go through kvm_mmu_load(), and that will flush the current ASID.
So the only problematic path is KVM_REQ_LOAD_MMU_PGD, which has two users, kvm_mmu_new_pgd() and load_pdptrs(). load_pdptrs() is benign because it triggers a "false" PGD load only top get PDPTRs updated on EPT, the actual PGD doesn't change (or rather isn't forced to change by load_pdptrs().
Nested SVM's use of kvm_mmu_new_pgd() is "ok" because KVM currently flushes on every transition.
That leaves kvm_set_cr3() via kvm_mmu_new_pgd(). For NPT, lack of a flush is moot because KVM shouldn't be loading a new PGD in the first place (see our other discussion about doing:
arch/x86/kvm/x86.c +1 -1
@@ -1212,7 +1212,7 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)if (is_pae_paging (vcpu) && !load_pdptrs (vcpu, cr3))return 1 ; -if (cr3 !=kvm_read_cr3 (vcpu)) +if (!tdp_enabled && cr3 !=kvm_read_cr3 (vcpu))kvm_mmu_new_pgd (vcpu, cr3); vcpu->arch.cr3 = cr3;
Non-NPT shadow paging is ok because either the MOV CR3 will do a TLB flush, or the guest explicitly says "don't do a TLB flush", in which case KVM is off the hook from a correctness perspective (guest's responsibility to ensure MMU in sync'd), and is ok from a safety perspective because the legacy MMU does a remote TLB flush if it zaps any pages, i.e. the guest can't do use-after-free.
All that said, this is another argument against dropping kvm_mmu_unload() from kvm_mmu_reset_context()[*]: SMM would theoretically be broken on AMD due to reusing the same ASID for both non-SMM and SMM roots/memslots.
In practice, I don't think it can actually happen, but that's mostly dumb luck. em_rsm() temporarily transitions back to Real Mode before loading the actual non-SMM guest state, so only SMI that arrives with CR0.PG=0 is problematic. In that case, TLB flushes may not be triggered by kvm_set_cr0() or kvm_set_cr4(), but kvm_set_cr3() will always trigger a flush because the "no flush" PCID bit will always be clear. Well, unless the SMM handler writes the read-only SMRAM field, at which point it deserves to die :-)
Anyways, before we transitions SMM away from kvm_mmu_reset_context(), we should add an explicit KVM_REQ_TLB_FLUSH_CURRENT in svm_{enter,leave}_smm(), with a TODO similar to nested_svm_transition_tlb_flush() to document that the explicit flush can go away when KVM ensures unique ASIDs for non-SMM vs. SMM.
Sean Christopherson
maintainerPATCH v2 5/7 KVM: s390: Replace KVM_REQ_MMU_RELOAD usage with arch specific request
In reply to Sean ChristophersonAdd an arch request, KVM_REQ_REFRESH_GUEST_PREFIX, to deal with guest prefix changes instead of piggybacking KVM_REQ_MMU_RELOAD. This will allow for the removal of the generic KVM_REQ_MMU_RELOAD, which isn't actually used by generic KVM.
No functional change intended.
--- arch/s390/include/asm/kvm_host.h | 2 ++ arch/s390/kvm/kvm-s390.c | 8 ++++---- arch/s390/kvm/kvm-s390.h | 2 +- 3 files changed, 7 insertions(+), 5 deletions(-)
arch/s390/include/asm/kvm_host.h +2 -0
@@ -45,6 +45,8 @@#define KVM_REQ_START_MIGRATION KVM_ARCH_REQ(3)#define KVM_REQ_STOP_MIGRATION KVM_ARCH_REQ(4)#define KVM_REQ_VSIE_RESTART KVM_ARCH_REQ(5) +#define KVM_REQ_REFRESH_GUEST_PREFIX \ + KVM_ARCH_REQ_FLAGS(6, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define SIGP_CTRL_C 0x80#define SIGP_CTRL_SCN_MASK 0x3f
arch/s390/kvm/kvm-s390.c +4 -4
@@ -3394,7 +3394,7 @@ static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start,if (prefix <= end && start <= prefix +2 *PAGE_SIZE -1 ) {VCPU_EVENT (vcpu,2 ,"gmap notifier for %lx-%lx" , start, end); -kvm_s390_sync_request (KVM_REQ_MMU_RELOAD , vcpu); +kvm_s390_sync_request (KVM_REQ_REFRESH_GUEST_PREFIX , vcpu); } } }
@@ -3796,19 +3796,19 @@ static int kvm_s390_handle_requests(struct kvm_vcpu *vcpu)if (!kvm_request_pending (vcpu))return 0 ;/* -* We use MMU_RELOAD just to re-arm the ipte notifier for the +* If the guest prefix changed, re-arm the ipte notifier for the * guest prefix page. gmap_mprotect_notify will wait on the ptl lock. * This ensures that the ipte instruction for this request has * already finished. We might race against a second unmapper that * wants to set the blocking bit. Lets just retry the request loop. */ -if (kvm_check_request (KVM_REQ_MMU_RELOAD , vcpu)) { +if (kvm_check_request (KVM_REQ_REFRESH_GUEST_PREFIX , vcpu)) {int rc; rc =gmap_mprotect_notify (vcpu->arch.gmap,kvm_s390_get_prefix (vcpu),PAGE_SIZE *2 ,PROT_WRITE );if (rc) { -kvm_make_request (KVM_REQ_MMU_RELOAD , vcpu); +kvm_make_request (KVM_REQ_REFRESH_GUEST_PREFIX , vcpu);return rc; } goto retry;
arch/s390/kvm/kvm-s390.h +1 -1
@@ -105,7 +105,7 @@ static inline void kvm_s390_set_prefix(struct kvm_vcpu *vcpu, u32 prefix) prefix); vcpu->arch.sie_block->prefix = prefix >>GUEST_PREFIX_SHIFT ;kvm_make_request (KVM_REQ_TLB_FLUSH , vcpu); -kvm_make_request (KVM_REQ_MMU_RELOAD , vcpu); +kvm_make_request (KVM_REQ_REFRESH_GUEST_PREFIX , vcpu); }static inline u64 kvm_s390_get_base_disp_s (struct kvm_vcpu *vcpu,u8 *ar)
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Sean Christopherson
maintainerPATCH v2 6/7 KVM: Drop KVM_REQ_MMU_RELOAD and update vcpu-requests.rst documentation
In reply to Sean ChristophersonRemove the now unused KVM_REQ_MMU_RELOAD, shift KVM_REQ_VM_DEAD into the unoccupied space, and update vcpu-requests.rst, which was missing an entry for KVM_REQ_VM_DEAD. Switching KVM_REQ_VM_DEAD to entry '1' also fixes the stale comment about bits 4-7 being reserved.
--- Documentation/virt/kvm/vcpu-requests.rst | 7 +++---- include/linux/kvm_host.h | 3 +-- 2 files changed, 4 insertions(+), 6 deletions(-)
Documentation/virt/kvm/vcpu-requests.rst +3 -4
@@ -112,11 +112,10 @@ KVM_REQ_TLB_FLUSH choose to use the common kvm_flush_remote_tlbs() implementation will need to handle this VCPU request. -KVM_REQ_MMU_RELOAD +KVM_REQ_VM_DEAD - When shadow page tables are used and memory slots are removed it's - necessary to inform each VCPU to completely refresh the tables. This - request is used for that. + This request informs all VCPUs that the VM is dead and unusable, e.g. due to + fatal error or because the VM's state has been intentionally destroyed. KVM_REQ_UNBLOCK
include/linux/kvm_host.h +1 -2
@@ -153,10 +153,9 @@ static inline bool is_error_page(struct page *page)* Bits 4-7 are reserved for more arch-independent bits. */ #define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) -#define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_VM_DEAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_UNBLOCK 2#define KVM_REQ_UNHALT 3 -#define KVM_REQ_VM_DEAD (4 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_GPC_INVALIDATE (5 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQUEST_ARCH_BASE 8
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Re: [PATCH v2 6/7] KVM: Drop KVM_REQ_MMU_RELOAD and update vcpu-requests.rst documentation
Quoted 7 lines · Sean Christopherson
Remove the now unused
KVM_REQ_MMU_RELOAD, shiftKVM_REQ_VM_DEADinto the unoccupied space, and updatevcpu-requests.rst, which was missing an entry forKVM_REQ_VM_DEAD. SwitchingKVM_REQ_VM_DEADto entry '1' also fixes the stale comment about bits 4-7 being reserved.Reviewed-by Claudio Imbrenda <[email protected]>
Quoted 44 lines
Signed-off-by Sean Christopherson <[email protected]>--- Documentation/virt/kvm/vcpu-requests.rst | 7 +++---- include/linux/kvm_host.h | 3 +-- 2 files changed, 4 insertions(+), 6 deletions(-)Documentation/virt/kvm/vcpu-requests.rst +3
-4@@ -112,11 +112,10 @@ KVM_REQ_TLB_FLUSH choose to use the common kvm_flush_remote_tlbs() implementation will need to handle this VCPU request. -KVM_REQ_MMU_RELOAD +KVM_REQ_VM_DEAD - When shadow page tables are used and memory slots are removed it's - necessary to inform each VCPU to completely refresh the tables. This - request is used for that. + This request informs all VCPUs that the VM is dead and unusable, e.g. due to + fatal error or because the VM's state has been intentionally destroyed. KVM_REQ_UNBLOCKinclude/linux/kvm_host.h +1
-3@@ -153,10 +153,9 @@ static inline bool is_error_page(struct page *page)* Bits 4-7 are reserved for more arch-independent bits. */ #define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) -#define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_VM_DEAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_UNBLOCK 2#define KVM_REQ_UNHALT 3 -#define KVM_REQ_VM_DEAD (4 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_GPC_INVALIDATE (5 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQUEST_ARCH_BASE 8 --2.35.1.574.g5d30c73bfb-goog
Sean Christopherson
maintainerPATCH v2 7/7 KVM: WARN if is_unsync_root() is called on a root without a shadow page
In reply to Sean ChristophersonWARN and bail if is_unsync_root() is passed a root for which there is no shadow page, i.e. is passed the physical address of one of the special roots, which do not have an associated shadow page. The current usage squeaks by without bug reports because neither kvm_mmu_sync_roots() nor kvm_mmu_sync_prev_roots() calls the helper with pae_root or pml4_root, and 5-level AMD CPUs are not generally available, i.e. no one can coerce KVM into calling is_unsync_root() on pml5_root.
Note, this doesn't fix the mess with 5-level nNPT, it just (hopefully) prevents KVM from crashing.
--- arch/x86/kvm/mmu/mmu.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/kvm/mmu/mmu.c +8 -0
@@ -3634,6 +3634,14 @@ static bool is_unsync_root(hpa_t root)*/ smp_rmb (); sp =to_shadow_page (root); + +/* +* PAE roots (somewhat arbitrarily) aren't backed by shadow pages, the +* PDPTEs for a given PAE root need to be synchronized individually. +*/ +if (WARN_ON_ONCE (!sp)) +return false; +if (sp->unsync || sp->unsync_children)return true;
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Re: [PATCH v2 7/7] KVM: WARN if is_unsync_root() is called on a root without a shadow page
Quoted 24 lines · Sean Christopherson
WARN and bail if
is_unsync_root()is passed a root for which there is no shadow page, i.e. is passed the physical address of one of the special roots, which do not have an associated shadow page. The current usage squeaks by without bug reports because neitherkvm_mmu_sync_roots()norkvm_mmu_sync_prev_roots()calls the helper withpae_rootorpml4_root, and 5-level AMD CPUs are not generally available, i.e. no one can coerce KVM into callingis_unsync_root()onpml5_root.Note, this doesn't fix the mess with 5-level nNPT, it just (hopefully) prevents KVM from crashing.
Cc Lai Jiangshan <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 8 ++++++++ 1 file changed, 8 insertions(+)arch/x86/kvm/mmu/mmu.c +0
-0@@ -3634,6 +3634,14 @@ static bool is_unsync_root(hpa_t root)*/
arch/x86/kvm/mmu/mmu.c static bool is_unsync_root(hpa_t root) · line 3644
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +8
-0smp_rmb (); sp =to_shadow_page (root); + +/* +* PAE roots (somewhat arbitrarily) aren't backed by shadow pages, the +* PDPTEs for a given PAE root need to be synchronized individually. +*/ +if (WARN_ON_ONCE (!sp)) +return false; +
I was trying to figure out if this should be returning true or false, but neither really seems correct. Since we never expect this to fire, perhaps it doesn't matter and it's easier to just return false so the callers don't need to be changed. If this did fire in a production scenario, I'd want it to terminate the VM too.
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +0
-1if (sp->unsync || sp->unsync_children)return true; --2.35.1.574.g5d30c73bfb-goog
Sean Christopherson
maintainerRe: [PATCH v2 7/7] KVM: WARN if is_unsync_root() is called on a root without a shadow page
Quoted 32 lines · Ben Gardon
Quoted 31 lines · Sean Christopherson
WARN and bail if
is_unsync_root()is passed a root for which there is no shadow page, i.e. is passed the physical address of one of the special roots, which do not have an associated shadow page. The current usage squeaks by without bug reports because neitherkvm_mmu_sync_roots()norkvm_mmu_sync_prev_roots()calls the helper withpae_rootorpml4_root, and 5-level AMD CPUs are not generally available, i.e. no one can coerce KVM into callingis_unsync_root()onpml5_root.Note, this doesn't fix the mess with 5-level nNPT, it just (hopefully) prevents KVM from crashing.
Cc Lai Jiangshan <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 8 ++++++++ 1 file changed, 8 insertions(+)arch/x86/kvm/mmu/mmu.c +5
-0@@ -3634,6 +3634,14 @@ static bool is_unsync_root(hpa_t root)*/ smp_rmb (); sp =to_shadow_page (root); + +/* +* PAE roots (somewhat arbitrarily) aren't backed by shadow pages, the +* PDPTEs for a given PAE root need to be synchronized individually. +*/
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c static bool is_unsync_root(hpa_t root) · line 3644
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3
-0+if (WARN_ON_ONCE (!sp)) +return false; +I was trying to figure out if this should be returning true or false, but neither really seems correct. Since we never expect this to fire, perhaps it doesn't matter and it's easier to just return false so the callers don't need to be changed.
Yep, neither is correct.
Quoted 2 lines
If this did fire in a production scenario, I'd want it to terminate the VM too.
Me too, but practically speaking this should never get anywhere near production. IMO, it's not worth plumbing in @kvm just to be able to do KVM_BUG_ON.
Quoted 12 lines · Sean Christopherson
For all intents and purposes, this is an x86/mmu series, but it touches s390 and common KVM code because
KVM_REQ_MMU_RELOADis currently a generic request despite its use being encapsulated entirely within arch code.The meat of the series is to zap only obsolete (a.k.a. invalid) roots in response to KVM marking a root obsolete/invalid due to it being zapped. KVM currently drops/zaps all roots, which, aside from being a performance hit if the guest is using multiple roots, complicates x86 KVM paths that load a new root because it raises the question of what should be done if there's a pending
KVM_REQ_MMU_RELOAD, i.e. if the path _knows_ that any root it loads will be obliterated.
Quoted 5 lines
Paolo, I'm hoping you can squash patch 01 with your patch it "fixes".
I'm also speculating that this will be applied after my patch to remove
KVM_REQ_GPC_INVALIDATE, otherwise the changelog in patch 06 will be wrong.
Queued, thanks.
Paolo
Quoted 31 lines
v2:
- Collect reviews. [Claudio, Janosch]
- Rebase to latest kvm/queue.v1: https://lore.kernel.org/all/[email protected]
Sean Christopherson (7):
KVM: x86: Remove spurious whitespaces fromkvm_post_set_cr4()
KVM: x86: Invokekvm_mmu_unload()directly on CR4.PCIDE change
KVM: Dropkvm_reload_remote_mmus(), open code request in x86 users
KVM: x86/mmu: Zap only obsolete roots if a root shadow page is zapped
KVM: s390: ReplaceKVM_REQ_MMU_RELOADusage with arch specific request
KVM: DropKVM_REQ_MMU_RELOADand updatevcpu-requests.rst
documentation
KVM: WARN ifis_unsync_root()is called on a root without a shadow
pageDocumentation/virt/kvm/vcpu-requests.rst | 7 +- arch/s390/include/asm/kvm_host.h | 2 + arch/s390/kvm/kvm-s390.c | 8 +-- arch/s390/kvm/kvm-s390.h | 2 +- arch/x86/include/asm/kvm_host.h | 2 + arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/mmu/mmu.c | 83 ++++++++++++++++++++---- arch/x86/kvm/x86.c | 10 +-- include/linux/kvm_host.h | 4 +- virt/kvm/kvm_main.c | 5 -- 10 files changed, 90 insertions(+), 34 deletions(-)base-commit:
f4bc051fc91ab9f1d5225d94e52d369ef58bec58