KVM: x86/mmu: Skip mmu_notifier changes when possible
v4 cover KVM: x86/mmu: Skip mmu_notifier changes when possible
These patches reduce how often mmu_notifier updates block guest page faults. The primary benefit of this is the reduction in the likelihood of extreme latency when handling a page fault due to another thread having been preempted while modifying host virtual addresses.
v3 -> v4:
- Fix bug by skipping prefetch during invalidation
v2 -> v3:
- Added patch to skip check for MMIO page faults
- Style changes
David Stevens (1):
KVM: x86/mmu: Consider the hva in mmu_notifier retry
Sean Christopherson (1):
KVM: x86/mmu: Skip mmu_notifier check when handling MMIO page fault
arch/powerpc/kvm/book3s_64_mmu_hv.c | 2 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 2 +- arch/x86/kvm/mmu/mmu.c | 23 ++++++++++++++------ arch/x86/kvm/mmu/paging_tmpl.h | 7 ++++--- include/linux/kvm_host.h | 25 +++++++++++++++++++++- virt/kvm/kvm_main.c | 29 ++++++++++++++++++++++---- 6 files changed, 72 insertions(+), 16 deletions(-)
Signature · 1 line
2.30.0.617.g56c4b15f3c-goog
PATCH v4 1/2 KVM: x86/mmu: Skip mmu_notifier check when handling MMIO page fault
From: Sean Christopherson <[email protected]>
Don't retry a page fault due to an mmu_notifier invalidation when handling a page fault for a GPA that did not resolve to a memslot, i.e. an MMIO page fault. Invalidations from the mmu_notifier signal a change in a host virtual address (HVA) mapping; without a memslot, there is no HVA and thus no possibility that the invalidation is relevant to the page fault being handled.
Note, the MMIO vs. memslot generation checks handle the case where a pending memslot will create a memslot overlapping the faulting GPA. The mmu_notifier checks are orthogonal to memslot updates.
--- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -3725,7 +3725,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, r =RET_PF_RETRY ;spin_lock (&vcpu->kvm->mmu_lock); -if (mmu_notifier_retry (vcpu->kvm, mmu_seq)) +if (!is_noslot_pfn (pfn) &&mmu_notifier_retry (vcpu->kvm, mmu_seq)) goto out_unlock; r =make_mmu_pages_available (vcpu);if (r)
arch/x86/kvm/mmu/paging_tmpl.h +1 -1
@@ -869,7 +869,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gpa_t addr, u32 error_code, r =RET_PF_RETRY ;spin_lock (&vcpu->kvm->mmu_lock); -if (mmu_notifier_retry (vcpu->kvm, mmu_seq)) +if (!is_noslot_pfn (pfn) &&mmu_notifier_retry (vcpu->kvm, mmu_seq)) goto out_unlock;kvm_mmu_audit (vcpu,AUDIT_PRE_PAGE_FAULT );
Signature · 1 line
2.30.0.617.g56c4b15f3c-goog
PATCH v4 2/2 KVM: x86/mmu: Consider the hva in mmu_notifier retry
In reply to David StevensFrom: David Stevens <[email protected]>
Track the range being invalidated by mmu_notifier and skip page fault retries if the fault address is not affected by the in-progress invalidation. Handle concurrent invalidations by finding the minimal range which includes all ranges being invalidated. Although the combined range may include unrelated addresses and cannot be shrunk as individual invalidation operations complete, it is unlikely the marginal gains of proper range tracking are worth the additional complexity.
The primary benefit of this change is the reduction in the likelihood of extreme latency when handing a page fault due to another thread having been preempted while modifying host virtual addresses.
---
v3 -> v4:
- Skip prefetch while invalidations are in progress
v2 -> v3:
- Removed unnecessary ifdef
- Style changes
v1 -> v2:
- Improve handling of concurrent invalidation requests by unioning ranges, instead of just giving up and using [0, ULONG_MAX).
- Add lockdep check
- Code comments and formatting
arch/powerpc/kvm/book3s_64_mmu_hv.c | 2 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 2 +- arch/x86/kvm/mmu/mmu.c | 23 ++++++++++++++------ arch/x86/kvm/mmu/paging_tmpl.h | 7 ++++--- include/linux/kvm_host.h | 25 +++++++++++++++++++++- virt/kvm/kvm_main.c | 29 ++++++++++++++++++++++---- 6 files changed, 72 insertions(+), 16 deletions(-)
arch/powerpc/kvm/book3s_64_mmu_hv.c +1 -1
@@ -590,7 +590,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_vcpu *vcpu, }else {/* Call KVM generic code to do the slow-path check */ pfn =__gfn_to_pfn_memslot (memslot, gfn, false,NULL , - writing, &write_ok); + writing, &write_ok,NULL );if (is_error_noslot_pfn (pfn))return -EFAULT ; page =NULL ;
arch/powerpc/kvm/book3s_64_mmu_radix.c +1 -1
@@ -822,7 +822,7 @@ int kvmppc_book3s_instantiate_page(struct kvm_vcpu *vcpu,/* Call KVM generic code to do the slow-path check */ pfn =__gfn_to_pfn_memslot (memslot, gfn, false,NULL , - writing, upgrade_p); + writing, upgrade_p,NULL );if (is_error_noslot_pfn (pfn))return -EFAULT ; page =NULL ;
arch/x86/kvm/mmu/mmu.c +17 -6
@@ -2758,6 +2758,13 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)if (sp->role.level >PG_LEVEL_4K )return ; +/* +* If addresses are being invalidated, skip prefetching to avoid +* accidentally prefetching those addresses. +*/ +if (unlikely (vcpu->kvm->mmu_notifier_count)) +return ; +__direct_pte_prefetch (vcpu, sp, sptep); }
@@ -3658,8 +3665,8 @@ static bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, }static bool try_async_pf (struct kvm_vcpu *vcpu,bool prefault,gfn_t gfn, -gpa_t cr2_or_gpa,kvm_pfn_t *pfn,bool write, -bool *writable) +gpa_t cr2_or_gpa,kvm_pfn_t *pfn,hva_t *hva, +bool write,bool *writable) {struct kvm_memory_slot *slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn);bool async;
@@ -3672,7 +3679,8 @@ static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn, } async = false; - *pfn =__gfn_to_pfn_memslot (slot, gfn, false, &async, write, writable); + *pfn =__gfn_to_pfn_memslot (slot, gfn, false, &async, + write, writable, hva);if (!async)return false;/* *pfn has correct page already */
@@ -3686,7 +3694,8 @@ static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn,return true; } - *pfn =__gfn_to_pfn_memslot (slot, gfn, false,NULL , write, writable); + *pfn =__gfn_to_pfn_memslot (slot, gfn, false,NULL , + write, writable, hva);return false; }
@@ -3699,6 +3708,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,gfn_t gfn = gpa >>PAGE_SHIFT ;unsigned long mmu_seq;kvm_pfn_t pfn; +hva_t hva;int r;if (page_fault_handle_page_track (vcpu, error_code, gfn))
@@ -3717,7 +3727,8 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, mmu_seq = vcpu->kvm->mmu_notifier_seq;smp_rmb (); -if (try_async_pf (vcpu, prefault, gfn, gpa, &pfn, write, &map_writable)) +if (try_async_pf (vcpu, prefault, gfn, gpa, &pfn, &hva, + write, &map_writable))return RET_PF_RETRY ;if (handle_abnormal_pfn (vcpu, is_tdp ?0 : gpa, gfn, pfn,ACC_ALL , &r))
@@ -3725,7 +3736,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, r =RET_PF_RETRY ;spin_lock (&vcpu->kvm->mmu_lock); -if (!is_noslot_pfn (pfn) &&mmu_notifier_retry (vcpu->kvm, mmu_seq)) +if (!is_noslot_pfn (pfn) &&mmu_notifier_retry_hva (vcpu->kvm, mmu_seq, hva)) goto out_unlock; r =make_mmu_pages_available (vcpu);if (r)
arch/x86/kvm/mmu/paging_tmpl.h +4 -3
@@ -790,6 +790,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gpa_t addr, u32 error_code,struct guest_walker walker;int r;kvm_pfn_t pfn; +hva_t hva;unsigned long mmu_seq;bool map_writable, is_self_change_mapping;int max_level;
@@ -840,8 +841,8 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gpa_t addr, u32 error_code, mmu_seq = vcpu->kvm->mmu_notifier_seq;smp_rmb (); -if (try_async_pf (vcpu, prefault, walker.gfn, addr, &pfn, write_fault, - &map_writable)) +if (try_async_pf (vcpu, prefault, walker.gfn, addr, &pfn, &hva, + write_fault, &map_writable))return RET_PF_RETRY ;if (handle_abnormal_pfn (vcpu, addr, walker.gfn, pfn, walker.pte_access, &r))
@@ -869,7 +870,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gpa_t addr, u32 error_code, r =RET_PF_RETRY ;spin_lock (&vcpu->kvm->mmu_lock); -if (!is_noslot_pfn (pfn) &&mmu_notifier_retry (vcpu->kvm, mmu_seq)) +if (!is_noslot_pfn (pfn) &&mmu_notifier_retry_hva (vcpu->kvm, mmu_seq, hva)) goto out_unlock;kvm_mmu_audit (vcpu,AUDIT_PRE_PAGE_FAULT );
include/linux/kvm_host.h +24 -1
@@ -11,6 +11,7 @@#include <linux/signal.h> #include <linux/sched.h> #include <linux/bug.h> +#include <linux/minmax.h> #include <linux/mm.h> #include <linux/mmu_notifier.h> #include <linux/preempt.h>
@@ -502,6 +503,8 @@ struct kvm {struct mmu_notifier mmu_notifier;unsigned long mmu_notifier_seq;long mmu_notifier_count; +unsigned long mmu_notifier_range_start; +unsigned long mmu_notifier_range_end;#endif long tlbs_dirty;struct list_head devices;
@@ -729,7 +732,7 @@ kvm_pfn_t gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t gfn);kvm_pfn_t gfn_to_pfn_memslot_atomic (struct kvm_memory_slot *slot,gfn_t gfn);kvm_pfn_t __gfn_to_pfn_memslot (struct kvm_memory_slot *slot,gfn_t gfn,bool atomic,bool *async,bool write_fault, -bool *writable); +bool *writable,hva_t *hva);void kvm_release_pfn_clean (kvm_pfn_t pfn);void kvm_release_pfn_dirty (kvm_pfn_t pfn);
@@ -1203,6 +1206,26 @@ static inline int mmu_notifier_retry(struct kvm *kvm, unsigned long mmu_seq)return 1 ; return 0; } + +static inline int mmu_notifier_retry_hva (struct kvm *kvm, +unsigned long mmu_seq, +unsigned long hva) +{ +lockdep_assert_held (&kvm->mmu_lock); +/* +* If mmu_notifier_count is non-zero, then the range maintained by +* kvm_mmu_notifier_invalidate_range_start contains all addresses that +* might be being invalidated. Note that it may include some false +* positives, due to shortcuts when handing concurrent invalidations. +*/ +if (unlikely (kvm->mmu_notifier_count) && + hva >= kvm->mmu_notifier_range_start && + hva < kvm->mmu_notifier_range_end) +return 1 ; +if (kvm->mmu_notifier_seq != mmu_seq) +return 1 ; +return 0 ; +}#endif #ifdef CONFIG_HAVE_KVM_IRQ_ROUTING
virt/kvm/kvm_main.c +23 -4
@@ -483,6 +483,24 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,* count is also read inside the mmu_lock critical section. */ kvm->mmu_notifier_count++; +if (likely (kvm->mmu_notifier_count =1 )) { + kvm->mmu_notifier_range_start = range->start; + kvm->mmu_notifier_range_end = range->end; + }else { +/* +* Fully tracking multiple concurrent ranges has dimishing +* returns. Keep things simple and just find the minimal range +* which includes the current and new ranges. As there won't be +* enough information to subtract a range after its invalidate +* completes, any ranges invalidated concurrently will +* accumulate and persist until all outstanding invalidates +* complete. +*/ + kvm->mmu_notifier_range_start +min (kvm->mmu_notifier_range_start, range->start); + kvm->mmu_notifier_range_end +max (kvm->mmu_notifier_range_end, range->end); + } need_tlb_flush =kvm_unmap_hva_range (kvm, range->start, range->end, range->flags);/* we've to flush the tlb before the pages can be freed */
@@ -2011,10 +2029,13 @@ static kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool *async,kvm_pfn_t __gfn_to_pfn_memslot (struct kvm_memory_slot *slot,gfn_t gfn,bool atomic,bool *async,bool write_fault, -bool *writable) +bool *writable,hva_t *hva) {unsigned long addr =__gfn_to_hva_many (slot, gfn,NULL , write_fault); +if (hva) + *hva = addr; +if (addr =KVM_HVA_ERR_RO_BAD ) {if (writable) *writable = false;
@@ -2042,19 +2063,19 @@ kvm_pfn_t gfn_to_pfn_prot(struct kvm *kvm, gfn_t gfn, bool write_fault, bool *writable) {return __gfn_to_pfn_memslot (gfn_to_memslot (kvm, gfn), gfn, false,NULL , - write_fault, writable); + write_fault, writable,NULL ); }EXPORT_SYMBOL_GPL (gfn_to_pfn_prot);kvm_pfn_t gfn_to_pfn_memslot (struct kvm_memory_slot *slot,gfn_t gfn) { -return __gfn_to_pfn_memslot (slot, gfn, false,NULL , true,NULL ); +return __gfn_to_pfn_memslot (slot, gfn, false,NULL , true,NULL ,NULL ); }EXPORT_SYMBOL_GPL (gfn_to_pfn_memslot);kvm_pfn_t gfn_to_pfn_memslot_atomic (struct kvm_memory_slot *slot,gfn_t gfn) { -return __gfn_to_pfn_memslot (slot, gfn, true,NULL , true,NULL ); +return __gfn_to_pfn_memslot (slot, gfn, true,NULL , true,NULL ,NULL ); }EXPORT_SYMBOL_GPL (gfn_to_pfn_memslot_atomic);
Signature · 1 line
2.30.0.617.g56c4b15f3c-goog
Sean Christopherson
maintainerRe: [PATCH v4 2/2] KVM: x86/mmu: Consider the hva in mmu_notifier retry
Quoted 3 lines · David Stevens
---
v3 -> v4:
- Skip prefetch while invalidations are in progress
Oof, nice catch.
...
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +0
-0
arch/x86/kvm/mmu/mmu.c static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep) · line 2766
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +6
-0@@ -2758,6 +2758,13 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)if (sp->role.level >PG_LEVEL_4K )return ; +/* +* If addresses are being invalidated, skip prefetching to avoid +* accidentally prefetching those addresses. +*/ +if (unlikely (vcpu->kvm->mmu_notifier_count)) +return ;
FNAME(pte_prefetch) needs the same check.
Paolo, this brings up a good addition for the work to integrate the mmu notifier into the rest of KVM, e.g. for vmcs12 pages. Ideally, gfn_to_page_many_atomic() and __gfn_to_pfn_memslot() would WARN if mmu_notifier_count is non-zero, but that will fire all over the place until the nested code properly integrates the notifier. There are a few use cases where racing with the notifier is acceptable, e.g. reexecute_instruction(), but hopefully we can address those flows without things getting too ugly.
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +1
-0+__direct_pte_prefetch (vcpu, sp, sptep); }
Paolo Bonzini
maintainerQuoted 18 lines · David Stevens
These patches reduce how often
mmu_notifierupdates block guest page faults. The primary benefit of this is the reduction in the likelihood of extreme latency when handling a page fault due to another thread having been preempted while modifying host virtual addresses.v3 -> v4:
- Fix bug by skipping prefetch during invalidationv2 -> v3:
- Added patch to skip check for MMIO page faults
- Style changesDavid Stevens (1):
KVM: x86/mmu: Consider the hva inmmu_notifierretrySean Christopherson (1):
KVM: x86/mmu: Skipmmu_notifiercheck when handling MMIO page fault
Quoted 8 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c | 2 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 2 +- arch/x86/kvm/mmu/mmu.c | 23 ++++++++++++++------ arch/x86/kvm/mmu/paging_tmpl.h | 7 ++++--- include/linux/kvm_host.h | 25 +++++++++++++++++++++- virt/kvm/kvm_main.c | 29 ++++++++++++++++++++++---- 6 files changed, 72 insertions(+), 16 deletions(-)
Rebased, and queued with the fix that Sean suggested.
Paolo