KVM: x86/mmu: Make tdp_mmu a read-only parameter
Sean Christopherson
maintainerv4 cover KVM: x86/mmu: Make tdp_mmu a read-only parameter
This is a variation of David's series to change tdp_mmu to a RO param[*]. The key difference is that instead of moving the TDP MMU page fault handler to its own function, use static branches to make TDP MMU page faults (and a few other paths) effectively branch free.
I'm not dead set against having a dedicated TDP MMU page fault handler, but IMO it's not really better once the TDP MMU vs. shadow MMU is reduced to a static branch, just different. The read vs. write mmu_lock is the most visible ugliness, and that can be buried in helpers if we really want to make the page fault handler easier on the eyes, e.g.
direct_page_fault_mmu_lock(vcpu);
if (is_page_fault_stale(vcpu, fault))
goto out_unlock;
if (is_tdp_mmu_enabled()) {
r = kvm_tdp_mmu_map(vcpu, fault);
} else {
r = make_mmu_pages_available(vcpu);
if (r)
goto out_unlock;
r = __direct_map(vcpu, fault);
}out_unlock:
direct_page_fault_mmu_unlock(vcpu);
v4:
- Keep is_tdp_mmu_page() in patch 1.
- Collect reviews. [Isaku]
- Skip "make MMU pages available" for root allocations.
- Rework "is TDP MMU" checks to take advantage of read-only param.
- Use a static key to track TDP MMU enabling.
[*] https://lkml.kernel.org/r/[email protected]
David Matlack (7):
KVM: x86/mmu: Change tdp_mmu to a read-only parameter
KVM: x86/mmu: Move TDP MMU VM init/uninit behind tdp_mmu_enabled
KVM: x86/mmu: Grab mmu_invalidate_seq in kvm_faultin_pfn()
KVM: x86/mmu: Handle error PFNs in kvm_faultin_pfn()
KVM: x86/mmu: Avoid memslot lookup during KVM_PFN_ERR_HWPOISON
handling
KVM: x86/mmu: Handle no-slot faults in kvm_faultin_pfn()
KVM: x86/mmu: Stop needlessly making MMU pages available for TDP MMU
Sean Christopherson (4):
KVM: x86/mmu: Pivot on "TDP MMU enabled" when handling direct page
faults
KVM: x86/mmu: Pivot on "TDP MMU enabled" to check if active MMU is TDP
MMU
KVM: x86/mmu: Replace open coded usage of tdp_mmu_page with
is_tdp_mmu_page()
KVM: x86/mmu: Use static key/branches for checking if TDP MMU is
enabled
arch/x86/include/asm/kvm_host.h | 9 -- arch/x86/kvm/mmu.h | 14 ++- arch/x86/kvm/mmu/mmu.c | 212 ++++++++++++++++++++------------ arch/x86/kvm/mmu/mmu_internal.h | 1 + arch/x86/kvm/mmu/paging_tmpl.h | 12 +- arch/x86/kvm/mmu/tdp_mmu.c | 13 +- arch/x86/kvm/mmu/tdp_mmu.h | 25 +--- 7 files changed, 149 insertions(+), 137 deletions(-)
base-commit: e18d6152ff0f41b7f01f9817372022df04e0d354
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Sean Christopherson
maintainerPATCH v4 1/11 KVM: x86/mmu: Change tdp_mmu to a read-only parameter
From: David Matlack <[email protected]>
Change tdp_mmu to a read-only parameter and drop the per-vm tdp_mmu_enabled. Keep the is_tdp_mmu_enabled() helper instead of referencing tdp_mmu_enabled directly to allow for future optimizations without needing to churn a lot of code, e.g. KVM can use a static key for now that the knob is read-only after the vendor module is loaded.
The TDP MMU was introduced in 5.10 and has been enabled by default since 5.15. At this point there are no known functionality gaps between the TDP MMU and the shadow MMU, and the TDP MMU uses less memory and scales better with the number of vCPUs. In other words, there is no good reason to disable the TDP MMU on a live system.
Purposely do not drop tdp_mmu=N support (i.e. do not force 64-bit KVM to always use the TDP MMU) since tdp_mmu=N is still used to get test coverage of KVM's shadow MMU TDP support, which is used in 32-bit KVM.
--- arch/x86/include/asm/kvm_host.h | 9 ------ arch/x86/kvm/mmu.h | 13 ++++++-- arch/x86/kvm/mmu/mmu.c | 53 ++++++++++++++++++++++----------- arch/x86/kvm/mmu/tdp_mmu.c | 9 ++---- 4 files changed, 48 insertions(+), 36 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -9
@@ -1270,15 +1270,6 @@ struct kvm_arch {struct task_struct *nx_lpage_recovery_thread;#ifdef CONFIG_X86_64 -/* -* Whether the TDP MMU is enabled for this VM. This contains a -* snapshot of the TDP MMU module parameter from when the VM was -* created and remains unchanged for the life of the VM. If this is -* true, TDP MMU handler functions will run for various MMU -* operations. -*/ -bool tdp_mmu_enabled; -/* * List of kvm_mmu_page structs being used as roots. * All kvm_mmu_page structs in the list should have
arch/x86/kvm/mmu.h +10 -3
@@ -230,14 +230,21 @@ static inline bool kvm_shadow_root_allocated(struct kvm *kvm) }#ifdef CONFIG_X86_64 -static inline bool is_tdp_mmu_enabled (struct kvm *kvm) {return kvm->arch.tdp_mmu_enabled; } +extern bool tdp_mmu_enabled; +#endif + +static inline bool is_tdp_mmu_enabled (void ) +{ +#ifdef CONFIG_X86_64 +return tdp_mmu_enabled;#else -static inline bool is_tdp_mmu_enabled (struct kvm *kvm) {return false; } +return false;#endif +}static inline bool kvm_memslots_have_rmaps (struct kvm *kvm) { -return !is_tdp_mmu_enabled (kvm) ||kvm_shadow_root_allocated (kvm); +return !is_tdp_mmu_enabled () ||kvm_shadow_root_allocated (kvm); }static inline gfn_t gfn_to_index (gfn_t gfn,gfn_t base_gfn,int level)
arch/x86/kvm/mmu/mmu.c +36 -17
173 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.c +2 -7
@@ -10,23 +10,18 @@#include <asm/cmpxchg.h> #include <trace/events/kvm.h> -static bool __read_mostly tdp_mmu_enabled = true; -module_param_named (tdp_mmu, tdp_mmu_enabled, bool,0644 ); -/* Initializes the TDP MMU for the VM, if enabled. */ int kvm_mmu_init_tdp_mmu (struct kvm *kvm) {struct workqueue_struct *wq; -if (!tdp_enabled || !READ_ONCE (tdp_mmu_enabled)) +if (!is_tdp_mmu_enabled ())return 0 ; wq =alloc_workqueue ("kvm" ,WQ_UNBOUND |WQ_MEM_RECLAIM |WQ_CPU_INTENSIVE ,0 );if (!wq)return -ENOMEM ; -/* This should not be changed for the lifetime of the VM. */ - kvm->arch.tdp_mmu_enabled = true;INIT_LIST_HEAD (&kvm->arch.tdp_mmu_roots);spin_lock_init (&kvm->arch.tdp_mmu_pages_lock);INIT_LIST_HEAD (&kvm->arch.tdp_mmu_pages);
@@ -48,7 +43,7 @@ static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm,void kvm_mmu_uninit_tdp_mmu (struct kvm *kvm) { -if (!kvm->arch.tdp_mmu_enabled) +if (!is_tdp_mmu_enabled ())return ;/* Also waits for any queued work items. */
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Re: [PATCH v4 01/11] KVM: x86/mmu: Change tdp_mmu to a read-only parameter
Quoted 14 lines · Sean Christopherson
From: David Matlack <[email protected]>
Change
tdp_mmuto a read-only parameter and drop the per-vm tdp_mmu_enabled. Keep theis_tdp_mmu_enabled()helper instead of referencingtdp_mmu_enableddirectly to allow for future optimizations without needing to churn a lot of code, e.g. KVM can use a static key for now that the knob is read-only after the vendor module is loaded.The TDP MMU was introduced in 5.10 and has been enabled by default since 5.15. At this point there are no known functionality gaps between the TDP MMU and the shadow MMU, and the TDP MMU uses less memory and scales better with the number of vCPUs. In other words, there is no good reason to disable the TDP MMU on a live system.
Quoted 7 lines
Purposely do not drop tdp_mmu=N support (i.e. do not force 64-bit KVM to always use the TDP MMU) since tdp_mmu=N is still used to get test coverage of KVM's shadow MMU TDP support, which is used in 32-bit KVM.
Signed-off-by David Matlack <[email protected]>[sean: keep is_tdp_mmu_enabled()]Signed-off-by Sean Christopherson <[email protected]>
Sean Christopherson
maintainerPATCH v4 2/11 KVM: x86/mmu: Move TDP MMU VM init/uninit behind tdp_mmu_enabled
In reply to Sean ChristophersonFrom: David Matlack <[email protected]>
Move kvm_mmu_{init,uninit}_tdp_mmu() behind tdp_mmu_enabled. This makes these functions consistent with the rest of the calls into the TDP MMU from mmu.c, and which is now possible since tdp_mmu_enabled is only modified when the x86 vendor module is loaded. i.e. It will never change during the lifetime of a VM.
This change also enabled removing the stub definitions for 32-bit KVM, as the compiler will just optimize the calls out like it does for all the other TDP MMU functions.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 11 +++++++---- arch/x86/kvm/mmu/tdp_mmu.c | 6 ------ arch/x86/kvm/mmu/tdp_mmu.h | 7 +++---- 3 files changed, 10 insertions(+), 14 deletions(-)
arch/x86/kvm/mmu/mmu.c +7 -4
@@ -5984,9 +5984,11 @@ int kvm_mmu_init_vm(struct kvm *kvm)INIT_LIST_HEAD (&kvm->arch.lpage_disallowed_mmu_pages);spin_lock_init (&kvm->arch.mmu_unsync_pages_lock); - r =kvm_mmu_init_tdp_mmu (kvm); -if (r <0 ) -return r; +if (is_tdp_mmu_enabled ()) { + r =kvm_mmu_init_tdp_mmu (kvm); +if (r <0 ) +return r; + } node->track_write = kvm_mmu_pte_write; node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
@@ -6016,7 +6018,8 @@ void kvm_mmu_uninit_vm(struct kvm *kvm)kvm_page_track_unregister_notifier (kvm, node); -kvm_mmu_uninit_tdp_mmu (kvm); +if (is_tdp_mmu_enabled ()) +kvm_mmu_uninit_tdp_mmu (kvm);mmu_free_vm_memory_caches (kvm); }
arch/x86/kvm/mmu/tdp_mmu.c +0 -6
@@ -15,9 +15,6 @@ int kvm_mmu_init_tdp_mmu(struct kvm *kvm) {struct workqueue_struct *wq; -if (!is_tdp_mmu_enabled ()) -return 0 ; - wq =alloc_workqueue ("kvm" ,WQ_UNBOUND |WQ_MEM_RECLAIM |WQ_CPU_INTENSIVE ,0 );if (!wq)return -ENOMEM ;
@@ -43,9 +40,6 @@ static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm,void kvm_mmu_uninit_tdp_mmu (struct kvm *kvm) { -if (!is_tdp_mmu_enabled ()) -return ; -/* Also waits for any queued work items. */ destroy_workqueue (kvm->arch.tdp_mmu_zap_wq);
arch/x86/kvm/mmu/tdp_mmu.h +3 -4
@@ -5,6 +5,9 @@#include <linux/kvm_host.h> +int kvm_mmu_init_tdp_mmu (struct kvm *kvm); +void kvm_mmu_uninit_tdp_mmu (struct kvm *kvm); +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu);__must_check static inline bool kvm_tdp_mmu_get_root (struct kvm_mmu_page *root)
@@ -66,8 +69,6 @@ u64 *kvm_tdp_mmu_fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, u64 addr, u64 *spte);#ifdef CONFIG_X86_64 -int kvm_mmu_init_tdp_mmu (struct kvm *kvm); -void kvm_mmu_uninit_tdp_mmu (struct kvm *kvm);static inline bool is_tdp_mmu_page (struct kvm_mmu_page *sp) {return sp->tdp_mmu_page; }static inline bool is_tdp_mmu (struct kvm_mmu *mmu)
@@ -87,8 +88,6 @@ static inline bool is_tdp_mmu(struct kvm_mmu *mmu) return sp && is_tdp_mmu_page(sp) && sp->root_count; }#else -static inline int kvm_mmu_init_tdp_mmu (struct kvm *kvm) {return 0 ; } -static inline void kvm_mmu_uninit_tdp_mmu (struct kvm *kvm) {}static inline bool is_tdp_mmu_page (struct kvm_mmu_page *sp) {return false; }static inline bool is_tdp_mmu (struct kvm_mmu *mmu) {return false; }#endif
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Sean Christopherson
maintainerPATCH v4 3/11 KVM: x86/mmu: Grab mmu_invalidate_seq in kvm_faultin_pfn()
In reply to Sean ChristophersonFrom: David Matlack <[email protected]>
Grab mmu_invalidate_seq in kvm_faultin_pfn() and stash it in struct kvm_page_fault. The eliminates duplicate code and reduces the amount of parameters needed for is_page_fault_stale().
Preemptively split out __kvm_faultin_pfn() to a separate function for use in subsequent commits.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 21 ++++++++++++--------- arch/x86/kvm/mmu/mmu_internal.h | 1 + arch/x86/kvm/mmu/paging_tmpl.h | 6 +----- 3 files changed, 14 insertions(+), 14 deletions(-)
arch/x86/kvm/mmu/mmu.c +12 -9
@@ -4143,7 +4143,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)kvm_mmu_do_page_fault (vcpu, work->cr2_or_gpa,0 , true); } -static int kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) +static int __kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) {struct kvm_memory_slot *slot = fault->slot;bool async;
@@ -4199,12 +4199,20 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return RET_PF_CONTINUE ; } +static int kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) +{ + fault->mmu_seq = vcpu->kvm->mmu_invalidate_seq; +smp_rmb (); + +return __kvm_faultin_pfn (vcpu, fault); +} +/* * Returns true if the page fault is stale and needs to be retried, i.e. if the * root was invalidated by a memslot update or a relevant mmu_notifier fired. */ static bool is_page_fault_stale (struct kvm_vcpu *vcpu, -struct kvm_page_fault *fault,int mmu_seq) +struct kvm_page_fault *fault) {struct kvm_mmu_page *sp =to_shadow_page (vcpu->arch.mmu->root.hpa);
@@ -4224,14 +4232,12 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,return true;return fault->slot && -mmu_invalidate_retry_hva (vcpu->kvm, mmu_seq, fault->hva); +mmu_invalidate_retry_hva (vcpu->kvm, fault->mmu_seq, fault->hva); }static int direct_page_fault (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) {bool is_tdp_mmu_fault =is_tdp_mmu (vcpu->arch.mmu); - -unsigned long mmu_seq;int r; fault->gfn = fault->addr >>PAGE_SHIFT ;
@@ -4248,9 +4254,6 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultif (r)return r; - mmu_seq = vcpu->kvm->mmu_invalidate_seq; -smp_rmb (); - r =kvm_faultin_pfn (vcpu, fault);if (r !=RET_PF_CONTINUE )return r;
@@ -4266,7 +4269,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultelse write_lock (&vcpu->kvm->mmu_lock); -if (is_page_fault_stale (vcpu, fault, mmu_seq)) +if (is_page_fault_stale (vcpu, fault)) goto out_unlock; r =make_mmu_pages_available (vcpu);
arch/x86/kvm/mmu/mmu_internal.h +1 -0
@@ -221,6 +221,7 @@ struct kvm_page_fault {struct kvm_memory_slot *slot;/* Outputs of kvm_faultin_pfn. */ +unsigned long mmu_seq;kvm_pfn_t pfn;hva_t hva;bool map_writable;
arch/x86/kvm/mmu/paging_tmpl.h +1 -5
@@ -791,7 +791,6 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault {struct guest_walker walker;int r; -unsigned long mmu_seq;bool is_self_change_mapping;pgprintk ("%s: addr %lx err %x\n" , __func__, fault->addr, fault->error_code);
@@ -838,9 +837,6 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultelse fault->max_level = walker.level; - mmu_seq = vcpu->kvm->mmu_invalidate_seq; -smp_rmb (); - r =kvm_faultin_pfn (vcpu, fault);if (r !=RET_PF_CONTINUE )return r;
@@ -871,7 +867,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault r =RET_PF_RETRY ;write_lock (&vcpu->kvm->mmu_lock); -if (is_page_fault_stale (vcpu, fault, mmu_seq)) +if (is_page_fault_stale (vcpu, fault)) goto out_unlock; r =make_mmu_pages_available (vcpu);
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Sean Christopherson
maintainerPATCH v4 4/11 KVM: x86/mmu: Handle error PFNs in kvm_faultin_pfn()
In reply to Sean ChristophersonFrom: David Matlack <[email protected]>
Handle error PFNs in kvm_faultin_pfn() rather than relying on the caller to invoke handle_abnormal_pfn() after kvm_faultin_pfn(). Opportunistically rename kvm_handle_bad_page() to kvm_handle_error_pfn() to make it more consistent with is_error_pfn().
This commit moves KVM closer to being able to drop handle_abnormal_pfn(), which will reduce the amount of duplicate code in the various page fault handlers.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
arch/x86/kvm/mmu/mmu.c +11 -6
@@ -3155,7 +3155,7 @@ static void kvm_send_hwpoison_signal(unsigned long address, struct task_struct *send_sig_mceerr (BUS_MCEERR_AR , (void __user *)address,PAGE_SHIFT , tsk); } -static int kvm_handle_bad_page (struct kvm_vcpu *vcpu,gfn_t gfn,kvm_pfn_t pfn) +static int kvm_handle_error_pfn (struct kvm_vcpu *vcpu,gfn_t gfn,kvm_pfn_t pfn) {/* * Do not cache the mmio info caused by writing the readonly gfn
@@ -3176,10 +3176,6 @@ static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)static int handle_abnormal_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault,unsigned int access) { -/* The pfn is invalid, report the error! */ -if (unlikely (is_error_pfn (fault->pfn))) -return kvm_handle_bad_page (vcpu, fault->gfn, fault->pfn); -if (unlikely (!fault->slot)) {gva_t gva = fault->is_tdp ?0 : fault->addr;
@@ -4201,10 +4197,19 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultstatic int kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) { +int ret; + fault->mmu_seq = vcpu->kvm->mmu_invalidate_seq;smp_rmb (); -return __kvm_faultin_pfn (vcpu, fault); + ret =__kvm_faultin_pfn (vcpu, fault); +if (ret !=RET_PF_CONTINUE ) +return ret; + +if (unlikely (is_error_pfn (fault->pfn))) +return kvm_handle_error_pfn (vcpu, fault->gfn, fault->pfn); + +return RET_PF_CONTINUE ; }/*
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Sean Christopherson
maintainerPATCH v4 5/11 KVM: x86/mmu: Avoid memslot lookup during KVM_PFN_ERR_HWPOISON handling
In reply to Sean ChristophersonFrom: David Matlack <[email protected]>
Pass the kvm_page_fault struct down to kvm_handle_error_pfn() to avoid a memslot lookup when handling KVM_PFN_ERR_HWPOISON. Opportunistically move the gfn_to_hva_memslot() call and @current down into kvm_send_hwpoison_signal() to cut down on line lengths.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/mmu.c +9 -7
@@ -3150,23 +3150,25 @@ static int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return ret; } -static void kvm_send_hwpoison_signal (unsigned long address,struct task_struct *tsk) +static void kvm_send_hwpoison_signal (struct kvm_memory_slot *slot,gfn_t gfn) { -send_sig_mceerr (BUS_MCEERR_AR , (void __user *)address,PAGE_SHIFT , tsk); +unsigned long hva =gfn_to_hva_memslot (slot, gfn); + +send_sig_mceerr (BUS_MCEERR_AR , (void __user *)hva,PAGE_SHIFT , current); } -static int kvm_handle_error_pfn (struct kvm_vcpu *vcpu,gfn_t gfn,kvm_pfn_t pfn) +static int kvm_handle_error_pfn (struct kvm_page_fault *fault) {/* * Do not cache the mmio info caused by writing the readonly gfn * into the spte otherwise read access on readonly gfn also can * caused mmio page fault and treat it as mmio access. */ -if (pfn ==KVM_PFN_ERR_RO_FAULT ) +if (fault->pfn ==KVM_PFN_ERR_RO_FAULT )return RET_PF_EMULATE ; -if (pfn ==KVM_PFN_ERR_HWPOISON ) { -kvm_send_hwpoison_signal (kvm_vcpu_gfn_to_hva (vcpu, gfn), current); +if (fault->pfn ==KVM_PFN_ERR_HWPOISON ) { +kvm_send_hwpoison_signal (fault->slot, fault->gfn);return RET_PF_RETRY ; }
@@ -4207,7 +4209,7 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return ret;if (unlikely (is_error_pfn (fault->pfn))) -return kvm_handle_error_pfn (vcpu, fault->gfn, fault->pfn); +return kvm_handle_error_pfn (fault);return RET_PF_CONTINUE ; }
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Sean Christopherson
maintainerPATCH v4 6/11 KVM: x86/mmu: Handle no-slot faults in kvm_faultin_pfn()
In reply to Sean ChristophersonFrom: David Matlack <[email protected]>
Handle faults on GFNs that do not have a backing memslot in kvm_faultin_pfn() and drop handle_abnormal_pfn(). This eliminates duplicate code in the various page fault handlers.
Opportunistically tweak the comment about handling gfn > host.MAXPHYADDR to reflect that the effect of returning RET_PF_EMULATE at that point is to avoid creating an MMIO SPTE for such GFNs.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 56 ++++++++++++++++++---------------- arch/x86/kvm/mmu/paging_tmpl.h | 6 +--- 2 files changed, 31 insertions(+), 31 deletions(-)
arch/x86/kvm/mmu/mmu.c +30 -26
@@ -3175,28 +3175,32 @@ static int kvm_handle_error_pfn(struct kvm_page_fault *fault)return -EFAULT ; } -static int handle_abnormal_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault, -unsigned int access) +static int kvm_handle_noslot_fault (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault, +unsigned int access) { -if (unlikely (!fault->slot)) { -gva_t gva = fault->is_tdp ?0 : fault->addr; +gva_t gva = fault->is_tdp ?0 : fault->addr; -vcpu_cache_mmio_info (vcpu, gva, fault->gfn, - access & shadow_mmio_access_mask); -/* -* If MMIO caching is disabled, emulate immediately without -* touching the shadow page tables as attempting to install an -* MMIO SPTE will just be an expensive nop. Do not cache MMIO -* whose gfn is greater than host.MAXPHYADDR, any guest that -* generates such gfns is running nested and is being tricked -* by L0 userspace (you can observe gfn > L1.MAXPHYADDR if -* and only if L1's MAXPHYADDR is inaccurate with respect to -* the hardware's). -*/ -if (unlikely (!enable_mmio_caching) || -unlikely (fault->gfn >kvm_mmu_max_gfn ())) -return RET_PF_EMULATE ; - } +vcpu_cache_mmio_info (vcpu, gva, fault->gfn, + access & shadow_mmio_access_mask); + +/* +* If MMIO caching is disabled, emulate immediately without +* touching the shadow page tables as attempting to install an +* MMIO SPTE will just be an expensive nop. +*/ +if (unlikely (!enable_mmio_caching)) +return RET_PF_EMULATE ; + +/* +* Do not create an MMIO SPTE for a gfn greater than host.MAXPHYADDR, +* any guest that generates such gfns is running nested and is being +* tricked by L0 userspace (you can observe gfn > L1.MAXPHYADDR if and +* only if L1's MAXPHYADDR is inaccurate with respect to the +* hardware's). +*/ +if (unlikely (fault->gfn >kvm_mmu_max_gfn ())) +return RET_PF_EMULATE ;return RET_PF_CONTINUE ; }
@@ -4197,7 +4201,8 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultreturn RET_PF_CONTINUE ; } -static int kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) +static int kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault, +unsigned int access) {int ret;
@@ -4211,6 +4216,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)if (unlikely (is_error_pfn (fault->pfn)))return kvm_handle_error_pfn (fault); +if (unlikely (!fault->slot)) +return kvm_handle_noslot_fault (vcpu, fault, access); +return RET_PF_CONTINUE ; }
@@ -4261,11 +4269,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultif (r)return r; - r =kvm_faultin_pfn (vcpu, fault); -if (r !=RET_PF_CONTINUE ) -return r; - - r =handle_abnormal_pfn (vcpu, fault,ACC_ALL ); + r =kvm_faultin_pfn (vcpu, fault,ACC_ALL );if (r !=RET_PF_CONTINUE )return r;
arch/x86/kvm/mmu/paging_tmpl.h +1 -5
@@ -837,11 +837,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultelse fault->max_level = walker.level; - r =kvm_faultin_pfn (vcpu, fault); -if (r !=RET_PF_CONTINUE ) -return r; - - r =handle_abnormal_pfn (vcpu, fault, walker.pte_access); + r =kvm_faultin_pfn (vcpu, fault, walker.pte_access);if (r !=RET_PF_CONTINUE )return r;
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Sean Christopherson
maintainerPATCH v4 7/11 KVM: x86/mmu: Pivot on "TDP MMU enabled" when handling direct page faults
In reply to Sean ChristophersonWhen handling direct page faults, pivot on the TDP MMU being globally enabled instead of checking if the target MMU is a TDP MMU. Now that the TDP MMU is all-or-nothing, if the TDP MMU is enabled, KVM will reach direct_page_fault() if and only if the MMU is a TDP MMU. When TDP is enabled (obviously required for the TDP MMU), only non-nested TDP page faults reach direct_page_fault(), i.e. nonpaging MMUs are impossible, as NPT requires paging to be enabled and EPT faults use ept_page_fault().
--- arch/x86/kvm/mmu/mmu.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +4 -5
@@ -3324,7 +3324,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) do { u64 new_spte; - if (is_tdp_mmu(vcpu->arch.mmu)) + if (is_tdp_mmu_enabled()) sptep = kvm_tdp_mmu_fast_pf_get_last_sptep(vcpu, fault->addr, &spte); else sptep = fast_pf_get_last_sptep(vcpu, fault->addr, &spte);
@@ -4252,7 +4252,6 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,static int direct_page_fault (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) { -bool is_tdp_mmu_fault =is_tdp_mmu (vcpu->arch.mmu);int r; fault->gfn = fault->addr >>PAGE_SHIFT ;
@@ -4275,7 +4274,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault r =RET_PF_RETRY ; -if (is_tdp_mmu_fault) +if (is_tdp_mmu_enabled ())read_lock (&vcpu->kvm->mmu_lock);else write_lock (&vcpu->kvm->mmu_lock);
@@ -4287,13 +4286,13 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultif (r) goto out_unlock; -if (is_tdp_mmu_fault) +if (is_tdp_mmu_enabled ()) r =kvm_tdp_mmu_map (vcpu, fault);else r =__direct_map (vcpu, fault); out_unlock: -if (is_tdp_mmu_fault) +if (is_tdp_mmu_enabled ())read_unlock (&vcpu->kvm->mmu_lock);else write_unlock (&vcpu->kvm->mmu_lock);
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Sean Christopherson
maintainerPATCH v4 8/11 KVM: x86/mmu: Pivot on "TDP MMU enabled" to check if active MMU is TDP MMU
In reply to Sean ChristophersonSimplify and optimize the logic for detecting if the current/active MMU is a TDP MMU. If the TDP MMU is globally enabled, then the active MMU is a TDP MMU if it is direct. When TDP is enabled, so called nonpaging MMUs are never used as the only form of shadow paging KVM uses is for nested TDP, and the active MMU can't be direct in that case.
Rename the helper and take the vCPU instead of an arbitrary MMU, as nonpaging MMUs can show up in the walk_mmu if L1 is using nested TDP and L2 has paging disabled. Taking the vCPU has the added bonus of cleaning up the callers, all of which check the current MMU but wrap code that consumes the vCPU.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 11 ++++++++--- arch/x86/kvm/mmu/tdp_mmu.h | 18 ------------------ 2 files changed, 8 insertions(+), 21 deletions(-)
arch/x86/kvm/mmu/mmu.c +8 -3
@@ -615,9 +615,14 @@ static bool mmu_spte_age(u64 *sptep)return true; } +static inline bool is_tdp_mmu_active (struct kvm_vcpu *vcpu) +{ +return is_tdp_mmu_enabled () && vcpu->arch.mmu->root_role.direct; +} +static void walk_shadow_page_lockless_begin (struct kvm_vcpu *vcpu) { -if (is_tdp_mmu (vcpu->arch.mmu)) { +if (is_tdp_mmu_active (vcpu)) {kvm_tdp_mmu_walk_lockless_begin (); }else {/*
@@ -636,7 +641,7 @@ static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu)static void walk_shadow_page_lockless_end (struct kvm_vcpu *vcpu) { -if (is_tdp_mmu (vcpu->arch.mmu)) { +if (is_tdp_mmu_active (vcpu)) {kvm_tdp_mmu_walk_lockless_end (); }else {/*
@@ -3997,7 +4002,7 @@ static bool get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr, u64 *sptep)walk_shadow_page_lockless_begin (vcpu); -if (is_tdp_mmu (vcpu->arch.mmu)) +if (is_tdp_mmu_active (vcpu)) leaf =kvm_tdp_mmu_get_walk (vcpu, addr, sptes, &root);else leaf =get_walk (vcpu, addr, sptes, &root);
arch/x86/kvm/mmu/tdp_mmu.h +0 -18
@@ -70,26 +70,8 @@ u64 *kvm_tdp_mmu_fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, u64 addr,#ifdef CONFIG_X86_64 static inline bool is_tdp_mmu_page (struct kvm_mmu_page *sp) {return sp->tdp_mmu_page; } - -static inline bool is_tdp_mmu (struct kvm_mmu *mmu) -{ -struct kvm_mmu_page *sp; -hpa_t hpa = mmu->root.hpa; - -if (WARN_ON (!VALID_PAGE (hpa))) -return false; - -/* -* A NULL shadow page is legal when shadowing a non-paging guest with -* PAE paging, as the MMU will be direct with root_hpa pointing at the -* pae_root page, not a shadow page. -*/ - sp =to_shadow_page (hpa); -return sp &&is_tdp_mmu_page (sp) && sp->root_count; -}#else static inline bool is_tdp_mmu_page (struct kvm_mmu_page *sp) {return false; } -static inline bool is_tdp_mmu (struct kvm_mmu *mmu) {return false; }#endif #endif /* __KVM_X86_MMU_TDP_MMU_H */
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Sean Christopherson
maintainerPATCH v4 9/11 KVM: x86/mmu: Replace open coded usage of tdp_mmu_page with is_tdp_mmu_page()
In reply to Sean ChristophersonUse is_tdp_mmu_page() instead of querying sp->tdp_mmu_page directly so that all users benefit if KVM ever finds a way to optimize the logic.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/mmu/tdp_mmu.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -1907,7 +1907,7 @@ static bool is_obsolete_sp(struct kvm *kvm, struct kvm_mmu_page *sp)return true;/* TDP MMU pages due not use the MMU generation. */ -return !sp->tdp_mmu_page && +return !is_tdp_mmu_page (sp) &&unlikely (sp->mmu_valid_gen != kvm->arch.mmu_valid_gen); }
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
@@ -134,7 +134,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,if (!refcount_dec_and_test (&root->tdp_mmu_root_count))return ; -WARN_ON (!root->tdp_mmu_page); +WARN_ON (!is_tdp_mmu_page (root));/* * The root now has refcount=0. It is valid, but readers already
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Sean Christopherson
maintainerPATCH v4 10/11 KVM: x86/mmu: Use static key/branches for checking if TDP MMU is enabled
In reply to Sean ChristophersonNow that the TDP MMU being enabled is read-only after the vendor module is loaded, use a static key to track whether or not the TDP MMU is enabled to avoid conditional branches in hot paths, e.g. in direct_page_fault() and fast_page_fault().
--- arch/x86/kvm/mmu.h | 5 +++-- arch/x86/kvm/mmu/mmu.c | 14 ++++++++++---- 2 files changed, 13 insertions(+), 6 deletions(-)
arch/x86/kvm/mmu.h +3 -2
@@ -2,6 +2,7 @@#ifndef __KVM_X86_MMU_H#define __KVM_X86_MMU_H +#include <linux/jump_label.h> #include <linux/kvm_host.h> #include "kvm_cache_regs.h" #include "cpuid.h"
@@ -230,13 +231,13 @@ static inline bool kvm_shadow_root_allocated(struct kvm *kvm) }#ifdef CONFIG_X86_64 -extern bool tdp_mmu_enabled; +DECLARE_STATIC_KEY_TRUE (tdp_mmu_enabled);#endif static inline bool is_tdp_mmu_enabled (void ) {#ifdef CONFIG_X86_64 -return tdp_mmu_enabled; +return static_branch_likely (&tdp_mmu_enabled);#else return false;#endif
arch/x86/kvm/mmu/mmu.c +10 -4
@@ -101,8 +101,10 @@ bool tdp_enabled = false;#ifdef CONFIG_X86_64 static bool __ro_after_init tdp_mmu_allowed; -bool __read_mostly tdp_mmu_enabled = true; -module_param_named (tdp_mmu, tdp_mmu_enabled, bool,0444 ); +static bool __read_mostly __tdp_mmu_enabled = true; +module_param_named (tdp_mmu, __tdp_mmu_enabled, bool,0444 ); + +DEFINE_STATIC_KEY_TRUE (tdp_mmu_enabled);#endif static int max_huge_page_level__read_mostly ;
@@ -5702,7 +5704,11 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level, max_tdp_level = tdp_max_root_level;#ifdef CONFIG_X86_64 - tdp_mmu_enabled = tdp_mmu_allowed && tdp_enabled; + __tdp_mmu_enabled = tdp_mmu_allowed && tdp_enabled; +if (__tdp_mmu_enabled) +static_branch_enable (&tdp_mmu_enabled); +else +static_branch_disable (&tdp_mmu_enabled);#endif /* * max_huge_page_level reflects KVM's MMU capabilities irrespective
@@ -6712,7 +6718,7 @@ void __init kvm_mmu_x86_module_init(void)* TDP MMU is actually enabled is determined in kvm_configure_mmu() * when the vendor module is loaded. */ - tdp_mmu_allowed = tdp_mmu_enabled; + tdp_mmu_allowed = __tdp_mmu_enabled;#endif kvm_mmu_spte_module_init ();
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Sean Christopherson
maintainerPATCH v4 11/11 KVM: x86/mmu: Stop needlessly making MMU pages available for TDP MMU
In reply to Sean ChristophersonFrom: David Matlack <[email protected]>
Stop calling make_mmu_pages_available() when handling TDP MMU faults and when allocating TDP MMU roots. The TDP MMU does not participate in the "available MMU pages" tracking and limiting so calling this function is unnecessary work when handling TDP MMU faults.
--- arch/x86/kvm/mmu/mmu.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-)
arch/x86/kvm/mmu/mmu.c +13 -9
@@ -3569,9 +3569,12 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)int r;write_lock (&vcpu->kvm->mmu_lock); - r =make_mmu_pages_available (vcpu); -if (r <0 ) - goto out_unlock; + +if (!is_tdp_mmu_enabled ()) { + r =make_mmu_pages_available (vcpu); +if (r <0 ) + goto out_unlock; + }if (is_tdp_mmu_enabled ()) { root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu);
@@ -4289,14 +4292,15 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultif (is_page_fault_stale (vcpu, fault)) goto out_unlock; - r =make_mmu_pages_available (vcpu); -if (r) - goto out_unlock; - -if (is_tdp_mmu_enabled ()) +if (is_tdp_mmu_enabled ()) { r =kvm_tdp_mmu_map (vcpu, fault); -else + }else { + r =make_mmu_pages_available (vcpu); +if (r) + goto out_unlock; + r =__direct_map (vcpu, fault); + } out_unlock:if (is_tdp_mmu_enabled ())
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Quoted 27 lines · Sean Christopherson
This is a variation of David's series to change
tdp_mmuto a RO param[*]. The key difference is that instead of moving the TDP MMU page fault handler to its own function, use static branches to make TDP MMU page faults (and a few other paths) effectively branch free.I'm not dead set against having a dedicated TDP MMU page fault handler, but IMO it's not really better once the TDP MMU vs. shadow MMU is reduced to a static branch, just different. The read vs. write
mmu_lockis the most visible ugliness, and that can be buried in helpers if we really want to make the page fault handler easier on the eyes, e.g.direct_page_fault_mmu_lock(vcpu); if (is_page_fault_stale(vcpu, fault)) goto out_unlock; if (is_tdp_mmu_enabled()) { r = kvm_tdp_mmu_map(vcpu, fault); } else { r = make_mmu_pages_available(vcpu); if (r) goto out_unlock; r = __direct_map(vcpu, fault); }
Quoted 2 lines
out_unlock:direct_page_fault_mmu_unlock(vcpu);
Thanks for the respin.
My preference is still separate handlers. When I am reading this code, I only care about one path (TDP MMU or Shadow MMU, usually TDP MMU). Having separate handlers makes it easy to read since I don't have to care about the implementation details of the other MMU.
And more importantly (but less certain), the TDP MMU fault handler is going to diverge further from the Shadow MMU fault handler in the near future. i.e. There will be more and more branches in a common fault handler, and the value of having a common fault handler diminishes. Specifically, to support moving the TDP MMU to common code, the TDP MMU is no longer going to topup the same mem caches as the Shadow MMU (TDP MMU is not going to use struct kvm_mmu_page), and the TDP MMU will probably have its own fast_page_fault() handler eventually.
If we do go the common handler route, I don't prefer the direct_page_fault_mmu_lock/unlock() wrapper since it further obscures the differences between the 2 MMUs.
Quoted 44 lines
v4:
- Keepis_tdp_mmu_page()in patch 1.
- Collect reviews. [Isaku]
- Skip "make MMU pages available" for root allocations.
- Rework "is TDP MMU" checks to take advantage of read-only param.
- Use a static key to track TDP MMU enabling.[*] https://lkml.kernel.org/r/[email protected]
David Matlack (7):
KVM: x86/mmu: Changetdp_mmuto a read-only parameter
KVM: x86/mmu: Move TDP MMU VM init/uninit behindtdp_mmu_enabled
KVM: x86/mmu: Grabmmu_invalidate_seqinkvm_faultin_pfn()
KVM: x86/mmu: Handle error PFNs inkvm_faultin_pfn()
KVM: x86/mmu: Avoid memslot lookup duringKVM_PFN_ERR_HWPOISON
handling
KVM: x86/mmu: Handle no-slot faults inkvm_faultin_pfn()
KVM: x86/mmu: Stop needlessly making MMU pages available for TDP MMUSean Christopherson (4):
KVM: x86/mmu: Pivot on "TDP MMU enabled" when handling direct page
faults
KVM: x86/mmu: Pivot on "TDP MMU enabled" to check if active MMU is TDP
MMU
KVM: x86/mmu: Replace open coded usage oftdp_mmu_pagewithis_tdp_mmu_page()KVM: x86/mmu: Use static key/branches for checking if TDP MMU is
enabledarch/x86/include/asm/kvm_host.h | 9 -- arch/x86/kvm/mmu.h | 14 ++- arch/x86/kvm/mmu/mmu.c | 212 ++++++++++++++++++++------------ arch/x86/kvm/mmu/mmu_internal.h | 1 + arch/x86/kvm/mmu/paging_tmpl.h | 12 +- arch/x86/kvm/mmu/tdp_mmu.c | 13 +- arch/x86/kvm/mmu/tdp_mmu.h | 25 +--- 7 files changed, 149 insertions(+), 137 deletions(-)base-commit:
e18d6152ff0f41b7f01f9817372022df04e0d354--
2.38.0.rc1.362.ged0d419d3c-goog
Sean Christopherson
maintainerQuoted 6 lines · David Matlack
Quoted 5 lines · Sean Christopherson
I'm not dead set against having a dedicated TDP MMU page fault handler, but IMO it's not really better once the TDP MMU vs. shadow MMU is reduced to a static branch, just different. The read vs. write
mmu_lockis the most visible ugliness, and that can be buried in helpers if we really want to make the page fault handler easier on the eyes, e.g.
...
Quoted 5 lines
My preference is still separate handlers. When I am reading this code, I only care about one path (TDP MMU or Shadow MMU, usually TDP MMU). Having separate handlers makes it easy to read since I don't have to care about the implementation details of the other MMU.
Quoted 8 lines
And more importantly (but less certain), the TDP MMU fault handler is going to diverge further from the Shadow MMU fault handler in the near future. i.e. There will be more and more branches in a common fault handler, and the value of having a common fault handler diminishes. Specifically, to support moving the TDP MMU to common code, the TDP MMU is no longer going to topup the same mem caches as the Shadow MMU (TDP MMU is not going to use struct
kvm_mmu_page), and the TDP MMU will probably have its ownfast_page_fault()handler eventually.
What if we hold off on the split for the moment, and then revisit the handler when a common MMU is closer to reality? I agree that a separate handler makes sense once things start diverging, but until that happens, supporting two flows instead of one seems like it would add (minor) maintenance cost without much benefit.
Quoted 3 lines
If we do go the common handler route, I don't prefer the direct_page_fault_mmu_lock/unlock() wrapper since it further obscures the differences between the 2 MMUs.
Yeah, I don't like the wrappers either.
Quoted 25 lines · Sean Christopherson
Quoted 6 lines · David Matlack
Quoted 5 lines · Sean Christopherson
I'm not dead set against having a dedicated TDP MMU page fault handler, but IMO it's not really better once the TDP MMU vs. shadow MMU is reduced to a static branch, just different. The read vs. write
mmu_lockis the most visible ugliness, and that can be buried in helpers if we really want to make the page fault handler easier on the eyes, e.g....
Quoted 5 lines
My preference is still separate handlers. When I am reading this code, I only care about one path (TDP MMU or Shadow MMU, usually TDP MMU). Having separate handlers makes it easy to read since I don't have to care about the implementation details of the other MMU.
Quoted 8 lines
And more importantly (but less certain), the TDP MMU fault handler is going to diverge further from the Shadow MMU fault handler in the near future. i.e. There will be more and more branches in a common fault handler, and the value of having a common fault handler diminishes. Specifically, to support moving the TDP MMU to common code, the TDP MMU is no longer going to topup the same mem caches as the Shadow MMU (TDP MMU is not going to use struct
kvm_mmu_page), and the TDP MMU will probably have its ownfast_page_fault()handler eventually.
Quoted 4 lines
What if we hold off on the split for the moment, and then revisit the handler when a common MMU is closer to reality? I agree that a separate handler makes sense once things start diverging, but until that happens, supporting two flows instead of one seems like it would add (minor) maintenance cost without much benefit.
Sure thing. I'll do the split as part of my series to split out the TDP MMU to common code and we can revisit the discussion then.
Quoted 6 lines
Quoted 4 lines
If we do go the common handler route, I don't prefer the direct_page_fault_mmu_lock/unlock() wrapper since it further obscures the differences between the 2 MMUs.
Yeah, I don't like the wrappers either.