FAILED: patch "[PATCH] KVM: x86: Check for invalid/obsolete root *after* making MMU" failed to apply to 5.15-stable tree
Subsystems (3)
[email protected]
maintainerThe patch below does not apply to the 5.15-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
To reproduce the conflict and resubmit, you may use the following commands:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-5.15.y
git checkout FETCH_HEAD
git cherry-pick -x 2abd5287f08319fa35764566b15c6e22cb1068db
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2026072557-gem-defensive-a757@gregkh' --subject-prefix 'PATCH 5.15.y' 'HEAD^..'
Possible dependencies:
thanks,
greg k-h
------------------ original commit in Linus's tree ------------------
From 2abd5287f08319fa35764566b15c6e22cb1068db Mon Sep 17 00:00:00 2001
From: Sean Christopherson <[email protected]>
Date: Mon, 13 Jul 2026 08:15:33 -0700
Subject: [PATCH] KVM: x86: Check for invalid/obsolete root *after* making MMU
pages available
Check for a "stale" page fault, i.e. for an invalid and/or obsolete root, after making MMU pages available for the shadow MMU. If reclaiming shadow pages zaps an in-use root, i.e. marks it invalid, then KVM will attempt to map memory into an invalid root. On its own, populating an invalid root is "fine", but because child shadow pages inherit their parent's role, any children created during the map/fetch will be created as invalid pages, thus violating KVM's invariant that invalid pages are never on the list of active MMU pages.
Note, the underlying flaw has existed since KVM first started tracking invalid roots in 2008 (commit 2e53d63acba7, "KVM: MMU: ignore zapped root pagetables"), but the true badness only came along in 2020 (Linux 5.9) with the invariant that invalid shadow pages can't be on the list of active pages.
Note #2, inheriting role.invalid when creating child shadow pages is also far from ideal; that flaw will be addressed separately.
arch/x86/kvm/mmu/mmu.c +5 -4
@@ -4852,16 +4852,17 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultif (r !=RET_PF_CONTINUE )return r; - r =RET_PF_RETRY ;write_lock (&vcpu->kvm->mmu_lock); -if (is_page_fault_stale (vcpu, fault)) - goto out_unlock; - r =make_mmu_pages_available (vcpu);if (r) goto out_unlock; +if (is_page_fault_stale (vcpu, fault)) { + r =RET_PF_RETRY ; + goto out_unlock; + } + r =direct_map (vcpu, fault); out_unlock:
arch/x86/kvm/mmu/paging_tmpl.h +6 -4
@@ -864,15 +864,17 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault }#endif - r =RET_PF_RETRY ;write_lock (&vcpu->kvm->mmu_lock); -if (is_page_fault_stale (vcpu, fault)) - goto out_unlock; - r =make_mmu_pages_available (vcpu);if (r) goto out_unlock; + +if (is_page_fault_stale (vcpu, fault)) { + r =RET_PF_RETRY ; + goto out_unlock; + } + r =FNAME (fetch)(vcpu, fault, &walker); out_unlock:
Sasha Levin
maintainerPATCH 1/6 KVM: x86/mmu: Directly "destroy" PTE list when recycling rmaps
From: Sean Christopherson <[email protected]>
[ Upstream commit a42989e7fbb0186d9fee05b29e0ea9cb639d0bd3 ]
Use pte_list_destroy() directly when recycling rmaps instead of bouncing through kvm_unmap_rmapp() and kvm_zap_rmapp(). Calling kvm_unmap_rmapp() is unnecessary and odd as it requires passing dummy parameters; passing NULL for @slot when __rmap_add() already has a valid slot is especially weird and confusing.
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/x86/kvm/mmu/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -1639,7 +1639,7 @@ static void rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) rmap_count =pte_list_add (vcpu, spte, rmap_head);if (rmap_count >RMAP_RECYCLE_THRESHOLD ) { -kvm_unmap_rmapp (vcpu->kvm, rmap_head,NULL , gfn, sp->role.level,__pte (0 )); +pte_list_destroy (vcpu->kvm, rmap_head);kvm_flush_remote_tlbs_with_address ( vcpu->kvm, sp->gfn,KVM_PAGES_PER_HPAGE (sp->role.level)); }
Signature · 1 line
2.53.0
Sasha Levin
maintainerPATCH 2/6 KVM: x86/mmu: Rename pte_list_{destroy,remove}() to show they zap SPTEs
From: Sean Christopherson <[email protected]>
[ Upstream commit 9202aee816c84d69179f94193c5dd321bb0e8530 ]
Rename pte_list_remove() and pte_list_destroy() to kvm_zap_one_rmap_spte() and kvm_zap_all_rmap_sptes() respectively to document that (a) they zap SPTEs and (b) to better document how they differ (remove vs. destroy does not exactly scream "one vs. all").
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/x86/kvm/mmu/mmu.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +9 -8
@@ -999,15 +999,16 @@ static void __pte_list_remove(u64 *spte, struct kvm_rmap_head *rmap_head) } } -static void pte_list_remove (struct kvm *kvm,struct kvm_rmap_head *rmap_head, -u64 *sptep) +static void kvm_zap_one_rmap_spte (struct kvm *kvm, +struct kvm_rmap_head *rmap_head,u64 *sptep) {mmu_spte_clear_track_bits (kvm, sptep);__pte_list_remove (sptep, rmap_head); } -/* Return true if rmap existed, false otherwise */ -static bool pte_list_destroy (struct kvm *kvm,struct kvm_rmap_head *rmap_head) +/* Return true if at least one SPTE was zapped, false otherwise */ +static bool kvm_zap_all_rmap_sptes (struct kvm *kvm, +struct kvm_rmap_head *rmap_head) {struct pte_list_desc *desc, *next;int i;
@@ -1434,7 +1435,7 @@ static bool rmap_write_protect(struct kvm_vcpu *vcpu, u64 gfn)static bool kvm_zap_rmapp (struct kvm *kvm,struct kvm_rmap_head *rmap_head,const struct kvm_memory_slot *slot) { -return pte_list_destroy (kvm, rmap_head); +return kvm_zap_all_rmap_sptes (kvm, rmap_head); } static bool kvm_unmap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
@@ -1465,7 +1466,7 @@ static bool kvm_set_pte_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, need_flush =1 ;if (pte_write (pte)) { -pte_list_remove (kvm, rmap_head, sptep); +kvm_zap_one_rmap_spte (kvm, rmap_head, sptep); goto restart; }else { new_spte = kvm_mmu_changed_pte_notifier_make_spte(
@@ -1639,7 +1640,7 @@ static void rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) rmap_count =pte_list_add (vcpu, spte, rmap_head);if (rmap_count >RMAP_RECYCLE_THRESHOLD ) { -pte_list_destroy (vcpu->kvm, rmap_head); +kvm_zap_all_rmap_sptes (vcpu->kvm, rmap_head);kvm_flush_remote_tlbs_with_address ( vcpu->kvm, sp->gfn,KVM_PAGES_PER_HPAGE (sp->role.level)); }
@@ -5855,7 +5856,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, !kvm_is_reserved_pfn(pfn) && sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn, pfn, PG_LEVEL_NUM)) { -pte_list_remove (kvm, rmap_head, sptep); +kvm_zap_one_rmap_spte (kvm, rmap_head, sptep);if (kvm_available_flush_tlb_with_range ()) kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
Signature · 1 line
2.53.0
Sasha Levin
maintainerPATCH 3/6 KVM: x86/mmu: Document the "rules" for using host_pfn_mapping_level()
In reply to Sasha LevinFrom: Sean Christopherson <[email protected]>
[ Upstream commit 65e3b446bcceaac7448cb25a2a5bf4adbcf25fe6 ]
Add a comment to document how host_pfn_mapping_level() can be used safely, as the line between safe and dangerous is quite thin. E.g. if KVM were to ever support in-place promotion to create huge pages, consuming the level is safe if the caller holds mmu_lock and checks that there's an existing _leaf_ SPTE, but unsafe if the caller only checks that there's a non-leaf SPTE.
Opportunistically tweak the existing comments to explicitly document why KVM needs to use READ_ONCE().
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/x86/kvm/mmu/mmu.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
arch/x86/kvm/mmu/mmu.c +25 -0
@@ -2869,6 +2869,31 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)__direct_pte_prefetch (vcpu, sp, sptep); } +/* +* Lookup the mapping level for @gfn in the current mm. +* +* WARNING! Use of host_pfn_mapping_level() requires the caller and the end +* consumer to be tied into KVM's handlers for MMU notifier events! +* +* There are several ways to safely use this helper: +* +* - Check mmu_notifier_retry_hva() after grabbing the mapping level, before +* consuming it. In this case, mmu_lock doesn't need to be held during the +* lookup, but it does need to be held while checking the MMU notifier. +* +* - Hold mmu_lock AND ensure there is no in-progress MMU notifier invalidation +* event for the hva. This can be done by explicit checking the MMU notifier +* or by ensuring that KVM already has a valid mapping that covers the hva. +* +* - Do not use the result to install new mappings, e.g. use the host mapping +* level only to decide whether or not to zap an entry. In this case, it's +* not required to hold mmu_lock (though it's highly likely the caller will +* want to hold mmu_lock anyways, e.g. to modify SPTEs). +* +* Note! The lookup can still race with modifications to host page tables, but +* the above "rules" ensure KVM will not _consume_ the result of the walk if a +* race with the primary MMU occurs. +*/ static int host_pfn_mapping_level (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn,const struct kvm_memory_slot *slot) {
Signature · 1 line
2.53.0
Sasha Levin
maintainerPATCH 4/6 KVM: Rename mmu_notifier_* to mmu_invalidate_*
In reply to Sasha LevinFrom: Chao Peng <[email protected]>
[ Upstream commit 20ec3ebd707c77fb9b11b37193449193d4649f33 ]
The motivation of this renaming is to make these variables and related helper functions less mmu_notifier bound and can also be used for non mmu_notifier based page invalidation. mmu_invalidate_* was chosen to better describe the purpose of 'invalidating' a page that those variables are used for.
- mmu_notifier_seq/range_start/range_end are renamed to mmu_invalidate_seq/range_start/range_end.
- mmu_notifier_retry{_hva} helper functions are renamed to mmu_invalidate_retry{_hva}.
- mmu_notifier_count is renamed to mmu_invalidate_in_progress to avoid confusion with mn_active_invalidate_count.
- While here, also update kvm_inc/dec_notifier_count() to kvm_mmu_invalidate_begin/end() to match the change for mmu_notifier_count.
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/arm64/kvm/mmu.c | 10 ++-- arch/mips/kvm/mmu.c | 12 ++--- arch/powerpc/include/asm/kvm_book3s_64.h | 2 +- arch/powerpc/kvm/book3s_64_mmu_host.c | 4 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 6 +-- arch/powerpc/kvm/book3s_64_vio_hv.c | 2 +- arch/powerpc/kvm/book3s_hv_nested.c | 2 +- arch/powerpc/kvm/book3s_hv_rm_mmu.c | 8 ++-- arch/powerpc/kvm/e500_mmu_host.c | 4 +- arch/x86/kvm/mmu/mmu.c | 14 +++--- arch/x86/kvm/mmu/paging_tmpl.h | 6 +-- include/linux/kvm_host.h | 60 ++++++++++++------------ virt/kvm/kvm_main.c | 49 +++++++++---------- 14 files changed, 93 insertions(+), 90 deletions(-)
arch/arm64/kvm/mmu.c +5 -5
@@ -881,7 +881,7 @@ transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot,* THP doesn't start to split while we are adjusting the * refcounts. * -* We are sure this doesn't happen, because mmu_notifier_retry +* We are sure this doesn't happen, because mmu_invalidate_retry * was successful and we are holding the mmu_lock, so if this * THP is trying to split, it will be blocked in the mmu * notifier before touching any of the pages, specifically
@@ -1074,14 +1074,14 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, gfn = fault_ipa >>PAGE_SHIFT ;/* -* Read mmu_notifier_seq so that KVM can detect if the results of +* Read mmu_invalidate_seq so that KVM can detect if the results of * vma_lookup() or __gfn_to_pfn_memslot() become stale prior to * acquiring kvm->mmu_lock. * * Rely on mmap_read_unlock() for an implicit smp_rmb(), which pairs -* with the smp_wmb() in kvm_dec_notifier_count(). +* with the smp_wmb() in kvm_mmu_invalidate_end(). */ - mmu_seq = vcpu->kvm->mmu_notifier_seq; + mmu_seq = vcpu->kvm->mmu_invalidate_seq;mmap_read_unlock (current->mm); pfn = __gfn_to_pfn_memslot(memslot, gfn, false, NULL,
@@ -1118,7 +1118,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,spin_lock (&kvm->mmu_lock); pgt = vcpu->arch.hw_mmu->pgt; -if (mmu_notifier_retry (kvm, mmu_seq)) +if (mmu_invalidate_retry (kvm, mmu_seq)) goto out_unlock;/*
arch/mips/kvm/mmu.c +6 -6
@@ -615,17 +615,17 @@ static int kvm_mips_map_page(struct kvm_vcpu *vcpu, unsigned long gpa,* Used to check for invalidations in progress, of the pfn that is * returned by pfn_to_pfn_prot below. */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;/* -* Ensure the read of mmu_notifier_seq isn't reordered with PTE reads in -* gfn_to_pfn_prot() (which calls get_user_pages()), so that we don't +* Ensure the read of mmu_invalidate_seq isn't reordered with PTE reads +* in gfn_to_pfn_prot() (which calls get_user_pages()), so that we don't * risk the page we get a reference to getting unmapped before we have a -* chance to grab the mmu_lock without mmu_notifier_retry() noticing. +* chance to grab the mmu_lock without mmu_invalidate_retry() noticing. * * This smp_rmb() pairs with the effective smp_wmb() of the combination * of the pte_unmap_unlock() after the PTE is zapped, and the * spin_lock() in kvm_mmu_notifier_invalidate_<page|range_end>() before -* mmu_notifier_seq is incremented. +* mmu_invalidate_seq is incremented. */ smp_rmb ();
@@ -638,7 +638,7 @@ static int kvm_mips_map_page(struct kvm_vcpu *vcpu, unsigned long gpa,spin_lock (&kvm->mmu_lock);/* Check if an invalidation has taken place since we got pfn */ -if (mmu_notifier_retry (kvm, mmu_seq)) { +if (mmu_invalidate_retry (kvm, mmu_seq)) {/* * This can happen when mappings are changed asynchronously, but * also synchronously if a COW is triggered by
arch/powerpc/include/asm/kvm_book3s_64.h +1 -1
@@ -673,7 +673,7 @@ static inline pte_t *find_kvm_host_pte(struct kvm *kvm, unsigned long mmu_seq,VM_WARN (!spin_is_locked (&kvm->mmu_lock),"%s called with kvm mmu_lock not held \n" , __func__); -if (mmu_notifier_retry (kvm, mmu_seq)) +if (mmu_invalidate_retry (kvm, mmu_seq))return NULL ; pte =__find_linux_pte (kvm->mm->pgd, ea,NULL , hshift);
arch/powerpc/kvm/book3s_64_mmu_host.c +2 -2
@@ -90,7 +90,7 @@ int kvmppc_mmu_map_page(struct kvm_vcpu *vcpu, struct kvmppc_pte *orig_pte,unsigned long pfn;/* used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();/* Get host physical address for gpa */
@@ -151,7 +151,7 @@ int kvmppc_mmu_map_page(struct kvm_vcpu *vcpu, struct kvmppc_pte *orig_pte, cpte =kvmppc_mmu_hpte_cache_next (vcpu);spin_lock (&kvm->mmu_lock); -if (!cpte ||mmu_notifier_retry (kvm, mmu_seq)) { +if (!cpte ||mmu_invalidate_retry (kvm, mmu_seq)) { r = -EAGAIN ; goto out_unlock; }
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -2
@@ -570,7 +570,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_vcpu *vcpu,return -EFAULT ;/* used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb (); ret = -EFAULT ;
@@ -685,7 +685,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_vcpu *vcpu,/* Check if we might have been invalidated; let the guest retry if so */ ret =RESUME_GUEST ; -if (mmu_notifier_retry (vcpu->kvm, mmu_seq)) { +if (mmu_invalidate_retry (vcpu->kvm, mmu_seq)) {unlock_rmap (rmap); goto out_unlock; }
arch/powerpc/kvm/book3s_64_mmu_radix.c +3 -3
@@ -634,7 +634,7 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Check if we might have been invalidated; let the guest retry if so */ spin_lock (&kvm->mmu_lock); ret = -EAGAIN ; -if (mmu_notifier_retry (kvm, mmu_seq)) +if (mmu_invalidate_retry (kvm, mmu_seq)) goto out_unlock;/* Now traverse again under the lock and change the tree */
@@ -824,7 +824,7 @@ int kvmppc_book3s_instantiate_page(struct kvm_vcpu *vcpu,bool large_enable;/* used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();/*
@@ -1185,7 +1185,7 @@ void kvmppc_radix_flush_memslot(struct kvm *kvm,* Increase the mmu notifier sequence number to prevent any page * fault that read the memslot earlier from writing a PTE. */ - kvm->mmu_notifier_seq++; + kvm->mmu_invalidate_seq++;spin_unlock (&kvm->mmu_lock); }
arch/powerpc/kvm/book3s_64_vio_hv.c +1 -1
@@ -499,7 +499,7 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,/* * used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb (); stt =kvmppc_find_table (vcpu->kvm, liobn);
arch/powerpc/kvm/book3s_hv_nested.c +1 -1
@@ -1574,7 +1574,7 @@ static long int __kvmhv_nested_page_fault(struct kvm_vcpu *vcpu,/* 2. Find the host pte for this L1 guest real address */ /* Used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();/* See if can find translation in our partition scoped tables for L1 */
arch/powerpc/kvm/book3s_hv_rm_mmu.c +4 -4
@@ -216,7 +216,7 @@ long kvmppc_do_h_enter(struct kvm *kvm, unsigned long flags, g_ptel = ptel;/* used later to detect if we might have been invalidated */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();/* Find the memslot (if any) for this address */
@@ -363,7 +363,7 @@ long kvmppc_do_h_enter(struct kvm *kvm, unsigned long flags, rmap =real_vmalloc_addr (rmap);lock_rmap (rmap);/* Check for pending invalidations under the rmap chain lock */ -if (mmu_notifier_retry (kvm, mmu_seq)) { +if (mmu_invalidate_retry (kvm, mmu_seq)) {/* inval in progress, write a non-present HPTE */ pteh |=HPTE_V_ABSENT ; pteh &= ~HPTE_V_VALID ;
@@ -929,7 +929,7 @@ static long kvmppc_do_h_page_init_zero(struct kvm_vcpu *vcpu,int i;/* Used later to detect if we might have been invalidated */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();arch_spin_lock (&kvm->mmu_lock.rlock.raw_lock);
@@ -957,7 +957,7 @@ static long kvmppc_do_h_page_init_copy(struct kvm_vcpu *vcpu,long ret =H_SUCCESS ;/* Used later to detect if we might have been invalidated */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();arch_spin_lock (&kvm->mmu_lock.rlock.raw_lock);
arch/powerpc/kvm/e500_mmu_host.c +2 -2
@@ -339,7 +339,7 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500,unsigned long flags;/* used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();/*
@@ -460,7 +460,7 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500, }spin_lock (&kvm->mmu_lock); -if (mmu_notifier_retry (kvm, mmu_seq)) { +if (mmu_invalidate_retry (kvm, mmu_seq)) { ret = -EAGAIN ; goto out; }
arch/x86/kvm/mmu/mmu.c +7 -7
@@ -2863,7 +2863,7 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)* If addresses are being invalidated, skip prefetching to avoid * accidentally prefetching those addresses. */ -if (unlikely (vcpu->kvm->mmu_notifier_count)) +if (unlikely (vcpu->kvm->mmu_invalidate_in_progress))return ;__direct_pte_prefetch (vcpu, sp, sptep);
@@ -2877,7 +2877,7 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)* * There are several ways to safely use this helper: * -* - Check mmu_notifier_retry_hva() after grabbing the mapping level, before +* - Check mmu_invalidate_retry_hva() after grabbing the mapping level, before * consuming it. In this case, mmu_lock doesn't need to be held during the * lookup, but it does need to be held while checking the MMU notifier. *
@@ -2972,7 +2972,7 @@ int kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, gfn_t gfn,return PG_LEVEL_4K ;/* -* mmu_notifier_retry() was successful and mmu_lock is held, so +* mmu_invalidate_retry() was successful and mmu_lock is held, so * the pmd can't be split from under us. */ mask =KVM_PAGES_PER_HPAGE (level) -1 ;
@@ -4018,7 +4018,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,if (r)return r; - mmu_seq = vcpu->kvm->mmu_notifier_seq; + mmu_seq = vcpu->kvm->mmu_invalidate_seq;smp_rmb (); if (kvm_faultin_pfn(vcpu, prefault, gfn, gpa, &pfn, &hva,
@@ -4035,7 +4035,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,else write_lock (&vcpu->kvm->mmu_lock); -if (!is_noslot_pfn (pfn) &&mmu_notifier_retry_hva (vcpu->kvm, mmu_seq, hva)) +if (!is_noslot_pfn (pfn) &&mmu_invalidate_retry_hva (vcpu->kvm, mmu_seq, hva)) goto out_unlock;if (is_tdp_mmu_fault) {
@@ -5767,7 +5767,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)write_lock (&kvm->mmu_lock); -kvm_inc_notifier_count (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_begin (kvm, gfn_start, gfn_end);if (kvm_memslots_have_rmaps (kvm)) {for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) {
@@ -5802,7 +5802,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)kvm_flush_remote_tlbs_with_address (kvm, gfn_start, gfn_end - gfn_start); -kvm_dec_notifier_count (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_end (kvm, gfn_start, gfn_end);write_unlock (&kvm->mmu_lock); }
arch/x86/kvm/mmu/paging_tmpl.h +3 -3
@@ -634,7 +634,7 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,* If addresses are being invalidated, skip prefetching to avoid * accidentally prefetching those addresses. */ -if (unlikely (vcpu->kvm->mmu_notifier_count)) +if (unlikely (vcpu->kvm->mmu_invalidate_in_progress))return ;if (sp->role.direct)
@@ -894,7 +894,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gpa_t addr, u32 error_code,else max_level = walker.level; - mmu_seq = vcpu->kvm->mmu_notifier_seq; + mmu_seq = vcpu->kvm->mmu_invalidate_seq;smp_rmb (); if (kvm_faultin_pfn(vcpu, prefault, walker.gfn, addr, &pfn, &hva,
@@ -925,7 +925,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gpa_t addr, u32 error_code, r =RET_PF_RETRY ;write_lock (&vcpu->kvm->mmu_lock); -if (!is_noslot_pfn (pfn) &&mmu_notifier_retry_hva (vcpu->kvm, mmu_seq, hva)) +if (!is_noslot_pfn (pfn) &&mmu_invalidate_retry_hva (vcpu->kvm, mmu_seq, hva)) goto out_unlock;kvm_mmu_audit (vcpu,AUDIT_PRE_PAGE_FAULT );
include/linux/kvm_host.h +31 -29
93 lines not on this page · raw message
virt/kvm/kvm_main.c +25 -24
115 lines not on this page · raw message
Signature · 1 line
2.53.0
Sasha Levin
maintainerPATCH 5/6 KVM: x86/mmu: Split out TDP MMU page fault handling
In reply to Sasha LevinFrom: David Matlack <[email protected]>
[ Upstream commit 9aa8ab43b38146029de807a8ff2696f51e15b226 ]
Split out the page fault handling for the TDP MMU to a separate function. This creates some duplicate code, but makes the TDP MMU fault handler simpler to read by eliminating branches and will enable future cleanups by allowing the TDP MMU and non-TDP MMU fault paths to diverge.
Only compile in the TDP MMU fault handler for 64-bit builds since kvm_tdp_mmu_map() does not exist in 32-bit builds.
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/x86/kvm/mmu/mmu.c | 84 ++++++++++++++++++++++++++++++++---------- 1 file changed, 64 insertions(+), 20 deletions(-)
arch/x86/kvm/mmu/mmu.c +64 -20
@@ -3997,7 +3997,6 @@ static bool kvm_faultin_pfn(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn,static int direct_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code,bool prefault,int max_level,bool is_tdp) { -bool is_tdp_mmu_fault =is_tdp_mmu (vcpu->arch.mmu);bool write = error_code &PFERR_WRITE_MASK ;bool map_writable;
@@ -4029,31 +4028,20 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,return r; r =RET_PF_RETRY ; - -if (is_tdp_mmu_fault) -read_lock (&vcpu->kvm->mmu_lock); -else -write_lock (&vcpu->kvm->mmu_lock); +write_lock (&vcpu->kvm->mmu_lock);if (!is_noslot_pfn (pfn) &&mmu_invalidate_retry_hva (vcpu->kvm, mmu_seq, hva)) goto out_unlock; -if (is_tdp_mmu_fault) { - r =kvm_tdp_mmu_map (vcpu, gpa, error_code, map_writable, max_level, - pfn, prefault); - }else { - r =make_mmu_pages_available (vcpu); -if (r) - goto out_unlock; - r =__direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, - prefault, is_tdp); - } + r =make_mmu_pages_available (vcpu); +if (r) + goto out_unlock; + + r =__direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, + prefault, is_tdp); out_unlock: -if (is_tdp_mmu_fault) -read_unlock (&vcpu->kvm->mmu_lock); -else -write_unlock (&vcpu->kvm->mmu_lock); +write_unlock (&vcpu->kvm->mmu_lock);kvm_release_pfn_clean (pfn);return r; }
@@ -4101,6 +4089,56 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, }EXPORT_SYMBOL_GPL (kvm_handle_page_fault); +#ifdef CONFIG_X86_64 +static int kvm_tdp_mmu_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa, +u32 error_code,bool prefault,int max_level) +{ +bool write = error_code &PFERR_WRITE_MASK ; +bool map_writable; + +gfn_t gfn = gpa >>PAGE_SHIFT ; +unsigned long mmu_seq; +kvm_pfn_t pfn; +hva_t hva; +int r; + +if (page_fault_handle_page_track (vcpu, error_code, gfn)) +return RET_PF_EMULATE ; + + r =fast_page_fault (vcpu, gpa, error_code); +if (r !=RET_PF_INVALID ) +return r; + + r =mmu_topup_memory_caches (vcpu, false); +if (r) +return r; + + mmu_seq = vcpu->kvm->mmu_invalidate_seq; +smp_rmb (); + +if (kvm_faultin_pfn (vcpu, prefault, gfn, gpa, &pfn, &hva, + write, &map_writable, &r)) +return r; + +if (handle_abnormal_pfn (vcpu,0 , gfn, pfn,ACC_ALL , &r)) +return r; + + r =RET_PF_RETRY ; +read_lock (&vcpu->kvm->mmu_lock); + +if (!is_noslot_pfn (pfn) &&mmu_invalidate_retry_hva (vcpu->kvm, mmu_seq, hva)) + goto out_unlock; + + r =kvm_tdp_mmu_map (vcpu, gpa, error_code, map_writable, max_level, + pfn, prefault); + +out_unlock: +read_unlock (&vcpu->kvm->mmu_lock); +kvm_release_pfn_clean (pfn); +return r; +} +#endif +int kvm_tdp_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code,bool prefault) {
@@ -4116,6 +4154,12 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,break ; } +#ifdef CONFIG_X86_64 +if (is_tdp_mmu (vcpu->arch.mmu)) +return kvm_tdp_mmu_page_fault (vcpu, gpa, error_code, prefault, + max_level); +#endif +return direct_page_fault (vcpu, gpa, error_code, prefault, max_level, true); }
Signature · 1 line
2.53.0
Sasha Levin
maintainerPATCH 6/6 KVM: x86/mmu: Rename __direct_map() to direct_map()
In reply to Sasha LevinFrom: David Matlack <[email protected]>
[ Upstream commit 6c882ef4fc7bd99b67ad152e75428b669281c521 ]
Rename __direct_map() to direct_map() since the leading underscores are unnecessary. This also makes the page fault handler names more
consistent: kvm_tdp_mmu_page_fault() calls kvm_tdp_mmu_map() and direct_page_fault() calls direct_map().
Opportunistically make some trivial cleanups to comments that had to be modified anyway since they mentioned __direct_map(). Specifically, use "()" when referring to functions, and include kvm_tdp_mmu_map() among the various callers of disallowed_hugepage_adjust().
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/x86/kvm/mmu/mmu.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +10 -10
@@ -2991,11 +2991,11 @@ void disallowed_hugepage_adjust(u64 spte, gfn_t gfn, int cur_level, is_shadow_present_pte(spte) && !is_large_pte(spte)) {/* -* A small SPTE exists for this pfn, but FNAME(fetch) -* and __direct_map would like to create a large PTE -* instead: just force them to go down another level, -* patching back for them into pfn the next 9 bits of -* the address. +* A small SPTE exists for this pfn, but FNAME(fetch), +* direct_map(), or kvm_tdp_mmu_map() would like to create a +* large PTE instead: just force them to go down another level, +* patching back for them into pfn the next 9 bits of the +* address. */ u64 page_mask =KVM_PAGES_PER_HPAGE (level) -KVM_PAGES_PER_HPAGE (level -1 );
@@ -3004,9 +3004,9 @@ void disallowed_hugepage_adjust(u64 spte, gfn_t gfn, int cur_level, } } -static int __direct_map (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, -int map_writable,int max_level,kvm_pfn_t pfn, -bool prefault,bool is_tdp) +static int direct_map (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, +int map_writable,int max_level,kvm_pfn_t pfn, +bool prefault,bool is_tdp) {bool nx_huge_page_workaround_enabled =is_nx_huge_page_enabled ();bool write = error_code &PFERR_WRITE_MASK ;
@@ -4037,8 +4037,8 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,if (r) goto out_unlock; - r =__direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, - prefault, is_tdp); + r =direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, + prefault, is_tdp); out_unlock:write_unlock (&vcpu->kvm->mmu_lock);
Signature · 1 line
2.53.0
Re: [PATCH 5.15.y 6/6] KVM: x86/mmu: Rename __direct_map() to direct_map()
Hi Sasha
Quoted 16 lines · Sasha Levin
From: David Matlack <[email protected]>
[ Upstream commit
6c882ef4fc7bd99b67ad152e75428b669281c521]Rename
__direct_map()todirect_map()since the leading underscores are unnecessary. This also makes the page fault handler names more
consistent:kvm_tdp_mmu_page_fault()callskvm_tdp_mmu_map()anddirect_page_fault()callsdirect_map().Opportunistically make some trivial cleanups to comments that had to be modified anyway since they mentioned
__direct_map(). Specifically, use "()" when referring to functions, and includekvm_tdp_mmu_map()among the various callers ofdisallowed_hugepage_adjust().No functional change intended.
Quoted 7 lines
Signed-off-by David Matlack <[email protected]>Reviewed-by Isaku Yamahata <[email protected]>Signed-off-by Paolo Bonzini <[email protected]>Message-Id: <[email protected]>
Signed-off-by Paolo Bonzini <[email protected]>Stable-dep-of:
2abd5287f083("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")Signed-off-by Sasha Levin <[email protected]>
I don't see them in queue-5.15 , FYI.
Also I think along with 6 prerequisites and final 7th patch, we also might need: commit: 9f33697ac7ff ("KVM: x86/mmu: Stop needlessly making MMU pages available for TDP MMU faults") should be a clean cherru-pick.
thanks,
harshit
Quoted 49 lines
--- arch/x86/kvm/mmu/mmu.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)arch/x86/kvm/mmu/mmu.c +10
-10@@ -2991,11 +2991,11 @@ void disallowed_hugepage_adjust(u64 spte, gfn_t gfn, int cur_level, is_shadow_present_pte(spte) && !is_large_pte(spte)) {/* -* A small SPTE exists for this pfn, but FNAME(fetch) -* and __direct_map would like to create a large PTE -* instead: just force them to go down another level, -* patching back for them into pfn the next 9 bits of -* the address. +* A small SPTE exists for this pfn, but FNAME(fetch), +* direct_map(), or kvm_tdp_mmu_map() would like to create a +* large PTE instead: just force them to go down another level, +* patching back for them into pfn the next 9 bits of the +* address. */ u64 page_mask =KVM_PAGES_PER_HPAGE (level) -KVM_PAGES_PER_HPAGE (level -1 );@@ -3004,9 +3004,9 @@ void disallowed_hugepage_adjust(u64 spte, gfn_t gfn, int cur_level, } } -static int __direct_map (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, -int map_writable,int max_level,kvm_pfn_t pfn, -bool prefault,bool is_tdp) +static int direct_map (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, +int map_writable,int max_level,kvm_pfn_t pfn, +bool prefault,bool is_tdp) {bool nx_huge_page_workaround_enabled =is_nx_huge_page_enabled ();bool write = error_code &PFERR_WRITE_MASK ;@@ -4037,8 +4037,8 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,if (r) goto out_unlock; - r =__direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, - prefault, is_tdp); + r =direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, + prefault, is_tdp); out_unlock:write_unlock (&vcpu->kvm->mmu_lock);
Sasha Levin
maintainerRe: [PATCH 5.15.y 6/6] KVM: x86/mmu: Rename __direct_map() to direct_map()
Quoted 27 lines · Harshit Mogalapalli
Hi Sasha
Quoted 16 lines · Sasha Levin
From: David Matlack <[email protected]>
[ Upstream commit
6c882ef4fc7bd99b67ad152e75428b669281c521]Rename
__direct_map()todirect_map()since the leading underscores are unnecessary. This also makes the page fault handler names more
consistent:kvm_tdp_mmu_page_fault()callskvm_tdp_mmu_map()anddirect_page_fault()callsdirect_map().Opportunistically make some trivial cleanups to comments that had to be modified anyway since they mentioned
__direct_map(). Specifically, use "()" when referring to functions, and includekvm_tdp_mmu_map()among the various callers ofdisallowed_hugepage_adjust().No functional change intended.
Quoted 7 lines
Signed-off-by David Matlack <[email protected]>Reviewed-by Isaku Yamahata <[email protected]>Signed-off-by Paolo Bonzini <[email protected]>Message-Id: <[email protected]>
Signed-off-by Paolo Bonzini <[email protected]>Stable-dep-of:
2abd5287f083("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")Signed-off-by Sasha Levin <[email protected]>
Quoted 1 line
I don't see them in queue-5.15 , FYI.
I don't think Greg got to these yet. We normally let these sit on the mailing list for a bit, for review.
Quoted 1 line
Also I think along with 6 prerequisites and final 7th patch, we also might need: commit:
9f33697ac7ff("KVM: x86/mmu: Stop needlessly making MMU pages available for TDP MMU faults") should be a clean cherru-pick.
Possibly, but there's no fixes or stable tag on it, so without an ack from the KVM folks it won't go in.
Signature · 2 lines
Thanks, Sasha
Re: [PATCH 5.15.y 6/6] KVM: x86/mmu: Rename __direct_map() to direct_map()
In reply to Sasha LevinHi Sasha,
Quoted 14 lines · Sasha Levin
From: David Matlack <[email protected]>
[ Upstream commit
6c882ef4fc7bd99b67ad152e75428b669281c521]Rename
__direct_map()todirect_map()since the leading underscores are unnecessary. This also makes the page fault handler names more
consistent:kvm_tdp_mmu_page_fault()callskvm_tdp_mmu_map()anddirect_page_fault()callsdirect_map().Opportunistically make some trivial cleanups to comments that had to be modified anyway since they mentioned
__direct_map(). Specifically, use "()" when referring to functions, and includekvm_tdp_mmu_map()among the various callers ofdisallowed_hugepage_adjust().
Quoted 8 lines
No functional change intended.
Signed-off-by David Matlack <[email protected]>Reviewed-by Isaku Yamahata <[email protected]>Signed-off-by Paolo Bonzini <[email protected]>Message-Id: <[email protected]>
Signed-off-by Paolo Bonzini <[email protected]>Stable-dep-of:
2abd5287f083("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
Maybe I'm missing something, but commit 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available") cannot be backported as-is. It depends on is_page_fault_stale(), introduced by commit a955cad84cda ("KVM: x86/mmu: Retry page fault if root is invalidated by memslot update"), which is missing in 5.15.y.
Furthermore, a955cad84cda is required to fix a regression in 5.15.y when applying the CVE-2026-46113 and CVE-2026-53359 fixes, but its backport remains pending [1][2].
I plan to send a manual backport of a955cad84cda and 2abd5287f083. In doing so, the interface of is_page_fault_stale() will end up incompatible with mainline. While not ideal, making the interface compatible requires struct kvm_page_fault and its 50+ prerequisite commits.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/CABgObfaKqT0gMUB_XRP5DW5HmpgaoTgmD-A8_zQVG2Vp5xZE4g@mail.gmail.com/
Thanks,
akagi
Quoted 55 lines
Signed-off-by Sasha Levin <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)arch/x86/kvm/mmu/mmu.c +10
-10@@ -2991,11 +2991,11 @@ void disallowed_hugepage_adjust(u64 spte, gfn_t gfn, int cur_level, is_shadow_present_pte(spte) && !is_large_pte(spte)) {/* -* A small SPTE exists for this pfn, but FNAME(fetch) -* and __direct_map would like to create a large PTE -* instead: just force them to go down another level, -* patching back for them into pfn the next 9 bits of -* the address. +* A small SPTE exists for this pfn, but FNAME(fetch), +* direct_map(), or kvm_tdp_mmu_map() would like to create a +* large PTE instead: just force them to go down another level, +* patching back for them into pfn the next 9 bits of the +* address. */ u64 page_mask =KVM_PAGES_PER_HPAGE (level) -KVM_PAGES_PER_HPAGE (level -1 );@@ -3004,9 +3004,9 @@ void disallowed_hugepage_adjust(u64 spte, gfn_t gfn, int cur_level, } } -static int __direct_map (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, -int map_writable,int max_level,kvm_pfn_t pfn, -bool prefault,bool is_tdp) +static int direct_map (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, +int map_writable,int max_level,kvm_pfn_t pfn, +bool prefault,bool is_tdp) {bool nx_huge_page_workaround_enabled =is_nx_huge_page_enabled ();bool write = error_code &PFERR_WRITE_MASK ;@@ -4037,8 +4037,8 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,if (r) goto out_unlock; - r =__direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, - prefault, is_tdp); + r =direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, + prefault, is_tdp); out_unlock:write_unlock (&vcpu->kvm->mmu_lock);Signature · 1 line
2.53.0
Sasha Levin
maintainerRe: [PATCH 5.15.y 6/6] KVM: x86/mmu: Rename __direct_map() to direct_map()
Quoted 5 lines · Kenta Akagi
Maybe I'm missing something, but commit
2abd5287f083("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available") cannot be backported as-is. It depends onis_page_fault_stale(), introduced by commita955cad84cda("KVM: x86/mmu: Retry page fault if root is invalidated by memslot update"), which is missing in 5.15.y.
You're not missing anything - you're right.
I've dropped all six from the 5.15 queue.
Quoted 1 line
I plan to send a manual backport of
a955cad84cdaand2abd5287f083.
Please do, and thanks for offering.
Signature · 2 lines
Thanks, Sasha
v2 cover backport for CVE-2026-64561
In reply to [email protected]This is v2 of the backport to fix CVE-2026-64561, v1 is Sasha's series [1].
Patches 2-7 in this series correspond to patches 1-6 in v1; Patches 1 and 8 are new in v2.
Changes in v2:
- Added Patch 1 to backport is_page_fault_stale() as a prerequisite for Patch 8 and for the CVE-2026-46113 and CVE-2026-53359 fixes, which will be sent in a separate series [2].
- Added Patch 8 as the fix for CVE-2026-64561.
- Adjusted Patch 5 to rename mmu_notifier_retry_hva() inside is_page_fault_stale().
- Adjusted Patch 6 to use is_page_fault_stale() in kvm_tdp_mmu_page_fault().
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/CABgObfaKqT0gMUB_XRP5DW5HmpgaoTgmD-A8_zQVG2Vp5xZE4g@mail.gmail.com/
Chao Peng (1):
KVM: Rename mmu_notifier_* to mmu_invalidate_*
David Matlack (2):
KVM: x86/mmu: Split out TDP MMU page fault handling
KVM: x86/mmu: Rename __direct_map() to direct_map()
Sean Christopherson (5):
KVM: x86/mmu: Retry page fault if root is invalidated by memslot
update
KVM: x86/mmu: Directly "destroy" PTE list when recycling rmaps
KVM: x86/mmu: Rename pte_list_{destroy,remove}() to show they zap
SPTEs
KVM: x86/mmu: Document the "rules" for using host_pfn_mapping_level()
KVM: x86: Check for invalid/obsolete root *after* making MMU pages
available
arch/arm64/kvm/mmu.c | 10 +- arch/mips/kvm/mmu.c | 12 +- arch/powerpc/include/asm/kvm_book3s_64.h | 2 +- arch/powerpc/kvm/book3s_64_mmu_host.c | 4 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 6 +- arch/powerpc/kvm/book3s_64_vio_hv.c | 2 +- arch/powerpc/kvm/book3s_hv_nested.c | 2 +- arch/powerpc/kvm/book3s_hv_rm_mmu.c | 8 +- arch/powerpc/kvm/e500_mmu_host.c | 4 +- arch/x86/kvm/mmu/mmu.c | 175 +++++++++++++++++------ arch/x86/kvm/mmu/paging_tmpl.h | 13 +- include/linux/kvm_host.h | 60 ++++---- virt/kvm/kvm_main.c | 49 +++---- 14 files changed, 223 insertions(+), 128 deletions(-)
Signature · 1 line
2.51.2
PATCH v2 1/8 KVM: x86/mmu: Retry page fault if root is invalidated by memslot update
From: Sean Christopherson <[email protected]>
[ Upstream commit a955cad84cdaffa282b3cf8f5ce69e9e5655e585 ]
Bail from the page fault handler if the root shadow page was obsoleted by a memslot update. Do the check _after_ acuiring mmu_lock, as the TDP MMU doesn't rely on the memslot/MMU generation, and instead relies on the root being explicit marked invalid by kvm_mmu_zap_all_fast(), which takes mmu_lock for write.
For the TDP MMU, inserting a SPTE into an obsolete root can leak a SP if kvm_tdp_mmu_zap_invalidated_roots() has already zapped the SP, i.e. has moved past the gfn associated with the SP.
For other MMUs, the resulting behavior is far more convoluted, though unlikely to be truly problematic. Installing SPs/SPTEs into the obsolete root isn't directly problematic, as the obsolete root will be unloaded and dropped before the vCPU re-enters the guest. But because the legacy MMU tracks shadow pages by their role, any SP created by the fault can can be reused in the new post-reload root. Again, that _shouldn't_ be problematic as any leaf child SPTEs will be created for the current/valid memslot generation, and kvm_mmu_get_page() will not reuse child SPs from the old generation as they will be flagged as obsolete. But, given that continuing with the fault is pointess (the root will be unloaded), apply the check to all MMUs.
Message-Id: <[email protected]>
[backport note: is_page_fault_stale() adapted to take individual parameters
instead of struct kvm_page_fault, since 4326e57ef40a ("KVM: MMU: change direct_page_fault() arguments to kvm_page_fault") is not in 5.15.y; required by the following backport of 2abd5287f083 and by 0cb2af2ea66a to be backported separately]
--- arch/x86/kvm/mmu/mmu.c | 22 ++++++++++++++++++++-- arch/x86/kvm/mmu/paging_tmpl.h | 3 ++- 2 files changed, 22 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +20 -2
@@ -1942,7 +1942,11 @@ static void mmu_audit_disable(void) { }static bool is_obsolete_sp (struct kvm *kvm,struct kvm_mmu_page *sp) { -return sp->role.invalid || +if (sp->role.invalid) +return true; + +/* TDP MMU pages due not use the MMU generation. */ +return !sp->tdp_mmu_page &&unlikely (sp->mmu_valid_gen != kvm->arch.mmu_valid_gen); }
@@ -3968,6 +3972,20 @@ static bool kvm_faultin_pfn(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn,return true; } +/* +* Returns true if the page fault is stale and needs to be retried, i.e. if the +* root was invalidated by a memslot update or a relevant mmu_notifier fired. +*/ +static bool is_page_fault_stale (struct kvm_vcpu *vcpu, +kvm_pfn_t pfn,unsigned long mmu_seq,hva_t hva) +{ +if (is_obsolete_sp (vcpu->kvm,to_shadow_page (vcpu->arch.mmu->root_hpa))) +return true; + +return !is_noslot_pfn (pfn) && +mmu_notifier_retry_hva (vcpu->kvm, mmu_seq, hva); +} +static int direct_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code,bool prefault,int max_level,bool is_tdp) {
@@ -4009,7 +4027,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,else write_lock (&vcpu->kvm->mmu_lock); -if (!is_noslot_pfn (pfn) &&mmu_notifier_retry_hva (vcpu->kvm, mmu_seq, hva)) +if (is_page_fault_stale (vcpu, pfn, mmu_seq, hva)) goto out_unlock;if (is_tdp_mmu_fault) {
arch/x86/kvm/mmu/paging_tmpl.h +2 -1
@@ -925,7 +925,8 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gpa_t addr, u32 error_code, r =RET_PF_RETRY ;write_lock (&vcpu->kvm->mmu_lock); -if (!is_noslot_pfn (pfn) &&mmu_notifier_retry_hva (vcpu->kvm, mmu_seq, hva)) + +if (is_page_fault_stale (vcpu, pfn, mmu_seq, hva)) goto out_unlock;kvm_mmu_audit (vcpu,AUDIT_PRE_PAGE_FAULT );
Signature · 1 line
2.51.2
PATCH v2 2/8 KVM: x86/mmu: Directly "destroy" PTE list when recycling rmaps
In reply to Kenta AkagiFrom: Sean Christopherson <[email protected]>
[ Upstream commit a42989e7fbb0186d9fee05b29e0ea9cb639d0bd3 ]
Use pte_list_destroy() directly when recycling rmaps instead of bouncing through kvm_unmap_rmapp() and kvm_zap_rmapp(). Calling kvm_unmap_rmapp() is unnecessary and odd as it requires passing dummy parameters; passing NULL for @slot when __rmap_add() already has a valid slot is especially weird and confusing.
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/x86/kvm/mmu/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -1639,7 +1639,7 @@ static void rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) rmap_count =pte_list_add (vcpu, spte, rmap_head);if (rmap_count >RMAP_RECYCLE_THRESHOLD ) { -kvm_unmap_rmapp (vcpu->kvm, rmap_head,NULL , gfn, sp->role.level,__pte (0 )); +pte_list_destroy (vcpu->kvm, rmap_head);kvm_flush_remote_tlbs_with_address ( vcpu->kvm, sp->gfn,KVM_PAGES_PER_HPAGE (sp->role.level)); }
Signature · 1 line
2.51.2
PATCH v2 3/8 KVM: x86/mmu: Rename pte_list_{destroy,remove}() to show they zap SPTEs
In reply to Kenta AkagiFrom: Sean Christopherson <[email protected]>
[ Upstream commit 9202aee816c84d69179f94193c5dd321bb0e8530 ]
Rename pte_list_remove() and pte_list_destroy() to kvm_zap_one_rmap_spte() and kvm_zap_all_rmap_sptes() respectively to document that (a) they zap SPTEs and (b) to better document how they differ (remove vs. destroy does not exactly scream "one vs. all").
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/x86/kvm/mmu/mmu.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +9 -8
@@ -999,15 +999,16 @@ static void __pte_list_remove(u64 *spte, struct kvm_rmap_head *rmap_head) } } -static void pte_list_remove (struct kvm *kvm,struct kvm_rmap_head *rmap_head, -u64 *sptep) +static void kvm_zap_one_rmap_spte (struct kvm *kvm, +struct kvm_rmap_head *rmap_head,u64 *sptep) {mmu_spte_clear_track_bits (kvm, sptep);__pte_list_remove (sptep, rmap_head); } -/* Return true if rmap existed, false otherwise */ -static bool pte_list_destroy (struct kvm *kvm,struct kvm_rmap_head *rmap_head) +/* Return true if at least one SPTE was zapped, false otherwise */ +static bool kvm_zap_all_rmap_sptes (struct kvm *kvm, +struct kvm_rmap_head *rmap_head) {struct pte_list_desc *desc, *next;int i;
@@ -1434,7 +1435,7 @@ static bool rmap_write_protect(struct kvm_vcpu *vcpu, u64 gfn)static bool kvm_zap_rmapp (struct kvm *kvm,struct kvm_rmap_head *rmap_head,const struct kvm_memory_slot *slot) { -return pte_list_destroy (kvm, rmap_head); +return kvm_zap_all_rmap_sptes (kvm, rmap_head); } static bool kvm_unmap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
@@ -1465,7 +1466,7 @@ static bool kvm_set_pte_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, need_flush =1 ;if (pte_write (pte)) { -pte_list_remove (kvm, rmap_head, sptep); +kvm_zap_one_rmap_spte (kvm, rmap_head, sptep); goto restart; }else { new_spte = kvm_mmu_changed_pte_notifier_make_spte(
@@ -1639,7 +1640,7 @@ static void rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) rmap_count =pte_list_add (vcpu, spte, rmap_head);if (rmap_count >RMAP_RECYCLE_THRESHOLD ) { -pte_list_destroy (vcpu->kvm, rmap_head); +kvm_zap_all_rmap_sptes (vcpu->kvm, rmap_head);kvm_flush_remote_tlbs_with_address ( vcpu->kvm, sp->gfn,KVM_PAGES_PER_HPAGE (sp->role.level)); }
@@ -5873,7 +5874,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, !kvm_is_reserved_pfn(pfn) && sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn, pfn, PG_LEVEL_NUM)) { -pte_list_remove (kvm, rmap_head, sptep); +kvm_zap_one_rmap_spte (kvm, rmap_head, sptep);if (kvm_available_flush_tlb_with_range ()) kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
Signature · 1 line
2.51.2
PATCH v2 4/8 KVM: x86/mmu: Document the "rules" for using host_pfn_mapping_level()
In reply to Kenta AkagiFrom: Sean Christopherson <[email protected]>
[ Upstream commit 65e3b446bcceaac7448cb25a2a5bf4adbcf25fe6 ]
Add a comment to document how host_pfn_mapping_level() can be used safely, as the line between safe and dangerous is quite thin. E.g. if KVM were to ever support in-place promotion to create huge pages, consuming the level is safe if the caller holds mmu_lock and checks that there's an existing _leaf_ SPTE, but unsafe if the caller only checks that there's a non-leaf SPTE.
Opportunistically tweak the existing comments to explicitly document why KVM needs to use READ_ONCE().
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/x86/kvm/mmu/mmu.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
arch/x86/kvm/mmu/mmu.c +25 -0
@@ -2873,6 +2873,31 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)__direct_pte_prefetch (vcpu, sp, sptep); } +/* +* Lookup the mapping level for @gfn in the current mm. +* +* WARNING! Use of host_pfn_mapping_level() requires the caller and the end +* consumer to be tied into KVM's handlers for MMU notifier events! +* +* There are several ways to safely use this helper: +* +* - Check mmu_notifier_retry_hva() after grabbing the mapping level, before +* consuming it. In this case, mmu_lock doesn't need to be held during the +* lookup, but it does need to be held while checking the MMU notifier. +* +* - Hold mmu_lock AND ensure there is no in-progress MMU notifier invalidation +* event for the hva. This can be done by explicit checking the MMU notifier +* or by ensuring that KVM already has a valid mapping that covers the hva. +* +* - Do not use the result to install new mappings, e.g. use the host mapping +* level only to decide whether or not to zap an entry. In this case, it's +* not required to hold mmu_lock (though it's highly likely the caller will +* want to hold mmu_lock anyways, e.g. to modify SPTEs). +* +* Note! The lookup can still race with modifications to host page tables, but +* the above "rules" ensure KVM will not _consume_ the result of the walk if a +* race with the primary MMU occurs. +*/ static int host_pfn_mapping_level (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn,const struct kvm_memory_slot *slot) {
Signature · 1 line
2.51.2
PATCH v2 5/8 KVM: Rename mmu_notifier_* to mmu_invalidate_*
In reply to Kenta AkagiFrom: Chao Peng <[email protected]>
[ Upstream commit 20ec3ebd707c77fb9b11b37193449193d4649f33 ]
The motivation of this renaming is to make these variables and related helper functions less mmu_notifier bound and can also be used for non mmu_notifier based page invalidation. mmu_invalidate_* was chosen to better describe the purpose of 'invalidating' a page that those variables are used for.
- mmu_notifier_seq/range_start/range_end are renamed to mmu_invalidate_seq/range_start/range_end.
- mmu_notifier_retry{_hva} helper functions are renamed to mmu_invalidate_retry{_hva}.
- mmu_notifier_count is renamed to mmu_invalidate_in_progress to avoid confusion with mn_active_invalidate_count.
- While here, also update kvm_inc/dec_notifier_count() to kvm_mmu_invalidate_begin/end() to match the change for mmu_notifier_count.
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/arm64/kvm/mmu.c | 10 ++-- arch/mips/kvm/mmu.c | 12 ++--- arch/powerpc/include/asm/kvm_book3s_64.h | 2 +- arch/powerpc/kvm/book3s_64_mmu_host.c | 4 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 6 +-- arch/powerpc/kvm/book3s_64_vio_hv.c | 2 +- arch/powerpc/kvm/book3s_hv_nested.c | 2 +- arch/powerpc/kvm/book3s_hv_rm_mmu.c | 8 ++-- arch/powerpc/kvm/e500_mmu_host.c | 4 +- arch/x86/kvm/mmu/mmu.c | 14 +++--- arch/x86/kvm/mmu/paging_tmpl.h | 4 +- include/linux/kvm_host.h | 60 ++++++++++++------------ virt/kvm/kvm_main.c | 49 +++++++++---------- 14 files changed, 92 insertions(+), 89 deletions(-)
arch/arm64/kvm/mmu.c +5 -5
@@ -881,7 +881,7 @@ transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot,* THP doesn't start to split while we are adjusting the * refcounts. * -* We are sure this doesn't happen, because mmu_notifier_retry +* We are sure this doesn't happen, because mmu_invalidate_retry * was successful and we are holding the mmu_lock, so if this * THP is trying to split, it will be blocked in the mmu * notifier before touching any of the pages, specifically
@@ -1074,14 +1074,14 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, gfn = fault_ipa >>PAGE_SHIFT ;/* -* Read mmu_notifier_seq so that KVM can detect if the results of +* Read mmu_invalidate_seq so that KVM can detect if the results of * vma_lookup() or __gfn_to_pfn_memslot() become stale prior to * acquiring kvm->mmu_lock. * * Rely on mmap_read_unlock() for an implicit smp_rmb(), which pairs -* with the smp_wmb() in kvm_dec_notifier_count(). +* with the smp_wmb() in kvm_mmu_invalidate_end(). */ - mmu_seq = vcpu->kvm->mmu_notifier_seq; + mmu_seq = vcpu->kvm->mmu_invalidate_seq;mmap_read_unlock (current->mm); pfn = __gfn_to_pfn_memslot(memslot, gfn, false, NULL,
@@ -1118,7 +1118,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,spin_lock (&kvm->mmu_lock); pgt = vcpu->arch.hw_mmu->pgt; -if (mmu_notifier_retry (kvm, mmu_seq)) +if (mmu_invalidate_retry (kvm, mmu_seq)) goto out_unlock;/*
arch/mips/kvm/mmu.c +6 -6
@@ -615,17 +615,17 @@ static int kvm_mips_map_page(struct kvm_vcpu *vcpu, unsigned long gpa,* Used to check for invalidations in progress, of the pfn that is * returned by pfn_to_pfn_prot below. */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;/* -* Ensure the read of mmu_notifier_seq isn't reordered with PTE reads in -* gfn_to_pfn_prot() (which calls get_user_pages()), so that we don't +* Ensure the read of mmu_invalidate_seq isn't reordered with PTE reads +* in gfn_to_pfn_prot() (which calls get_user_pages()), so that we don't * risk the page we get a reference to getting unmapped before we have a -* chance to grab the mmu_lock without mmu_notifier_retry() noticing. +* chance to grab the mmu_lock without mmu_invalidate_retry() noticing. * * This smp_rmb() pairs with the effective smp_wmb() of the combination * of the pte_unmap_unlock() after the PTE is zapped, and the * spin_lock() in kvm_mmu_notifier_invalidate_<page|range_end>() before -* mmu_notifier_seq is incremented. +* mmu_invalidate_seq is incremented. */ smp_rmb ();
@@ -638,7 +638,7 @@ static int kvm_mips_map_page(struct kvm_vcpu *vcpu, unsigned long gpa,spin_lock (&kvm->mmu_lock);/* Check if an invalidation has taken place since we got pfn */ -if (mmu_notifier_retry (kvm, mmu_seq)) { +if (mmu_invalidate_retry (kvm, mmu_seq)) {/* * This can happen when mappings are changed asynchronously, but * also synchronously if a COW is triggered by
arch/powerpc/include/asm/kvm_book3s_64.h +1 -1
@@ -673,7 +673,7 @@ static inline pte_t *find_kvm_host_pte(struct kvm *kvm, unsigned long mmu_seq,VM_WARN (!spin_is_locked (&kvm->mmu_lock),"%s called with kvm mmu_lock not held \n" , __func__); -if (mmu_notifier_retry (kvm, mmu_seq)) +if (mmu_invalidate_retry (kvm, mmu_seq))return NULL ; pte =__find_linux_pte (kvm->mm->pgd, ea,NULL , hshift);
arch/powerpc/kvm/book3s_64_mmu_host.c +2 -2
@@ -90,7 +90,7 @@ int kvmppc_mmu_map_page(struct kvm_vcpu *vcpu, struct kvmppc_pte *orig_pte,unsigned long pfn;/* used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();/* Get host physical address for gpa */
@@ -151,7 +151,7 @@ int kvmppc_mmu_map_page(struct kvm_vcpu *vcpu, struct kvmppc_pte *orig_pte, cpte =kvmppc_mmu_hpte_cache_next (vcpu);spin_lock (&kvm->mmu_lock); -if (!cpte ||mmu_notifier_retry (kvm, mmu_seq)) { +if (!cpte ||mmu_invalidate_retry (kvm, mmu_seq)) { r = -EAGAIN ; goto out_unlock; }
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -2
@@ -570,7 +570,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_vcpu *vcpu,return -EFAULT ;/* used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb (); ret = -EFAULT ;
@@ -685,7 +685,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_vcpu *vcpu,/* Check if we might have been invalidated; let the guest retry if so */ ret =RESUME_GUEST ; -if (mmu_notifier_retry (vcpu->kvm, mmu_seq)) { +if (mmu_invalidate_retry (vcpu->kvm, mmu_seq)) {unlock_rmap (rmap); goto out_unlock; }
arch/powerpc/kvm/book3s_64_mmu_radix.c +3 -3
@@ -634,7 +634,7 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Check if we might have been invalidated; let the guest retry if so */ spin_lock (&kvm->mmu_lock); ret = -EAGAIN ; -if (mmu_notifier_retry (kvm, mmu_seq)) +if (mmu_invalidate_retry (kvm, mmu_seq)) goto out_unlock;/* Now traverse again under the lock and change the tree */
@@ -824,7 +824,7 @@ int kvmppc_book3s_instantiate_page(struct kvm_vcpu *vcpu,bool large_enable;/* used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();/*
@@ -1185,7 +1185,7 @@ void kvmppc_radix_flush_memslot(struct kvm *kvm,* Increase the mmu notifier sequence number to prevent any page * fault that read the memslot earlier from writing a PTE. */ - kvm->mmu_notifier_seq++; + kvm->mmu_invalidate_seq++;spin_unlock (&kvm->mmu_lock); }
arch/powerpc/kvm/book3s_64_vio_hv.c +1 -1
@@ -499,7 +499,7 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,/* * used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb (); stt =kvmppc_find_table (vcpu->kvm, liobn);
arch/powerpc/kvm/book3s_hv_nested.c +1 -1
@@ -1574,7 +1574,7 @@ static long int __kvmhv_nested_page_fault(struct kvm_vcpu *vcpu,/* 2. Find the host pte for this L1 guest real address */ /* Used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();/* See if can find translation in our partition scoped tables for L1 */
arch/powerpc/kvm/book3s_hv_rm_mmu.c +4 -4
@@ -216,7 +216,7 @@ long kvmppc_do_h_enter(struct kvm *kvm, unsigned long flags, g_ptel = ptel;/* used later to detect if we might have been invalidated */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();/* Find the memslot (if any) for this address */
@@ -363,7 +363,7 @@ long kvmppc_do_h_enter(struct kvm *kvm, unsigned long flags, rmap =real_vmalloc_addr (rmap);lock_rmap (rmap);/* Check for pending invalidations under the rmap chain lock */ -if (mmu_notifier_retry (kvm, mmu_seq)) { +if (mmu_invalidate_retry (kvm, mmu_seq)) {/* inval in progress, write a non-present HPTE */ pteh |=HPTE_V_ABSENT ; pteh &= ~HPTE_V_VALID ;
@@ -929,7 +929,7 @@ static long kvmppc_do_h_page_init_zero(struct kvm_vcpu *vcpu,int i;/* Used later to detect if we might have been invalidated */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();arch_spin_lock (&kvm->mmu_lock.rlock.raw_lock);
@@ -957,7 +957,7 @@ static long kvmppc_do_h_page_init_copy(struct kvm_vcpu *vcpu,long ret =H_SUCCESS ;/* Used later to detect if we might have been invalidated */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();arch_spin_lock (&kvm->mmu_lock.rlock.raw_lock);
arch/powerpc/kvm/e500_mmu_host.c +2 -2
@@ -339,7 +339,7 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500,unsigned long flags;/* used to check for invalidations in progress */ - mmu_seq = kvm->mmu_notifier_seq; + mmu_seq = kvm->mmu_invalidate_seq;smp_rmb ();/*
@@ -460,7 +460,7 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500, }spin_lock (&kvm->mmu_lock); -if (mmu_notifier_retry (kvm, mmu_seq)) { +if (mmu_invalidate_retry (kvm, mmu_seq)) { ret = -EAGAIN ; goto out; }
arch/x86/kvm/mmu/mmu.c +7 -7
@@ -2867,7 +2867,7 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)* If addresses are being invalidated, skip prefetching to avoid * accidentally prefetching those addresses. */ -if (unlikely (vcpu->kvm->mmu_notifier_count)) +if (unlikely (vcpu->kvm->mmu_invalidate_in_progress))return ;__direct_pte_prefetch (vcpu, sp, sptep);
@@ -2881,7 +2881,7 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)* * There are several ways to safely use this helper: * -* - Check mmu_notifier_retry_hva() after grabbing the mapping level, before +* - Check mmu_invalidate_retry_hva() after grabbing the mapping level, before * consuming it. In this case, mmu_lock doesn't need to be held during the * lookup, but it does need to be held while checking the MMU notifier. *
@@ -2976,7 +2976,7 @@ int kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, gfn_t gfn,return PG_LEVEL_4K ;/* -* mmu_notifier_retry() was successful and mmu_lock is held, so +* mmu_invalidate_retry() was successful and mmu_lock is held, so * the pmd can't be split from under us. */ mask =KVM_PAGES_PER_HPAGE (level) -1 ;
@@ -4009,7 +4009,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,return true;return !is_noslot_pfn (pfn) && -mmu_notifier_retry_hva (vcpu->kvm, mmu_seq, hva); +mmu_invalidate_retry_hva (vcpu->kvm, mmu_seq, hva); } static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
@@ -4036,7 +4036,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,if (r)return r; - mmu_seq = vcpu->kvm->mmu_notifier_seq; + mmu_seq = vcpu->kvm->mmu_invalidate_seq;smp_rmb (); if (kvm_faultin_pfn(vcpu, prefault, gfn, gpa, &pfn, &hva,
@@ -5785,7 +5785,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)write_lock (&kvm->mmu_lock); -kvm_inc_notifier_count (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_begin (kvm, gfn_start, gfn_end);if (kvm_memslots_have_rmaps (kvm)) {for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) {
@@ -5820,7 +5820,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)kvm_flush_remote_tlbs_with_address (kvm, gfn_start, gfn_end - gfn_start); -kvm_dec_notifier_count (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_end (kvm, gfn_start, gfn_end);write_unlock (&kvm->mmu_lock); }
arch/x86/kvm/mmu/paging_tmpl.h +2 -2
@@ -634,7 +634,7 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,* If addresses are being invalidated, skip prefetching to avoid * accidentally prefetching those addresses. */ -if (unlikely (vcpu->kvm->mmu_notifier_count)) +if (unlikely (vcpu->kvm->mmu_invalidate_in_progress))return ;if (sp->role.direct)
@@ -894,7 +894,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gpa_t addr, u32 error_code,else max_level = walker.level; - mmu_seq = vcpu->kvm->mmu_notifier_seq; + mmu_seq = vcpu->kvm->mmu_invalidate_seq;smp_rmb (); if (kvm_faultin_pfn(vcpu, prefault, walker.gfn, addr, &pfn, &hva,
include/linux/kvm_host.h +31 -29
93 lines not on this page · raw message
virt/kvm/kvm_main.c +25 -24
115 lines not on this page · raw message
Signature · 1 line
2.51.2
PATCH v2 6/8 KVM: x86/mmu: Split out TDP MMU page fault handling
In reply to Kenta AkagiFrom: David Matlack <[email protected]>
[ Upstream commit 9aa8ab43b38146029de807a8ff2696f51e15b226 ]
Split out the page fault handling for the TDP MMU to a separate function. This creates some duplicate code, but makes the TDP MMU fault handler simpler to read by eliminating branches and will enable future cleanups by allowing the TDP MMU and non-TDP MMU fault paths to diverge.
Only compile in the TDP MMU fault handler for 64-bit builds since kvm_tdp_mmu_map() does not exist in 32-bit builds.
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/x86/kvm/mmu/mmu.c | 84 ++++++++++++++++++++++++++++++++---------- 1 file changed, 64 insertions(+), 20 deletions(-)
arch/x86/kvm/mmu/mmu.c +64 -20
@@ -4015,7 +4015,6 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,static int direct_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code,bool prefault,int max_level,bool is_tdp) { -bool is_tdp_mmu_fault =is_tdp_mmu (vcpu->arch.mmu);bool write = error_code &PFERR_WRITE_MASK ;bool map_writable;
@@ -4047,31 +4046,20 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,return r; r =RET_PF_RETRY ; - -if (is_tdp_mmu_fault) -read_lock (&vcpu->kvm->mmu_lock); -else -write_lock (&vcpu->kvm->mmu_lock); +write_lock (&vcpu->kvm->mmu_lock);if (is_page_fault_stale (vcpu, pfn, mmu_seq, hva)) goto out_unlock; -if (is_tdp_mmu_fault) { - r =kvm_tdp_mmu_map (vcpu, gpa, error_code, map_writable, max_level, - pfn, prefault); - }else { - r =make_mmu_pages_available (vcpu); -if (r) - goto out_unlock; - r =__direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, - prefault, is_tdp); - } + r =make_mmu_pages_available (vcpu); +if (r) + goto out_unlock; + + r =__direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, + prefault, is_tdp); out_unlock: -if (is_tdp_mmu_fault) -read_unlock (&vcpu->kvm->mmu_lock); -else -write_unlock (&vcpu->kvm->mmu_lock); +write_unlock (&vcpu->kvm->mmu_lock);kvm_release_pfn_clean (pfn);return r; }
@@ -4119,6 +4107,56 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, }EXPORT_SYMBOL_GPL (kvm_handle_page_fault); +#ifdef CONFIG_X86_64 +static int kvm_tdp_mmu_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa, +u32 error_code,bool prefault,int max_level) +{ +bool write = error_code &PFERR_WRITE_MASK ; +bool map_writable; + +gfn_t gfn = gpa >>PAGE_SHIFT ; +unsigned long mmu_seq; +kvm_pfn_t pfn; +hva_t hva; +int r; + +if (page_fault_handle_page_track (vcpu, error_code, gfn)) +return RET_PF_EMULATE ; + + r =fast_page_fault (vcpu, gpa, error_code); +if (r !=RET_PF_INVALID ) +return r; + + r =mmu_topup_memory_caches (vcpu, false); +if (r) +return r; + + mmu_seq = vcpu->kvm->mmu_invalidate_seq; +smp_rmb (); + +if (kvm_faultin_pfn (vcpu, prefault, gfn, gpa, &pfn, &hva, + write, &map_writable, &r)) +return r; + +if (handle_abnormal_pfn (vcpu,0 , gfn, pfn,ACC_ALL , &r)) +return r; + + r =RET_PF_RETRY ; +read_lock (&vcpu->kvm->mmu_lock); + +if (is_page_fault_stale (vcpu, pfn, mmu_seq, hva)) + goto out_unlock; + + r =kvm_tdp_mmu_map (vcpu, gpa, error_code, map_writable, max_level, + pfn, prefault); + +out_unlock: +read_unlock (&vcpu->kvm->mmu_lock); +kvm_release_pfn_clean (pfn); +return r; +} +#endif +int kvm_tdp_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code,bool prefault) {
@@ -4134,6 +4172,12 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,break ; } +#ifdef CONFIG_X86_64 +if (is_tdp_mmu (vcpu->arch.mmu)) +return kvm_tdp_mmu_page_fault (vcpu, gpa, error_code, prefault, + max_level); +#endif +return direct_page_fault (vcpu, gpa, error_code, prefault, max_level, true); }
Signature · 1 line
2.51.2
PATCH v2 7/8 KVM: x86/mmu: Rename __direct_map() to direct_map()
In reply to Kenta AkagiFrom: David Matlack <[email protected]>
[ Upstream commit 6c882ef4fc7bd99b67ad152e75428b669281c521 ]
Rename __direct_map() to direct_map() since the leading underscores are unnecessary. This also makes the page fault handler names more
consistent: kvm_tdp_mmu_page_fault() calls kvm_tdp_mmu_map() and direct_page_fault() calls direct_map().
Opportunistically make some trivial cleanups to comments that had to be modified anyway since they mentioned __direct_map(). Specifically, use "()" when referring to functions, and include kvm_tdp_mmu_map() among the various callers of disallowed_hugepage_adjust().
No functional change intended.
Message-Id: <[email protected]>
Stable-dep-of: 2abd5287f083 ("KVM: x86: Check for invalid/obsolete root *after* making MMU pages available")
--- arch/x86/kvm/mmu/mmu.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +10 -10
@@ -2995,11 +2995,11 @@ void disallowed_hugepage_adjust(u64 spte, gfn_t gfn, int cur_level, is_shadow_present_pte(spte) && !is_large_pte(spte)) {/* -* A small SPTE exists for this pfn, but FNAME(fetch) -* and __direct_map would like to create a large PTE -* instead: just force them to go down another level, -* patching back for them into pfn the next 9 bits of -* the address. +* A small SPTE exists for this pfn, but FNAME(fetch), +* direct_map(), or kvm_tdp_mmu_map() would like to create a +* large PTE instead: just force them to go down another level, +* patching back for them into pfn the next 9 bits of the +* address. */ u64 page_mask =KVM_PAGES_PER_HPAGE (level) -KVM_PAGES_PER_HPAGE (level -1 );
@@ -3008,9 +3008,9 @@ void disallowed_hugepage_adjust(u64 spte, gfn_t gfn, int cur_level, } } -static int __direct_map (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, -int map_writable,int max_level,kvm_pfn_t pfn, -bool prefault,bool is_tdp) +static int direct_map (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, +int map_writable,int max_level,kvm_pfn_t pfn, +bool prefault,bool is_tdp) {bool nx_huge_page_workaround_enabled =is_nx_huge_page_enabled ();bool write = error_code &PFERR_WRITE_MASK ;
@@ -4055,8 +4055,8 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,if (r) goto out_unlock; - r =__direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, - prefault, is_tdp); + r =direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, + prefault, is_tdp); out_unlock:write_unlock (&vcpu->kvm->mmu_lock);
Signature · 1 line
2.51.2
PATCH v2 8/8 KVM: x86: Check for invalid/obsolete root *after* making MMU pages available
In reply to Kenta AkagiFrom: Sean Christopherson <[email protected]>
[ Upstream commit 2abd5287f08319fa35764566b15c6e22cb1068db ]
Check for a "stale" page fault, i.e. for an invalid and/or obsolete root, after making MMU pages available for the shadow MMU. If reclaiming shadow pages zaps an in-use root, i.e. marks it invalid, then KVM will attempt to map memory into an invalid root. On its own, populating an invalid root is "fine", but because child shadow pages inherit their parent's role, any children created during the map/fetch will be created as invalid pages, thus violating KVM's invariant that invalid pages are never on the list of active MMU pages.
Note, the underlying flaw has existed since KVM first started tracking invalid roots in 2008 (commit 2e53d63acba7, "KVM: MMU: ignore zapped root pagetables"), but the true badness only came along in 2020 (Linux 5.9) with the invariant that invalid shadow pages can't be on the list of active pages.
Note #2, inheriting role.invalid when creating child shadow pages is also far from ideal; that flaw will be addressed separately.
[backport note: upstream passes struct kvm_page_fault to is_page_fault_stale();
here individual parameters are used, as is_page_fault_stale() was backported with that interface in the preceding commit]
--- arch/x86/kvm/mmu/mmu.c | 9 +++++---- arch/x86/kvm/mmu/paging_tmpl.h | 10 ++++++---- 2 files changed, 11 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +5 -4
@@ -4045,16 +4045,17 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,if (handle_abnormal_pfn (vcpu, is_tdp ?0 : gpa, gfn, pfn,ACC_ALL , &r))return r; - r =RET_PF_RETRY ;write_lock (&vcpu->kvm->mmu_lock); -if (is_page_fault_stale (vcpu, pfn, mmu_seq, hva)) - goto out_unlock; - r =make_mmu_pages_available (vcpu);if (r) goto out_unlock; +if (is_page_fault_stale (vcpu, pfn, mmu_seq, hva)) { + r =RET_PF_RETRY ; + goto out_unlock; + } + r =direct_map (vcpu, gpa, error_code, map_writable, max_level, pfn, prefault, is_tdp);
arch/x86/kvm/mmu/paging_tmpl.h +6 -4
@@ -923,16 +923,18 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gpa_t addr, u32 error_code, walker.pte_access &= ~ACC_EXEC_MASK ; } - r =RET_PF_RETRY ;write_lock (&vcpu->kvm->mmu_lock); -if (is_page_fault_stale (vcpu, pfn, mmu_seq, hva)) - goto out_unlock; -kvm_mmu_audit (vcpu,AUDIT_PRE_PAGE_FAULT ); r =make_mmu_pages_available (vcpu);if (r) goto out_unlock; + +if (is_page_fault_stale (vcpu, pfn, mmu_seq, hva)) { + r =RET_PF_RETRY ; + goto out_unlock; + } + r =FNAME (fetch)(vcpu, addr, &walker, error_code, max_level, pfn, map_writable, prefault);kvm_mmu_audit (vcpu,AUDIT_POST_PAGE_FAULT );
Signature · 1 line
2.51.2
Re: [PATCH 5.15.y v2 0/8] backport for CVE-2026-64561
In reply to Kenta AkagiQuoted 3 lines · Kenta Akagi
This is v2 of the backport to fix CVE-2026-64561, v1 is Sasha's series [1].
Patches 2-7 in this series correspond to patches 1-6 in v1; Patches 1 and 8 are new in v2.
Queued the series for 5.15, thanks.
One addition on my side: I inserted 18c841e1f411 ("KVM: x86: Retry page fault if MMU reload is pending and root has no sp") immediately after patch 1/8. It is the Cc: stable follow-up to a955cad84cda and the two need to ship together.
Signature · 2 lines
Thanks, Sasha