KVM: Skip gfn-based MMU notifier invalidations for guest_memfd-only memslots
Sean Christopherson
maintainerPATCH v3 1/1 KVM: Skip gfn-based MMU notifier invalidations for guest_memfd-only memslots
From: Alexandru Elisei <[email protected]>
Don't invalidate gmem-only memslots when responding to MMU notifier invalidations, as the entire point of gmem-only memslots is so that KVM doesn't need a VMA to install mappings into KVM's MMU. Specifically, page faults are resolved by going straight to guest_memfd, using the gfn and the associated memslot's gmem.file. I.e. whether or not userspace happens to have valid mappings to the same gmem instance is irrelevant, as are any invalidations for those mappings.
Note! The exception is KVM's gfn_to_pfn_cache implementation which isn't yet wired up to pull pfns directly from guest_memfd, i.e. still relies on userspace mappings/VMAs and thus need to be invalidated, e.g. to ensure KVM doesn't retain a kernel mapping+pointer to PRIVATE memory. However, as of commit da260bcb56fe ("KVM: pfncache: use a dedicated invalidation sequence for cache refresh"), GPC invalidation no longer relies on memslot lookups, because GPCs also support HVA-based mappings. I.e. KVM *must* be able to detect and handle relevant MMU notifier invalidations even if there is *no* memslot whatsoever.
In addition to the obvious performance benefits, avoiding unnecessary and spurious SPTE zaps and TLB flushes will allow KVM to support features like ARM's Statistical Profiling Extension (SPE)[1] and x86's Secure AVIC[2], both of which rely on the pages *never* being unmapped from the stage-2 page tables.
---
v3: v1, but with a different changelog.
v2: https://lore.kernel.org/all/[email protected]
v1: https://lore.kernel.org/all/[email protected]
virt/kvm/kvm_main.c | 4 ++++ 1 file changed, 4 insertions(+)
virt/kvm/kvm_main.c +4 -0
@@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,unsigned long hva_start, hva_end; slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]); + +if (kvm_slot_has_gmem (slot) &&kvm_memslot_is_gmem_only (slot)) +continue ; + hva_start = max_t(unsigned long, range->start, slot->userspace_addr); hva_end = min_t(unsigned long, range->end, slot->userspace_addr + (slot->npages <<PAGE_SHIFT ));
base-commit: d21eef2084a9af87ac2a783bc7c2b175ba114e4e
Signature · 1 line
2.56.1.703.gf848ed807c-goog
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Critical] KVM: x86/mmu: host page UAF via kvm_mmu_prefetch_sptes() on gmem-only memslots --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Critical] KVM: x86/mmu: host page UAF via kvm_mmu_prefetch_sptes() on gmem-only memslots
--
commit 8fb2cb0acca2cbb14c599c1c5881381eb579d79c
Author: Alexandru Elisei <[email protected]>
KVM: Skip gfn-based MMU notifier invalidations for guest_memfd-only memslots
This commit skips guest_memfd-only memslots in kvm_handle_hva_range() when handling MMU notifier invalidations.
virt/kvm/kvm_main.c static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, · line 606
Quoted 11 lines
virt/kvm/kvm_main.c +3
-0@@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,unsigned long hva_start, hva_end; slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]); + +if (kvm_slot_has_gmem (slot) &&kvm_memslot_is_gmem_only (slot)) +continue ;
[Severity: Critical]
Can skipping guest_memfd-only memslots here leave writable SPTEs installed by x86's kvm_mmu_prefetch_sptes() unzapped when the VMA at slot->userspace_addr is unmapped?
While __kvm_mmu_faultin_pfn() resolves the primary fault from guest_memfd for gmem-only memslots, the post-fault PTE prefetch paths (such as FNAME(fetch)() -> FNAME(pte_prefetch)() for nested EPT/NPT or shadow paging) call kvm_mmu_prefetch_sptes() for neighboring GFNs without checking kvm_memslot_is_gmem_only():
arch/x86/kvm/mmu/mmu.c:kvm_mmu_prefetch_sptes() {
slot = gfn_to_memslot_dirty_bitmap(vcpu, gfn, access & ACC_WRITE_MASK);
if (!slot)
return false;
nr_pages = kvm_prefetch_pages(slot, gfn, pages, nr_pages);
if (nr_pages <= 0)
return false;
for (i = 0; i < nr_pages; i++, gfn++, sptep++) {
mmu_set_spte(vcpu, slot, sptep, access, gfn,
page_to_pfn(pages[i]), NULL);
...
kvm_release_page_dirty(pages[i]);
}
}kvm_prefetch_pages() resolves PFNs from slot->userspace_addr via get_user_pages_fast_only(), and kvm_mmu_prefetch_sptes() installs writable SPTEs and drops the page references. With kvm_handle_hva_range() ignoring gmem-only memslots, unmapping or reclaiming the VMA at slot->userspace_addr will not zap those prefetched SPTEs, giving the guest a host page use-after-free.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Sean Christopherson
maintainerQuoted 40 lines · On Fri, Nov 13, 2026, [email protected] wrote:
virt/kvm/kvm_main.c static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, · line 606
Quoted 11 lines
virt/kvm/kvm_main.c +3
-0@@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,unsigned long hva_start, hva_end; slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]); + +if (kvm_slot_has_gmem (slot) &&kvm_memslot_is_gmem_only (slot)) +continue ;[Severity: Critical]
Can skipping guest_memfd-only memslots here leave writable SPTEs installed by x86'skvm_mmu_prefetch_sptes()unzapped when the VMA atslot->userspace_addris unmapped?While
__kvm_mmu_faultin_pfn()resolves the primary fault fromguest_memfdfor gmem-only memslots, the post-fault PTE prefetch paths (such as FNAME(fetch)() -> FNAME(pte_prefetch)() for nested EPT/NPT or shadow paging) callkvm_mmu_prefetch_sptes()for neighboring GFNs without checkingkvm_memslot_is_gmem_only():arch/x86/kvm/mmu/mmu.c:kvm_mmu_prefetch_sptes() { slot = gfn_to_memslot_dirty_bitmap(vcpu, gfn, access & ACC_WRITE_MASK); if (!slot) return false; nr_pages = kvm_prefetch_pages(slot, gfn, pages, nr_pages); if (nr_pages <= 0) return false; for (i = 0; i < nr_pages; i++, gfn++, sptep++) { mmu_set_spte(vcpu, slot, sptep, access, gfn, page_to_pfn(pages[i]), NULL); ... kvm_release_page_dirty(pages[i]); } }
Quoted 6 lines
kvm_prefetch_pages()resolves PFNs fromslot->userspace_addrviaget_user_pages_fast_only(), andkvm_mmu_prefetch_sptes()installs writable SPTEs and drops the page references. Withkvm_handle_hva_range()ignoring gmem-only memslots, unmapping or reclaiming the VMA atslot->userspace_addrwill not zap those prefetched SPTEs, giving the guest a host page use-after-free.
Well fudge. That's a bug that needs to be fixed irrespective of this patch, though it obviously needs to land first.
The TDP MMU doesn't support automatic prefetching, yet[*], and I'm planning on using the "normal" fault flow for that support, so I think we can just disable prefetching for gmem-only slots without negatively impacting any real-world workload, and punt on proper support until someone cares, e.g.
arch/x86/kvm/mmu/mmu.c +3 -0
@@ -3245,6 +3245,9 @@ static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep,if (!slot)return false; +if (kvm_memslot_is_gmem_only (slot)) +return false; + nr_pages =kvm_prefetch_pages (slot, gfn, pages, nr_pages);if (nr_pages <=0 )return false;
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [High] KVM: x86/mmu: missing Signed-off-by trailer in commit message - [Medium] KVM: x86/mmu: commit title…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] KVM: x86/mmu: missing Signed-off-by trailer in commit message
- [Medium] KVM: x86/mmu: commit title and message do not match change in kvm_mmu_prefetch_sptes()
- [Low] KVM: x86/mmu: missing Fixes trailer for commit f029f04ddbec --
commit 0029f5853016e6378c3fb60ef0b654f4ef090476
Author: Sean Christopherson <[email protected]>
Quoted 50 lines
KVM: Skip gfn-based MMU notifier invalidations for guest_memfd-only memslots
Quoted 40 lines · On Fri, Nov 13, 2026, [email protected] wrote:
virt/kvm/kvm_main.c static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, · line 606
Quoted 11 lines
virt/kvm/kvm_main.c +3
-0@@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,unsigned long hva_start, hva_end; slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]); + +if (kvm_slot_has_gmem (slot) &&kvm_memslot_is_gmem_only (slot)) +continue ;[Severity: Critical]
Can skipping guest_memfd-only memslots here leave writable SPTEs installed by x86'skvm_mmu_prefetch_sptes()unzapped when the VMA atslot->userspace_addris unmapped?While
__kvm_mmu_faultin_pfn()resolves the primary fault fromguest_memfdfor gmem-only memslots, the post-fault PTE prefetch paths (such as FNAME(fetch)() -> FNAME(pte_prefetch)() for nested EPT/NPT or shadow paging) callkvm_mmu_prefetch_sptes()for neighboring GFNs without checkingkvm_memslot_is_gmem_only():arch/x86/kvm/mmu/mmu.c:kvm_mmu_prefetch_sptes() { slot = gfn_to_memslot_dirty_bitmap(vcpu, gfn, access & ACC_WRITE_MASK); if (!slot) return false; nr_pages = kvm_prefetch_pages(slot, gfn, pages, nr_pages); if (nr_pages <= 0) return false; for (i = 0; i < nr_pages; i++, gfn++, sptep++) { mmu_set_spte(vcpu, slot, sptep, access, gfn, page_to_pfn(pages[i]), NULL); ... kvm_release_page_dirty(pages[i]); } }Quoted 6 lines
kvm_prefetch_pages()resolves PFNs fromslot->userspace_addrviaget_user_pages_fast_only(), andkvm_mmu_prefetch_sptes()installs writable SPTEs and drops the page references. Withkvm_handle_hva_range()ignoring gmem-only memslots, unmapping or reclaiming the VMA atslot->userspace_addrwill not zap those prefetched SPTEs, giving the guest a host page use-after-free.
Quoted 7 lines
Well fudge. That's a bug that needs to be fixed irrespective of this patch, though it obviously needs to land first.
The TDP MMU doesn't support automatic prefetching, yet[*], and I'm planning on using the "normal" fault flow for that support, so I think we can just disable prefetching for gmem-only slots without negatively impacting any real-world workload, and punt on proper support until someone cares, e.g.
[Severity: High]
The commit message is missing a Signed-off-by trailer.
[Severity: Medium]
The commit subject and message are an unedited email reply to a different patch and end mid-sentence, rather than describing the change in kvm_mmu_prefetch_sptes() to skip SPTE prefetching for guest_memfd-only memslots.
[Severity: Low]
Since commit f029f04ddbec ("KVM: x86/mmu: Handle guest page faults for guest_memfd with shared memory") updated __kvm_mmu_faultin_pfn() to fault in pages from guest_memfd for gmem-only memslots while leaving kvm_mmu_prefetch_sptes() resolving PFNs from slot->userspace_addr, should this include:
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 26 lines · Sean Christopherson
From: Alexandru Elisei <[email protected]>
Don't invalidate gmem-only memslots when responding to MMU notifier invalidations, as the entire point of gmem-only memslots is so that KVM doesn't need a VMA to install mappings into KVM's MMU. Specifically, page faults are resolved by going straight to
guest_memfd, using the gfn and the associated memslot's gmem.file. I.e. whether or not userspace happens to have valid mappings to the same gmem instance is irrelevant, as are any invalidations for those mappings.Note! The exception is KVM's
gfn_to_pfn_cacheimplementation which isn't yet wired up to pull pfns directly fromguest_memfd, i.e. still relies on userspace mappings/VMAs and thus need to be invalidated, e.g. to ensure KVM doesn't retain a kernel mapping+pointer to PRIVATE memory. However, as of commitda260bcb56fe("KVM: pfncache: use a dedicated invalidation sequence for cache refresh"), GPC invalidation no longer relies on memslot lookups, because GPCs also support HVA-based mappings. I.e. KVM *must* be able to detect and handle relevant MMU notifier invalidations even if there is *no* memslot whatsoever.In addition to the obvious performance benefits, avoiding unnecessary and spurious SPTE zaps and TLB flushes will allow KVM to support features like ARM's Statistical Profiling Extension (SPE)[1] and x86's Secure AVIC[2], both of which rely on the pages *never* being unmapped from the stage-2 page tables.
Quoted 7 lines
Cc Lorenzo Stoakes (ARM) <[email protected]>Cc Naveen N Rao (AMD) <[email protected]>Signed-off-by Alexandru Elisei <[email protected]>[sean: rewrite changelog to capture the new gpc behavior]Signed-off-by Sean Christopherson <[email protected]>
Thanks a lot for this Sean + Alex! :)
Though, having local AI do a review pass on it flagged an issue (don't worry my own analysis - not pasting some slop here :).
I think actually this might need an additional change for x86, since it doesn't have a gmem-only check for the shadow MMU PTE prefetch:
direct_map()
-> direct_pte_prefetch()
-> __direct_pte_prefetch()
-> direct_pte_prefetch_many()
-> kvm_mmu_prefetch_sptes()
-> kvm_prefetch_pages()
-> get_user_pages_fast_only()And a VMM that's doing something broken (i.e. trying to take advantage of this as a vulnerability), could then intentionally set up a gmem-only slot to point at ordinary anon memory, munmap(), and have a UAF on the underlying folio due to no MMU notifier invalidation.
Fix would be something like:
virt/kvm/kvm_main.c +3 -0
@@ -3172,6 +3172,9 @@ int kvm_prefetch_pages(struct kvm_memory_slot *slot, gfn_t gfn,unsigned long addr;gfn_t entry =0 ; +if (kvm_memslot_is_gmem_only (slot)) +return 0 ; + addr =gfn_to_hva_many (slot, gfn, &entry);if (kvm_is_error_hva (addr))return -1 ;
Quoted 27 lines
---
v3: v1, but with a different changelog.
v2: https://lore.kernel.org/all/[email protected]
v1: https://lore.kernel.org/all/[email protected]
virt/kvm/kvm_main.c | 4 ++++ 1 file changed, 4 insertions(+)virt/kvm/kvm_main.c +4
-0@@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,unsigned long hva_start, hva_end; slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]); + +if (kvm_slot_has_gmem (slot) &&kvm_memslot_is_gmem_only (slot)) +continue ; + hva_start = max_t(unsigned long, range->start, slot->userspace_addr); hva_end = min_t(unsigned long, range->end, slot->userspace_addr + (slot->npages <<PAGE_SHIFT ));
Quoted 4 lines
base-commit:
d21eef2084a9af87ac2a783bc7c2b175ba114e4e--
2.56.1.703.gf848ed807c-goog
--
Cheers, Lorenzo
Sean Christopherson
maintainerQuoted 51 lines · Lorenzo Stoakes (ARM)
Quoted 26 lines · Sean Christopherson
From: Alexandru Elisei <[email protected]>
Don't invalidate gmem-only memslots when responding to MMU notifier invalidations, as the entire point of gmem-only memslots is so that KVM doesn't need a VMA to install mappings into KVM's MMU. Specifically, page faults are resolved by going straight to
guest_memfd, using the gfn and the associated memslot's gmem.file. I.e. whether or not userspace happens to have valid mappings to the same gmem instance is irrelevant, as are any invalidations for those mappings.Note! The exception is KVM's
gfn_to_pfn_cacheimplementation which isn't yet wired up to pull pfns directly fromguest_memfd, i.e. still relies on userspace mappings/VMAs and thus need to be invalidated, e.g. to ensure KVM doesn't retain a kernel mapping+pointer to PRIVATE memory. However, as of commitda260bcb56fe("KVM: pfncache: use a dedicated invalidation sequence for cache refresh"), GPC invalidation no longer relies on memslot lookups, because GPCs also support HVA-based mappings. I.e. KVM *must* be able to detect and handle relevant MMU notifier invalidations even if there is *no* memslot whatsoever.In addition to the obvious performance benefits, avoiding unnecessary and spurious SPTE zaps and TLB flushes will allow KVM to support features like ARM's Statistical Profiling Extension (SPE)[1] and x86's Secure AVIC[2], both of which rely on the pages *never* being unmapped from the stage-2 page tables.
Quoted 7 lines
Cc Lorenzo Stoakes (ARM) <[email protected]>Cc Naveen N Rao (AMD) <[email protected]>Signed-off-by Alexandru Elisei <[email protected]>[sean: rewrite changelog to capture the new gpc behavior]Signed-off-by Sean Christopherson <[email protected]>Thanks a lot for this Sean + Alex! :)
Though, having local AI do a review pass on it flagged an issue (don't worry my own analysis - not pasting some slop here :).
I think actually this might need an additional change for x86, since it doesn't have a gmem-only check for the shadow MMU PTE prefetch:
direct_map()
->direct_pte_prefetch()-> __direct_pte_prefetch() -> direct_pte_prefetch_many() -> kvm_mmu_prefetch_sptes() -> kvm_prefetch_pages() -> get_user_pages_fast_only()
Quoted 4 lines
And a VMM that's doing something broken (i.e. trying to take advantage of this as a vulnerability), could then intentionally set up a gmem-only slot to point at ordinary anon memory,
munmap(), and have a UAF on the underlying folio due to no MMU notifier invalidation.
Ya, Sashiko flagged this as well. I was initially planning on adding the check in kvm_mmu_prefetch_sptes(), but adding the check in kvm_prefetch_pages() is better since the work-in-progress prefetching for x86's TDP MMU will share some of the prefetching checks, but will go through the normal fault-in path instead of using kvm_prefetch_pages(). The return should be -EINVAL or so though.
I'll send a patch on Monday. Thanks much!
https://lore.kernel.org/all/[email protected]
Quoted 52 lines
Fix would be something like:
virt/kvm/kvm_main.c +3
-0@@ -3172,6 +3172,9 @@ int kvm_prefetch_pages(struct kvm_memory_slot *slot, gfn_t gfn,unsigned long addr;gfn_t entry =0 ; +if (kvm_memslot_is_gmem_only (slot)) +return 0 ; + addr =gfn_to_hva_many (slot, gfn, &entry);if (kvm_is_error_hva (addr))return -1 ;Quoted 28 lines
---
v3: v1, but with a different changelog.
v2: https://lore.kernel.org/all/[email protected]
v1: https://lore.kernel.org/all/[email protected]
virt/kvm/kvm_main.c | 4 ++++ 1 file changed, 4 insertions(+)virt/kvm/kvm_main.c +4
-0@@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,unsigned long hva_start, hva_end; slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]); + +if (kvm_slot_has_gmem (slot) &&kvm_memslot_is_gmem_only (slot)) +continue ; + hva_start = max_t(unsigned long, range->start, slot->userspace_addr); hva_end = min_t(unsigned long, range->end, slot->userspace_addr + (slot->npages <<PAGE_SHIFT ));Quoted 4 lines
base-commit:
d21eef2084a9af87ac2a783bc7c2b175ba114e4e--
2.56.1.703.gf848ed807c-goog--
Cheers, Lorenzo
Sean Christopherson
maintainerQuoted 7 lines · Sean Christopherson
Quoted 5 lines · Lorenzo Stoakes (ARM)
On Fri, Nov 13, 2026 at 08:40:00AM -0800, Sean Christopherson wrote:
And a VMM that's doing something broken (i.e. trying to take advantage of this as a vulnerability), could then intentionally set up a gmem-only slot to point at ordinary anon memory,munmap(), and have a UAF on the underlying folio due to no MMU notifier invalidation.
Quoted 7 lines
Ya, Sashiko flagged this as well. I was initially planning on adding the check in
kvm_mmu_prefetch_sptes(), but adding the check inkvm_prefetch_pages()is better since the work-in-progress prefetching for x86's TDP MMU will share some of the prefetching checks, but will go through the normal fault-in path instead of usingkvm_prefetch_pages(). The return should be-EINVALor so though.I'll send a patch on Monday.
Slight addendum: I'm going to a v4 of this "series". I original thought the bug was already reachable, but I was conflating this issue with something else.