Skip to contentSkip to conversation

KVM: Skip mmu_notifier invalidations for gmem-only slots

PATCH In review Sean Christopherson 4 participants 9 messages Nov 17 – Nov 18 KVM

lore
Share & tools
lore

Sean Christopherson

author reviewer
#0 lore raw

v4 cover KVM: Skip mmu_notifier invalidations for gmem-only slots

Fix a lurking bug where KVM incorrectly prefetches SPTEs for gmem-only memslots using fast GUP, i.e. VMAs, and then skip mmu_notifier invalidations for gmem-only memslots (because as the name suggests, KVM MMUs are supposed to only get PFNs for gmem-only memslots straight from guest_memfd).

v4:
- Tack on a patch to disallow prefetching gmem-only memslots using a VMA. [Sashiko, Lorenzo]

v3:
- https://lore.kernel.org/all/[email protected]
- v1, but with a different changelog.

v2: https://lore.kernel.org/all/[email protected]

v1: https://lore.kernel.org/all/[email protected]

Alexandru Elisei (1):
KVM: Skip gfn-based MMU notifier invalidations for guest_memfd-only
memslots

Sean Christopherson (1):
KVM: Disallow prefetching PFNs (for SPTEs) via VMAs for gmem-only
memslots

 virt/kvm/kvm_main.c | 7 +++++++
 1 file changed, 7 insertions(+)

base-commit: d21eef2084a9af87ac2a783bc7c2b175ba114e4e

Signature · 1 line
2.56.1.703.gf848ed807c-goog
author reviewer
#1 lore raw

PATCH v4 1/2 KVM: Disallow prefetching PFNs (for SPTEs) via VMAs for gmem-only memslots

Disallow prefetching SPTEs via fast GUP, i.e. by VMA lookup, for gmem-only memslots as the whole point of gmem-only memslots is that KVM MMUs should get PFNs *only* via the direct guest_memfd interface. More importantly, this will allow skipping/ignoring/eliding mmu_notifier invalidations for gmem-only memslots, since it will be impossible for KVM MMUs to have any mappings based on the to-be-invalidated VMA (as is intended).

---
 virt/kvm/kvm_main.c | 3 +++
 1 file changed, 3 insertions(+)
virt/kvm/kvm_main.c +3 -0
index cf4cb586f140..9938b753948c 100644
@@ -3172,6 +3172,9 @@ int kvm_prefetch_pages(struct kvm_memory_slot *slot, gfn_t gfn,
 	unsigned long addr;
 	gfn_t entry = 0;
 
+	if (kvm_memslot_is_gmem_only(slot))
+		return -EINVAL;
+
 	addr = gfn_to_hva_many(slot, gfn, &entry);
 	if (kvm_is_error_hva(addr))
 		return -1;
Signature · 1 line
2.56.1.703.gf848ed807c-goog
Reviewed-by on v4 1/2
#2 lore raw

Re: [PATCH v4 1/2] KVM: Disallow prefetching PFNs (for SPTEs) via VMAs for gmem-only memslots

Quoted 33 lines · Sean Christopherson

Disallow prefetching SPTEs via fast GUP, i.e. by VMA lookup, for gmem-only memslots as the whole point of gmem-only memslots is that KVM MMUs should get PFNs *only* via the direct guest_memfd interface. More importantly, this will allow skipping/ignoring/eliding mmu_notifier invalidations for gmem-only memslots, since it will be impossible for KVM MMUs to have any mappings based on the to-be-invalidated VMA (as is intended).

---
 virt/kvm/kvm_main.c | 3 +++
 1 file changed, 3 insertions(+)
virt/kvm/kvm_main.c +3 -1
index cf4cb586f140..9938b753948c 100644
@@ -3172,6 +3172,9 @@ int kvm_prefetch_pages(struct kvm_memory_slot *slot, gfn_t gfn,
 	unsigned long addr;
 	gfn_t entry = 0;
 
+	if (kvm_memslot_is_gmem_only(slot))
+		return -EINVAL;
+
 	addr = gfn_to_hva_many(slot, gfn, &entry);
 	if (kvm_is_error_hva(addr))
 		return -1;
--
Quoted 1 line

2.56.1.703.gf848ed807c-goog

FWIW, I also looked into kvm_read_guest*() after being pointed there by AI. Those take a HVA, but don't take any kind of long term pin or have any reliance on MMU notifiers, so those don't need extra guards.

Reviewed-by on v4 1/2
#3 lore raw

Re: [PATCH v4 1/2] KVM: Disallow prefetching PFNs (for SPTEs) via VMAs for gmem-only memslots

In reply to Sean Christopherson
Quoted 33 lines · Sean Christopherson

Disallow prefetching SPTEs via fast GUP, i.e. by VMA lookup, for gmem-only memslots as the whole point of gmem-only memslots is that KVM MMUs should get PFNs *only* via the direct guest_memfd interface. More importantly, this will allow skipping/ignoring/eliding mmu_notifier invalidations for gmem-only memslots, since it will be impossible for KVM MMUs to have any mappings based on the to-be-invalidated VMA (as is intended).

---
 virt/kvm/kvm_main.c | 3 +++
 1 file changed, 3 insertions(+)
virt/kvm/kvm_main.c +3 -1
index cf4cb586f140..9938b753948c 100644
@@ -3172,6 +3172,9 @@ int kvm_prefetch_pages(struct kvm_memory_slot *slot, gfn_t gfn,
 	unsigned long addr;
 	gfn_t entry = 0;
 
+	if (kvm_memslot_is_gmem_only(slot))
+		return -EINVAL;
+
 	addr = gfn_to_hva_many(slot, gfn, &entry);
 	if (kvm_is_error_hva(addr))
 		return -1;
--
Quoted 1 line

2.56.1.703.gf848ed807c-goog

FWIW, I also looked into kvm_read_guest*() after being pointed there by AI. Those take a HVA, but don't take any kind of long term pin or have any reliance on MMU notifiers, so those don't need extra guards.

Reviewed-by on v4 1/2
#4 lore raw

Re: [PATCH v4 1/2] KVM: Disallow prefetching PFNs (for SPTEs) via VMAs for gmem-only memslots

In reply to Sean Christopherson
Quoted 7 lines · Sean Christopherson

Disallow prefetching SPTEs via fast GUP, i.e. by VMA lookup, for gmem-only memslots as the whole point of gmem-only memslots is that KVM MMUs should get PFNs *only* via the direct guest_memfd interface. More importantly, this will allow skipping/ignoring/eliding mmu_notifier invalidations for gmem-only memslots, since it will be impossible for KVM MMUs to have any mappings based on the to-be-invalidated VMA (as is intended).

Quoted 7 lines

LGTM, thanks! :)

Quoted 19 lines
---
 virt/kvm/kvm_main.c | 3 +++
 1 file changed, 3 insertions(+)
virt/kvm/kvm_main.c +3 -1
index cf4cb586f140..9938b753948c 100644
@@ -3172,6 +3172,9 @@ int kvm_prefetch_pages(struct kvm_memory_slot *slot, gfn_t gfn,
 	unsigned long addr;
 	gfn_t entry = 0;
 
+	if (kvm_memslot_is_gmem_only(slot))
+		return -EINVAL;
+
 	addr = gfn_to_hva_many(slot, gfn, &entry);
 	if (kvm_is_error_hva(addr))
 		return -1;
--
Quoted 2 lines

2.56.1.703.gf848ed807c-goog

--
Cheers, Lorenzo

author reviewer
#5 lore raw

PATCH v4 2/2 KVM: Skip gfn-based MMU notifier invalidations for guest_memfd-only memslots

In reply to Sean Christopherson

From: Alexandru Elisei <[email protected]>

Don't invalidate gmem-only memslots when responding to MMU notifier invalidations, as the entire point of gmem-only memslots is so that KVM doesn't need a VMA to install mappings into KVM's MMU. Specifically, page faults are resolved by going straight to guest_memfd, using the gfn and the associated memslot's gmem.file. I.e. whether or not userspace happens to have valid mappings to the same gmem instance is irrelevant, as are any invalidations for those mappings.

Note! The exception is KVM's gfn_to_pfn_cache implementation which isn't yet wired up to pull pfns directly from guest_memfd, i.e. still relies on userspace mappings/VMAs and thus need to be invalidated, e.g. to ensure KVM doesn't retain a kernel mapping+pointer to PRIVATE memory. However, as of commit da260bcb56fe ("KVM: pfncache: use a dedicated invalidation sequence for cache refresh"), GPC invalidation no longer relies on memslot lookups, because GPCs also support HVA-based mappings. I.e. KVM *must* be able to detect and handle relevant MMU notifier invalidations even if there is *no* memslot whatsoever.

In addition to the obvious performance benefits, avoiding unnecessary and spurious SPTE zaps and TLB flushes will allow KVM to support features like ARM's Statistical Profiling Extension (SPE)[1] and x86's Secure AVIC[2], both of which rely on the pages *never* being unmapped from the stage-2 page tables.

Cc Lorenzo Stoakes (ARM) <[email protected]>
Cc Naveen N Rao (AMD) <[email protected]>
[sean: rewrite changelog to capture the new gpc behavior]
---
 virt/kvm/kvm_main.c | 4 ++++
 1 file changed, 4 insertions(+)
virt/kvm/kvm_main.c +4 -0
index 9938b753948c..f0d95e8c37f4 100644
@@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 			unsigned long hva_start, hva_end;
 
 			slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]);
+
+			if (kvm_slot_has_gmem(slot) && kvm_memslot_is_gmem_only(slot))
+				continue;
+
 			hva_start = max_t(unsigned long, range->start, slot->userspace_addr);
 			hva_end = min_t(unsigned long, range->end,
 					slot->userspace_addr + (slot->npages << PAGE_SHIFT));
Signature · 1 line
2.56.1.703.gf848ed807c-goog
Reviewed-by on v4 2/2
#6 lore raw

Re: [PATCH v4 2/2] KVM: Skip gfn-based MMU notifier invalidations for guest_memfd-only memslots

Quoted 53 lines · Sean Christopherson

From: Alexandru Elisei <[email protected]>

Don't invalidate gmem-only memslots when responding to MMU notifier invalidations, as the entire point of gmem-only memslots is so that KVM doesn't need a VMA to install mappings into KVM's MMU. Specifically, page faults are resolved by going straight to guest_memfd, using the gfn and the associated memslot's gmem.file. I.e. whether or not userspace happens to have valid mappings to the same gmem instance is irrelevant, as are any invalidations for those mappings.

Note! The exception is KVM's gfn_to_pfn_cache implementation which isn't yet wired up to pull pfns directly from guest_memfd, i.e. still relies on userspace mappings/VMAs and thus need to be invalidated, e.g. to ensure KVM doesn't retain a kernel mapping+pointer to PRIVATE memory. However, as of commit da260bcb56fe ("KVM: pfncache: use a dedicated invalidation sequence for cache refresh"), GPC invalidation no longer relies on memslot lookups, because GPCs also support HVA-based mappings. I.e. KVM *must* be able to detect and handle relevant MMU notifier invalidations even if there is *no* memslot whatsoever.

In addition to the obvious performance benefits, avoiding unnecessary and spurious SPTE zaps and TLB flushes will allow KVM to support features like ARM's Statistical Profiling Extension (SPE)[1] and x86's Secure AVIC[2], both of which rely on the pages *never* being unmapped from the stage-2 page tables.

Cc Lorenzo Stoakes (ARM) <[email protected]>
Cc Naveen N Rao (AMD) <[email protected]>
[sean: rewrite changelog to capture the new gpc behavior]
---
 virt/kvm/kvm_main.c | 4 ++++
 1 file changed, 4 insertions(+)
virt/kvm/kvm_main.c +4 -1
index 9938b753948c..f0d95e8c37f4 100644
@@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 			unsigned long hva_start, hva_end;
 
 			slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]);
+
+			if (kvm_slot_has_gmem(slot) && kvm_memslot_is_gmem_only(slot))
+				continue;
+
 			hva_start = max_t(unsigned long, range->start, slot->userspace_addr);
 			hva_end = min_t(unsigned long, range->end,
 					slot->userspace_addr + (slot->npages << PAGE_SHIFT));
--
Quoted 1 line

2.56.1.703.gf848ed807c-goog

Reviewed-by on v4 2/2
#7 lore raw

Re: [PATCH v4 2/2] KVM: Skip gfn-based MMU notifier invalidations for guest_memfd-only memslots

In reply to Sean Christopherson
Quoted 26 lines · Sean Christopherson

From: Alexandru Elisei <[email protected]>

Don't invalidate gmem-only memslots when responding to MMU notifier invalidations, as the entire point of gmem-only memslots is so that KVM doesn't need a VMA to install mappings into KVM's MMU. Specifically, page faults are resolved by going straight to guest_memfd, using the gfn and the associated memslot's gmem.file. I.e. whether or not userspace happens to have valid mappings to the same gmem instance is irrelevant, as are any invalidations for those mappings.

Note! The exception is KVM's gfn_to_pfn_cache implementation which isn't yet wired up to pull pfns directly from guest_memfd, i.e. still relies on userspace mappings/VMAs and thus need to be invalidated, e.g. to ensure KVM doesn't retain a kernel mapping+pointer to PRIVATE memory. However, as of commit da260bcb56fe ("KVM: pfncache: use a dedicated invalidation sequence for cache refresh"), GPC invalidation no longer relies on memslot lookups, because GPCs also support HVA-based mappings. I.e. KVM *must* be able to detect and handle relevant MMU notifier invalidations even if there is *no* memslot whatsoever.

In addition to the obvious performance benefits, avoiding unnecessary and spurious SPTE zaps and TLB flushes will allow KVM to support features like ARM's Statistical Profiling Extension (SPE)[1] and x86's Secure AVIC[2], both of which rely on the pages *never* being unmapped from the stage-2 page tables.

Quoted 7 lines
Cc Lorenzo Stoakes (ARM) <[email protected]>
Cc Naveen N Rao (AMD) <[email protected]>
[sean: rewrite changelog to capture the new gpc behavior]

LGTM, thanks Alex and Sean! :)

Quoted 20 lines
---
 virt/kvm/kvm_main.c | 4 ++++
 1 file changed, 4 insertions(+)
virt/kvm/kvm_main.c +4 -1
index 9938b753948c..f0d95e8c37f4 100644
@@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 			unsigned long hva_start, hva_end;
 
 			slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]);
+
+			if (kvm_slot_has_gmem(slot) && kvm_memslot_is_gmem_only(slot))
+				continue;
+
 			hva_start = max_t(unsigned long, range->start, slot->userspace_addr);
 			hva_end = min_t(unsigned long, range->end,
 					slot->userspace_addr + (slot->npages << PAGE_SHIFT));
--
Quoted 2 lines

2.56.1.703.gf848ed807c-goog

--
Cheers, Lorenzo

Reviewed-by on v4 2/2
#8 lore raw

Re: [PATCH v4 2/2] KVM: Skip gfn-based MMU notifier invalidations for guest_memfd-only memslots

In reply to Sean Christopherson
Quoted 27 lines

From: Alexandru Elisei <[email protected]>

Don't invalidate gmem-only memslots when responding to MMU notifier invalidations, as the entire point of gmem-only memslots is so that KVM doesn't need a VMA to install mappings into KVM's MMU. Specifically, page faults are resolved by going straight to guest_memfd, using the gfn and the associated memslot's gmem.file. I.e. whether or not userspace happens to have valid mappings to the same gmem instance is irrelevant, as are any invalidations for those mappings.

Note! The exception is KVM's gfn_to_pfn_cache implementation which isn't yet wired up to pull pfns directly from guest_memfd, i.e. still relies on userspace mappings/VMAs and thus need to be invalidated, e.g. to ensure KVM doesn't retain a kernel mapping+pointer to PRIVATE memory. However, as of commit da260bcb56fe ("KVM: pfncache: use a dedicated invalidation sequence for cache refresh"), GPC invalidation no longer relies on memslot lookups, because GPCs also support HVA-based mappings. I.e. KVM *must* be able to detect and handle relevant MMU notifier invalidations even if there is *no* memslot whatsoever.

In addition to the obvious performance benefits, avoiding unnecessary and spurious SPTE zaps and TLB flushes will allow KVM to support features like ARM's Statistical Profiling Extension (SPE)[1] and x86's Secure AVIC[2], both of which rely on the pages *never* being unmapped from the stage-2 page tables.

Quoted 7 lines
Cc Lorenzo Stoakes (ARM) <[email protected]>
Cc Naveen N Rao (AMD) <[email protected]>
[sean: rewrite changelog to capture the new gpc behavior]
Quoted 19 lines
---
  virt/kvm/kvm_main.c | 4 ++++
  1 file changed, 4 insertions(+)
virt/kvm/kvm_main.c +4 -0
index 9938b753948c..f0d95e8c37f4 100644
@@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
  			unsigned long hva_start, hva_end;
 
  			slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]);
+
+			if (kvm_slot_has_gmem(slot) && kvm_memslot_is_gmem_only(slot))
+				continue;
+
  			hva_start = max_t(unsigned long, range->start, slot->userspace_addr);
  			hva_end = min_t(unsigned long, range->end,
  					slot->userspace_addr + (slot->npages << PAGE_SHIFT));