Skip to contentSkip to conversation

KVM: SEV: Limit cache flush operations in sev guest memory reclaim events

RFC PATCH In review Jacky Li 4 participants 17 messages Nov 10, 2023 – Dec 2, 2023

Subsystems (4)
lore
Share & tools
lore

Jacky Li

author
#0 lore raw

cover KVM: SEV: Limit cache flush operations in sev guest memory reclaim events

The cache flush operation in sev guest memory reclaim events was originally introduced to prevent security issues due to cache incoherence and untrusted VMM. However when this operation gets triggered, it causes performance degradation to the whole machine.

This cache flush operation is performed in mmu_notifiers, in particular, in the mmu_notifier_invalidate_range_start() function, unconditionally on all guest memory regions. Although the intention was to flush cache lines only when guest memory was deallocated, the excessive invocations include many other cases where this flush is unnecessary.

This RFC proposes using the mmu notifier event to determine whether a cache flush is needed. Specifically, only do the cache flush when the address range is unmapped, cleared, released or migrated. A bitmap module param is also introduced to provide flexibility when flush is needed in more events or no flush is needed depending on the hardware platform.

Note that the cache flush operation in memory reclamation only targets SEV/SEV-ES platforms and no cache flush is needed in SEV-SNP VMs. Therefore the patch series does not apply to the SEV-SNP context.

Jacky Li (4):
KVM: SEV: Drop wbinvd_on_all_cpus() as kvm mmu notifier would flush
the cache
KVM: SEV: Plumb mmu_notifier_event into sev function
KVM: SEV: Limit the call of WBINVDs based on the event type of mmu
notifier
KVM: SEV: Use a bitmap module param to decide whether a cache flush is
needed during the guest memory reclaim

 arch/x86/include/asm/kvm_host.h |  3 +-
 arch/x86/kvm/svm/sev.c          | 62 ++++++++++++++++++++++++---------
 arch/x86/kvm/svm/svm.h          |  3 +-
 arch/x86/kvm/x86.c              |  5 +--
 include/linux/kvm_host.h        |  3 +-
 include/linux/mmu_notifier.h    |  4 +++
 virt/kvm/kvm_main.c             | 14 +++++---
 7 files changed, 68 insertions(+), 26 deletions(-)
Signature · 1 line
2.43.0.rc0.421.g78406f8d94-goog
author
#1 lore raw

RFC PATCH 1/4 KVM: SEV: Drop wbinvd_on_all_cpus() as kvm mmu notifier would flush the cache

Remove the wbinvd_on_all_cpus inside sev_mem_enc_unregister_region() and sev_vm_destroy() because kvm mmu notifier invalidation event would flush the cache.

---
 arch/x86/kvm/svm/sev.c | 14 --------------
 1 file changed, 14 deletions(-)
arch/x86/kvm/svm/sev.c +0 -14
index 4900c078045a..7fbcb7dea2c0 100644
@@ -2043,13 +2043,6 @@ int sev_mem_enc_unregister_region(struct kvm *kvm,
 		goto failed;
 	}
 
-	/*
-	 * Ensure that all guest tagged cache entries are flushed before
-	 * releasing the pages back to the system for use. CLFLUSH will
-	 * not do this, so issue a WBINVD.
-	 */
-	wbinvd_on_all_cpus();
-
 	__unregister_enc_region_locked(kvm, region);
 
 	mutex_unlock(&kvm->lock);
@@ -2147,13 +2140,6 @@ void sev_vm_destroy(struct kvm *kvm)
 		return;
 	}
 
-	/*
-	 * Ensure that all guest tagged cache entries are flushed before
-	 * releasing the pages back to the system for use. CLFLUSH will
-	 * not do this, so issue a WBINVD.
-	 */
-	wbinvd_on_all_cpus();
-
 	/*
 	 * if userspace was terminated before unregistering the memory regions
 	 * then lets unpin all the registered memory.
Signature · 1 line
2.43.0.rc0.421.g78406f8d94-goog
#2 lore raw

Re: [RFC PATCH 1/4] KVM: SEV: Drop wbinvd_on_all_cpus() as kvm mmu notifier would flush the cache

Quoted 3 lines · Jacky Li

Remove the wbinvd_on_all_cpus inside sev_mem_enc_unregister_region() and sev_vm_destroy() because kvm mmu notifier invalidation event would flush the cache.

This needs a much longer explanation of why this is safe. This might also need an opt-in, e.g. if userspace is reusing the memory for something else without freeing it back to the kernel, and thus is relying on KVM to do the WBINVD.

The key thing is that userspace can access the memory at any time and _can_ do CLFLUSH{OPT} if userspace wants to do its own conversions. I.e. the WBINVD doesn't protect against a misbehaving corrupting guest/userspace data. But it's still possible that userspace is relying on the WBINVD, and thou shalt not break userspace.

author
#3 lore raw

RFC PATCH 2/4 KVM: SEV: Plumb mmu_notifier_event into sev function

In reply to Jacky Li

Plumb mmu_notifier_event enum all the way to sev function so that the enum can provide proper information for SEV/SEV-ES VMs to do the cache flush when necessary.

---
 arch/x86/include/asm/kvm_host.h |  3 ++-
 arch/x86/kvm/svm/sev.c          |  3 ++-
 arch/x86/kvm/svm/svm.h          |  3 ++-
 arch/x86/kvm/x86.c              |  5 +++--
 include/linux/kvm_host.h        |  3 ++-
 virt/kvm/kvm_main.c             | 14 +++++++++-----
 6 files changed, 20 insertions(+), 11 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -1
index d7036982332e..c026e171a8c8 100644
@@ -1741,7 +1741,8 @@ struct kvm_x86_ops {
 	int (*mem_enc_unregister_region)(struct kvm *kvm, struct kvm_enc_region *argp);
 	int (*vm_copy_enc_context_from)(struct kvm *kvm, unsigned int source_fd);
 	int (*vm_move_enc_context_from)(struct kvm *kvm, unsigned int source_fd);
-	void (*guest_memory_reclaimed)(struct kvm *kvm);
+	void (*guest_memory_reclaimed)(struct kvm *kvm,
+				       unsigned int mmu_notifier_event);
 
 	int (*get_msr_feature)(struct kvm_msr_entry *entry);
 
arch/x86/kvm/svm/sev.c +2 -1
index 7fbcb7dea2c0..8d30f6c5e872 100644
@@ -2329,7 +2329,8 @@ static void sev_flush_encrypted_page(struct kvm_vcpu *vcpu, void *va)
 	wbinvd_on_all_cpus();
 }
 
-void sev_guest_memory_reclaimed(struct kvm *kvm)
+void sev_guest_memory_reclaimed(struct kvm *kvm,
+				unsigned int mmu_notifier_event)
 {
 	if (!sev_guest(kvm))
 		return;
arch/x86/kvm/svm/svm.h +2 -1
index be67ab7fdd10..c8a911a02509 100644
@@ -676,7 +676,8 @@ int sev_mem_enc_unregister_region(struct kvm *kvm,
 				  struct kvm_enc_region *range);
 int sev_vm_copy_enc_context_from(struct kvm *kvm, unsigned int source_fd);
 int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd);
-void sev_guest_memory_reclaimed(struct kvm *kvm);
+void sev_guest_memory_reclaimed(struct kvm *kvm,
+				unsigned int mmu_notifier_event);
 
 void pre_sev_run(struct vcpu_svm *svm, int cpu);
 void __init sev_set_cpu_caps(void);
arch/x86/kvm/x86.c +3 -2
index 2c924075f6f1..2cde9a836bf7 100644
@@ -10592,9 +10592,10 @@ static void vcpu_load_eoi_exitmap(struct kvm_vcpu *vcpu)
 		vcpu, (u64 *)vcpu->arch.ioapic_handled_vectors);
 }
 
-void kvm_arch_guest_memory_reclaimed(struct kvm *kvm)
+void kvm_arch_guest_memory_reclaimed(struct kvm *kvm,
+				     unsigned int mmu_notifier_event)
 {
-	static_call_cond(kvm_x86_guest_memory_reclaimed)(kvm);
+	static_call_cond(kvm_x86_guest_memory_reclaimed)(kvm, mmu_notifier_event);
 }
 
 static void kvm_vcpu_reload_apic_access_page(struct kvm_vcpu *vcpu)
include/linux/kvm_host.h +2 -1
index 4944136efaa2..8984414c5b7a 100644
@@ -2270,7 +2270,8 @@ static inline long kvm_arch_vcpu_async_ioctl(struct file *filp,
 }
 #endif /* CONFIG_HAVE_KVM_VCPU_ASYNC_IOCTL */
 
-void kvm_arch_guest_memory_reclaimed(struct kvm *kvm);
+void kvm_arch_guest_memory_reclaimed(struct kvm *kvm,
+				     unsigned int mmu_notifier_event);
 
 #ifdef CONFIG_HAVE_KVM_VCPU_RUN_PID_CHANGE
 int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu);
virt/kvm/kvm_main.c +9 -5
index 486800a7024b..18526e198993 100644
@@ -154,7 +154,8 @@ static unsigned long long kvm_active_vms;
 
 static DEFINE_PER_CPU(cpumask_var_t, cpu_kick_mask);
 
-__weak void kvm_arch_guest_memory_reclaimed(struct kvm *kvm)
+__weak void kvm_arch_guest_memory_reclaimed(struct kvm *kvm,
+					    unsigned int mmu_notifier_event)
 {
 }
 
@@ -396,7 +397,7 @@ void kvm_flush_remote_tlbs_memslot(struct kvm *kvm,
 static void kvm_flush_shadow_all(struct kvm *kvm)
 {
 	kvm_arch_flush_shadow_all(kvm);
-	kvm_arch_guest_memory_reclaimed(kvm);
+	kvm_arch_guest_memory_reclaimed(kvm, MMU_NOTIFY_RELEASE);
 }
 
 #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
@@ -546,11 +547,13 @@ typedef bool (*hva_handler_t)(struct kvm *kvm, struct kvm_gfn_range *range);
 typedef void (*on_lock_fn_t)(struct kvm *kvm, unsigned long start,
 			     unsigned long end);
 
-typedef void (*on_unlock_fn_t)(struct kvm *kvm);
+typedef void (*on_unlock_fn_t)(struct kvm *kvm,
+			       unsigned int mmu_notifier_event);
 
 struct kvm_hva_range {
 	unsigned long start;
 	unsigned long end;
+	unsigned int event;
 	union kvm_mmu_notifier_arg arg;
 	hva_handler_t handler;
 	on_lock_fn_t on_lock;
@@ -647,7 +650,7 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm,
 	if (locked) {
 		KVM_MMU_UNLOCK(kvm);
 		if (!IS_KVM_NULL_FN(range->on_unlock))
-			range->on_unlock(kvm);
+			range->on_unlock(kvm, range->event);
 	}
 
 	srcu_read_unlock(&kvm->srcu, idx);
@@ -774,6 +777,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 	const struct kvm_hva_range hva_range = {
 		.start		= range->start,
 		.end		= range->end,
+		.event		= range->event,
 		.handler	= kvm_unmap_gfn_range,
 		.on_lock	= kvm_mmu_invalidate_begin,
 		.on_unlock	= kvm_arch_guest_memory_reclaimed,
@@ -1769,7 +1773,7 @@ static void kvm_invalidate_memslot(struct kvm *kvm,
 	 *	- kvm_is_visible_gfn (mmu_check_root)
 	 */
 	kvm_arch_flush_shadow_memslot(kvm, old);
-	kvm_arch_guest_memory_reclaimed(kvm);
+	kvm_arch_guest_memory_reclaimed(kvm, MMU_NOTIFY_UNMAP);
 
 	/* Was released by kvm_swap_active_memslots(), reacquire. */
 	mutex_lock(&kvm->slots_arch_lock);
Signature · 1 line
2.43.0.rc0.421.g78406f8d94-goog
author
#4 lore raw

RFC PATCH 3/4 KVM: SEV: Limit the call of WBINVDs based on the event type of mmu notifier

In reply to Jacky Li

The cache flush was originally introduced to enforce the cache coherency across VM boundary in SEV, so the flush is not needed in some cases when the page remains in the same VM. wbinvd_on_all_cpus() is a costly operation so use the mmu notifier event type information in the range struct to only do cache flush when needed.

The physical page might be allocated to a different VM after the range is unmapped, cleared, released or migrated. So do a cache flush only on those events.

---
 arch/x86/kvm/svm/sev.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kvm/svm/sev.c +5 -1
index 8d30f6c5e872..477df8a06629 100644
@@ -2335,7 +2335,11 @@ void sev_guest_memory_reclaimed(struct kvm *kvm,
 	if (!sev_guest(kvm))
 		return;
 
-	wbinvd_on_all_cpus();
+	if (mmu_notifier_event == MMU_NOTIFY_UNMAP ||
+	    mmu_notifier_event == MMU_NOTIFY_CLEAR ||
+	    mmu_notifier_event == MMU_NOTIFY_RELEASE ||
+	    mmu_notifier_event == MMU_NOTIFY_MIGRATE)
+		wbinvd_on_all_cpus();
 }
 
 void sev_free_vcpu(struct kvm_vcpu *vcpu)
Signature · 1 line
2.43.0.rc0.421.g78406f8d94-goog
#5 lore raw

Re: [RFC PATCH 3/4] KVM: SEV: Limit the call of WBINVDs based on the event type of mmu notifier

Quoted 22 lines · Jacky Li

The cache flush was originally introduced to enforce the cache coherency across VM boundary in SEV, so the flush is not needed in some cases when the page remains in the same VM. wbinvd_on_all_cpus() is a costly operation so use the mmu notifier event type information in the range struct to only do cache flush when needed.

The physical page might be allocated to a different VM after the range is unmapped, cleared, released or migrated. So do a cache flush only on those events.

---
  arch/x86/kvm/svm/sev.c | 6 +++++-
  1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kvm/svm/sev.c +0 -0
index 8d30f6c5e872..477df8a06629 100644

arch/x86/kvm/svm/sev.c void sev_guest_memory_reclaimed(struct kvm *kvm, · line 2339

Quoted 6 lines
arch/x86/kvm/svm/sev.c +2 -1
@@ -2335,7 +2335,11 @@ void sev_guest_memory_reclaimed(struct kvm *kvm,
  	if (!sev_guest(kvm))
  		return;
 
-	wbinvd_on_all_cpus();
+	if (mmu_notifier_event == MMU_NOTIFY_UNMAP ||
+	    mmu_notifier_event == MMU_NOTIFY_CLEAR ||

When i looked at MMU notifier event filtering and the various code paths (of interest) from where the MMU invalidation notifier gets invoked:

For NUMA load balancing during #PF fault handling, try_to_migrate_one() does MMU invalidation notifier invocation with MMU_NOTIFY_CLEAR event and then looking at KSM code paths, try_to_merge_one_page() -> write_protect_page() and try_to_merge_one_page() -> replace_page() does MMU invalidation notifier invocations also with MMU_NOTIFY_CLEAR event.

Now, i remember from previous discussions, that the CLEAR event is an overloaded event used for page zapping, madvise, etc., so i don't think we will be able to filter *out* this event and if this is the case then i have concerns about this whole patch-series as this event is triggering most of the performance issues we are observing.

Thanks,
Ashish

Quoted 7 lines
arch/x86/kvm/svm/sev.c +3 -0
+	    mmu_notifier_event == MMU_NOTIFY_RELEASE ||
+	    mmu_notifier_event == MMU_NOTIFY_MIGRATE)
+		wbinvd_on_all_cpus();
  }
 
  void sev_free_vcpu(struct kvm_vcpu *vcpu)
author
#6 lore raw

RFC PATCH 4/4 KVM: SEV: Use a bitmap module param to decide whether a cache flush is needed during the guest memory reclaim

In reply to Jacky Li

Use a bitmap to provide the flexibility on deciding whether the flush is needed for a specific mmu notifier event. The cache flush during memory reclamation was originally introduced to address the cache incoherency issues in some SME_COHERENT platforms. User may configure the bitmap depending on the hardware (e.g. No flush needed when SME_COHERENT can extend to DMA devices) or userspace VMM (e.g. No flush needed when VMM ensures guest memory is properly unpinned).

The bitmap also decouples itself from the mmu_notifier_event type to provide a consistent interface. The parameter remains same behavior regardless of the changes to mmu notifier event in future kernel versions. When a new mmu notifier event is added, the new event will be defaulted to BIT(0) so that no additional cache flush will be accidentally introduced.

---
 arch/x86/kvm/svm/sev.c       | 47 +++++++++++++++++++++++++++++++++---
 include/linux/mmu_notifier.h |  4 +++
 2 files changed, 47 insertions(+), 4 deletions(-)
arch/x86/kvm/svm/sev.c +43 -4
index 477df8a06629..6e7530b4ae5d 100644
@@ -65,6 +65,47 @@ module_param_named(debug_swap, sev_es_debug_swap_enabled, bool, 0444);
 #define sev_es_debug_swap_enabled false
 #endif /* CONFIG_KVM_AMD_SEV */
 
+#define MMU_NOTIFY_OTHERS_BIT BIT_ULL(0)
+#define MMU_NOTIFY_UNMAP_BIT BIT_ULL(1)
+#define MMU_NOTIFY_CLEAR_BIT BIT_ULL(2)
+#define MMU_NOTIFY_PROTECTION_VMA_BIT BIT_ULL(3)
+#define MMU_NOTIFY_PROTECTION_PAGE_BIT BIT_ULL(4)
+#define MMU_NOTIFY_SOFT_DIRTY_BIT BIT_ULL(5)
+#define MMU_NOTIFY_RELEASE_BIT BIT_ULL(6)
+#define MMU_NOTIFY_MIGRATE_BIT BIT_ULL(7)
+#define MMU_NOTIFY_EXCLUSIVE_BIT BIT_ULL(8)
+
+/*
+ * Explicitly decouple with the mmu_notifier_event enum, so that the interface
+ * (i.e. bit definitions in the module param bitmp) remains the same when the
+ * original enum get updated.
+ */
+static const int mmu_notifier_event_map[NR_MMU_NOTIFY_EVENTS] = {
+	[MMU_NOTIFY_UNMAP] = MMU_NOTIFY_UNMAP_BIT,
+	[MMU_NOTIFY_CLEAR] = MMU_NOTIFY_CLEAR_BIT,
+	[MMU_NOTIFY_PROTECTION_VMA] = MMU_NOTIFY_PROTECTION_VMA_BIT,
+	[MMU_NOTIFY_PROTECTION_PAGE] = MMU_NOTIFY_PROTECTION_PAGE_BIT,
+	[MMU_NOTIFY_SOFT_DIRTY] = MMU_NOTIFY_SOFT_DIRTY_BIT,
+	[MMU_NOTIFY_RELEASE] = MMU_NOTIFY_RELEASE_BIT,
+	[MMU_NOTIFY_MIGRATE] = MMU_NOTIFY_MIGRATE_BIT,
+	[MMU_NOTIFY_EXCLUSIVE] = MMU_NOTIFY_EXCLUSIVE_BIT
+};
+unsigned long flush_on_mmu_notifier_event_bitmap = MMU_NOTIFY_UNMAP_BIT |
+	MMU_NOTIFY_CLEAR_BIT | MMU_NOTIFY_RELEASE_BIT | MMU_NOTIFY_MIGRATE_BIT;
+EXPORT_SYMBOL_GPL(flush_on_mmu_notifier_event_bitmap);
+module_param(flush_on_mmu_notifier_event_bitmap, ulong, 0644);
+MODULE_PARM_DESC(flush_on_mmu_notifier_event_bitmap,
+"Whether a cache flush is needed when the sev guest memory is reclaimed with a specific mmu notifier event.\n"
+"\tBit 0 (0x01)  left to any event not yet defined in the map\n"
+"\tBit 1 (0x02)  corresponds to MMU_NOTIFY_UNMAP event\n"
+"\tBit 2 (0x04)  corresponds to MMU_NOTIFY_CLEAR event\n"
+"\tBit 3 (0x08)  corresponds to MMU_NOTIFY_PROTECTION_VMA event\n"
+"\tBit 4 (0x10)  corresponds to MMU_NOTIFY_PROTECTION_PAGE event\n"
+"\tBit 5 (0x20)  corresponds to MMU_NOTIFY_SOFT_DIRTY event\n"
+"\tBit 6 (0x80)  corresponds to MMU_NOTIFY_RELEASE event\n"
+"\tBit 7 (0x100) corresponds to MMU_NOTIFY_MIGRATE event\n"
+"\tBit 8 (0x200) corresponds to MMU_NOTIFY_EXCLUSIVE event");
+
 static u8 sev_enc_bit;
 static DECLARE_RWSEM(sev_deactivate_lock);
 static DEFINE_MUTEX(sev_bitmap_lock);
@@ -2335,10 +2376,8 @@ void sev_guest_memory_reclaimed(struct kvm *kvm,
 	if (!sev_guest(kvm))
 		return;
 
-	if (mmu_notifier_event == MMU_NOTIFY_UNMAP ||
-	    mmu_notifier_event == MMU_NOTIFY_CLEAR ||
-	    mmu_notifier_event == MMU_NOTIFY_RELEASE ||
-	    mmu_notifier_event == MMU_NOTIFY_MIGRATE)
+	if (mmu_notifier_event_map[mmu_notifier_event] &
+	    flush_on_mmu_notifier_event_bitmap)
 		wbinvd_on_all_cpus();
 }
 
include/linux/mmu_notifier.h +4 -0
index f349e08a9dfe..b40db51d76a4 100644
@@ -47,6 +47,9 @@ struct mmu_interval_notifier;
  * longer have exclusive access to the page. When sent during creation of an
  * exclusive range the owner will be initialised to the value provided by the
  * caller of make_device_exclusive_range(), otherwise the owner will be NULL.
+ *
+ * @NR_MMU_NOTIFY_EVENTS: number of mmu notifier events, should always be at
+ * the end of the enum list.
  */
 enum mmu_notifier_event {
 	MMU_NOTIFY_UNMAP = 0,
@@ -57,6 +60,7 @@ enum mmu_notifier_event {
 	MMU_NOTIFY_RELEASE,
 	MMU_NOTIFY_MIGRATE,
 	MMU_NOTIFY_EXCLUSIVE,
+	NR_MMU_NOTIFY_EVENTS,
 };
 
 #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0)
Signature · 1 line
2.43.0.rc0.421.g78406f8d94-goog
#7 lore raw

Re: [RFC PATCH 4/4] KVM: SEV: Use a bitmap module param to decide whether a cache flush is needed during the guest memory reclaim

Quoted 7 lines · Jacky Li

Use a bitmap to provide the flexibility on deciding whether the flush is needed for a specific mmu notifier event. The cache flush during memory reclamation was originally introduced to address the cache incoherency issues in some SME_COHERENT platforms. User may configure the bitmap depending on the hardware (e.g. No flush needed when SME_COHERENT can extend to DMA devices) or userspace VMM (e.g. No flush needed when VMM ensures guest memory is properly unpinned).

Absolutely not. KVM and the kernel must know when a flush is needed and when it is not. The various errata around cache line aliasing is far too messy to punt to userspace. Not to mention this bleeds kernel internals to userspace and thus risks unintentionally creating an ABI.

#8 lore raw
In reply to Jacky Li
Quoted 11 lines · Jacky Li

The cache flush operation in sev guest memory reclaim events was originally introduced to prevent security issues due to cache incoherence and untrusted VMM. However when this operation gets triggered, it causes performance degradation to the whole machine.

This cache flush operation is performed in mmu_notifiers, in particular, in the mmu_notifier_invalidate_range_start() function, unconditionally on all guest memory regions. Although the intention was to flush cache lines only when guest memory was deallocated, the excessive invocations include many other cases where this flush is unnecessary.

Quoted 6 lines

This RFC proposes using the mmu notifier event to determine whether a cache flush is needed. Specifically, only do the cache flush when the address range is unmapped, cleared, released or migrated. A bitmap module param is also introduced to provide flexibility when flush is needed in more events or no flush is needed depending on the hardware platform.

I'm still not at all convinced that this is worth doing. We have clear line of sight to cleanly and optimally handling SNP and beyond. If there is an actual use case that wants to run SEV and/or SEV-ES VMs, which can't support page migration, on the same host as traditional VMs, _and_ for some reason their userspace is incapable of providing reasonable NUMA locality, then the owners of that use case can speak up and provide justification for taking on this extra complexity in KVM.

#9 lore raw
Quoted 20 lines · Sean Christopherson
Quoted 11 lines · Jacky Li

The cache flush operation in sev guest memory reclaim events was originally introduced to prevent security issues due to cache incoherence and untrusted VMM. However when this operation gets triggered, it causes performance degradation to the whole machine.

This cache flush operation is performed in mmu_notifiers, in particular, in the mmu_notifier_invalidate_range_start() function, unconditionally on all guest memory regions. Although the intention was to flush cache lines only when guest memory was deallocated, the excessive invocations include many other cases where this flush is unnecessary.

Quoted 6 lines

This RFC proposes using the mmu notifier event to determine whether a cache flush is needed. Specifically, only do the cache flush when the address range is unmapped, cleared, released or migrated. A bitmap module param is also introduced to provide flexibility when flush is needed in more events or no flush is needed depending on the hardware platform.

Quoted 7 lines

I'm still not at all convinced that this is worth doing. We have clear line of sight to cleanly and optimally handling SNP and beyond. If there is an actual use case that wants to run SEV and/or SEV-ES VMs, which can't support page migration, on the same host as traditional VMs, _and_ for some reason their userspace is incapable of providing reasonable NUMA locality, then the owners of that use case can speak up and provide justification for taking on this extra complexity in KVM.

Hi Sean,

Jacky and I were looking at some cases like mmu_notifier calls triggered by the overloaded reason "MMU_NOTIFY_CLEAR". Even if we turn off page migration etc, splitting PMD may still happen at some point under this reason, and we will never be able to turn it off by tweaking kernel CONFIG options. So, I think this is the line of sight for this series.

Handling SNP could be separate, since in SNP we have per-page properties, which allow KVM to know which page to flush individually.

Thanks.
-Mingwei

#10 lore raw
Quoted 38 lines · Mingwei Zhang
Quoted 20 lines · Sean Christopherson
Quoted 11 lines · Jacky Li

The cache flush operation in sev guest memory reclaim events was originally introduced to prevent security issues due to cache incoherence and untrusted VMM. However when this operation gets triggered, it causes performance degradation to the whole machine.

This cache flush operation is performed in mmu_notifiers, in particular, in the mmu_notifier_invalidate_range_start() function, unconditionally on all guest memory regions. Although the intention was to flush cache lines only when guest memory was deallocated, the excessive invocations include many other cases where this flush is unnecessary.

Quoted 6 lines

This RFC proposes using the mmu notifier event to determine whether a cache flush is needed. Specifically, only do the cache flush when the address range is unmapped, cleared, released or migrated. A bitmap module param is also introduced to provide flexibility when flush is needed in more events or no flush is needed depending on the hardware platform.

Quoted 7 lines

I'm still not at all convinced that this is worth doing. We have clear line of sight to cleanly and optimally handling SNP and beyond. If there is an actual use case that wants to run SEV and/or SEV-ES VMs, which can't support page migration, on the same host as traditional VMs, _and_ for some reason their userspace is incapable of providing reasonable NUMA locality, then the owners of that use case can speak up and provide justification for taking on this extra complexity in KVM.

Hi Sean,

Jacky and I were looking at some cases like mmu_notifier calls triggered by the overloaded reason "MMU_NOTIFY_CLEAR". Even if we turn off page migration etc, splitting PMD may still happen at some point under this reason, and we will never be able to turn it off by tweaking kernel CONFIG options. So, I think this is the line of sight for this series.

Quoted 3 lines

Handling SNP could be separate, since in SNP we have per-page properties, which allow KVM to know which page to flush individually.

For SNP + gmem, where the HVA ranges covered by the MMU notifiers are not acting on encrypted pages, we are ignoring MMU invalidation notifiers for SNP guests as part of the SNP host patches being posted upstream and instead relying on gmem own invalidation stuff to clean them up on a per-folio basis.

Thanks,
Ashish

#11 lore raw
Quoted 44 lines · Ashish
Quoted 38 lines · Mingwei Zhang
Quoted 20 lines · Sean Christopherson
Quoted 11 lines · Jacky Li

The cache flush operation in sev guest memory reclaim events was originally introduced to prevent security issues due to cache incoherence and untrusted VMM. However when this operation gets triggered, it causes performance degradation to the whole machine.

This cache flush operation is performed in mmu_notifiers, in particular, in the mmu_notifier_invalidate_range_start() function, unconditionally on all guest memory regions. Although the intention was to flush cache lines only when guest memory was deallocated, the excessive invocations include many other cases where this flush is unnecessary.

Quoted 6 lines

This RFC proposes using the mmu notifier event to determine whether a cache flush is needed. Specifically, only do the cache flush when the address range is unmapped, cleared, released or migrated. A bitmap module param is also introduced to provide flexibility when flush is needed in more events or no flush is needed depending on the hardware platform.

Quoted 7 lines

I'm still not at all convinced that this is worth doing. We have clear line of sight to cleanly and optimally handling SNP and beyond. If there is an actual use case that wants to run SEV and/or SEV-ES VMs, which can't support page migration, on the same host as traditional VMs, _and_ for some reason their userspace is incapable of providing reasonable NUMA locality, then the owners of that use case can speak up and provide justification for taking on this extra complexity in KVM.

Hi Sean,

Jacky and I were looking at some cases like mmu_notifier calls triggered by the overloaded reason "MMU_NOTIFY_CLEAR". Even if we turn off page migration etc, splitting PMD may still happen at some point under this reason, and we will never be able to turn it off by tweaking kernel CONFIG options. So, I think this is the line of sight for this series.

Quoted 3 lines

Handling SNP could be separate, since in SNP we have per-page properties, which allow KVM to know which page to flush individually.

Quoted 8 lines

For SNP + gmem, where the HVA ranges covered by the MMU notifiers are not acting on encrypted pages, we are ignoring MMU invalidation notifiers for SNP guests as part of the SNP host patches being posted upstream and instead relying on gmem own invalidation stuff to clean them up on a per-folio basis.

Thanks,
Ashish

oh, I have no question about that. This series only applies to SEV/SEV-ES type of VMs.

For SNP + guest_memfd, I don't see the implementation details, but I doubt you can ignore mmu_notifiers if the request does cover some encrypted memory in error cases or corner cases. Does the SNP enforce the usage of guest_memfd? How do we prevent exceptional cases? I am sure you guys already figured out the answers, so I don't plan to dig deeper until SNP host pages are accepted.

Clearly, for SEV/SEV-ES, there is no such guarantee like guest_memfd. Applying guest_memfd on SEV/SEV-ES might require changes on SEV API I suspect, so I think that's equally non-trivial and thus may not be worth pursuing.

Thanks.
-Mingwei

#12 lore raw
Quoted 51 lines · Mingwei Zhang
Quoted 44 lines · Ashish
Quoted 38 lines · Mingwei Zhang
Quoted 20 lines · Sean Christopherson
Quoted 11 lines · Jacky Li

The cache flush operation in sev guest memory reclaim events was originally introduced to prevent security issues due to cache incoherence and untrusted VMM. However when this operation gets triggered, it causes performance degradation to the whole machine.

This cache flush operation is performed in mmu_notifiers, in particular, in the mmu_notifier_invalidate_range_start() function, unconditionally on all guest memory regions. Although the intention was to flush cache lines only when guest memory was deallocated, the excessive invocations include many other cases where this flush is unnecessary.

Quoted 6 lines

This RFC proposes using the mmu notifier event to determine whether a cache flush is needed. Specifically, only do the cache flush when the address range is unmapped, cleared, released or migrated. A bitmap module param is also introduced to provide flexibility when flush is needed in more events or no flush is needed depending on the hardware platform.

Quoted 7 lines

I'm still not at all convinced that this is worth doing. We have clear line of sight to cleanly and optimally handling SNP and beyond. If there is an actual use case that wants to run SEV and/or SEV-ES VMs, which can't support page migration, on the same host as traditional VMs, _and_ for some reason their userspace is incapable of providing reasonable NUMA locality, then the owners of that use case can speak up and provide justification for taking on this extra complexity in KVM.

Hi Sean,

Jacky and I were looking at some cases like mmu_notifier calls triggered by the overloaded reason "MMU_NOTIFY_CLEAR". Even if we turn off page migration etc, splitting PMD may still happen at some point under this reason, and we will never be able to turn it off by tweaking kernel CONFIG options. So, I think this is the line of sight for this series.

Quoted 3 lines

Handling SNP could be separate, since in SNP we have per-page properties, which allow KVM to know which page to flush individually.

Quoted 6 lines

For SNP + gmem, where the HVA ranges covered by the MMU notifiers are not acting on encrypted pages, we are ignoring MMU invalidation notifiers for SNP guests as part of the SNP host patches being posted upstream and instead relying on gmem own invalidation stuff to clean them up on a per-folio basis.

Quoted 6 lines

oh, I have no question about that. This series only applies to SEV/SEV-ES type of VMs.

For SNP + guest_memfd, I don't see the implementation details, but I doubt you can ignore mmu_notifiers if the request does cover some encrypted memory in error cases or corner cases.

I believe that all page state transitions from private->shared will invoke gmem's own invalidation stuff which should cover such corner cases.

Mike Roth can provide more specific details about that.

Quoted 1 line

Does the SNP enforce the usage of guest_memfd?

Again i believe that SNP implementation is only based on and uses guest memfd support.

Thanks,
Ashish

Quoted 12 lines

How do we prevent exceptional cases? I am
sure you guys already figured out the answers, so I don't plan to dig deeper until SNP host pages are accepted.

Clearly, for SEV/SEV-ES, there is no such guarantee like guest_memfd. Applying guest_memfd on SEV/SEV-ES might require changes on SEV API I suspect, so I think that's equally non-trivial and thus may not be worth pursuing.

Thanks.
-Mingwei

#13 lore raw
In reply to Mingwei Zhang
Quoted 13 lines · Mingwei Zhang
Quoted 8 lines · Ashish

For SNP + gmem, where the HVA ranges covered by the MMU notifiers are not acting on encrypted pages, we are ignoring MMU invalidation notifiers for SNP guests as part of the SNP host patches being posted upstream and instead relying on gmem own invalidation stuff to clean them up on a per-folio basis.

Thanks,
Ashish

oh, I have no question about that. This series only applies to SEV/SEV-ES type of VMs.

Quoted 6 lines

For SNP + guest_memfd, I don't see the implementation details, but I doubt you can ignore mmu_notifiers if the request does cover some encrypted memory in error cases or corner cases. Does the SNP enforce the usage of guest_memfd? How do we prevent exceptional cases? I am sure you guys already figured out the answers, so I don't plan to dig deeper until SNP host pages are accepted.

KVM will not allow SNP guests to map VMA-based memory as encrypted/private, full stop. Any invalidations initiated by mmu_notifiers will therefore apply only to shared memory.

That approach doesn't work for SEV/SEV-ES because KVM can't prevent the guest from accessing memory as encrypted, i.e. KVM needs the #NPF due to RMP violation to intercept attempts to convert a GFN from shared to private.

#14 lore raw
Quoted 22 lines · Sean Christopherson
Quoted 13 lines · Mingwei Zhang
Quoted 8 lines · Ashish

For SNP + gmem, where the HVA ranges covered by the MMU notifiers are not acting on encrypted pages, we are ignoring MMU invalidation notifiers for SNP guests as part of the SNP host patches being posted upstream and instead relying on gmem own invalidation stuff to clean them up on a per-folio basis.

Thanks,
Ashish

oh, I have no question about that. This series only applies to SEV/SEV-ES type of VMs.

Quoted 6 lines

For SNP + guest_memfd, I don't see the implementation details, but I doubt you can ignore mmu_notifiers if the request does cover some encrypted memory in error cases or corner cases. Does the SNP enforce the usage of guest_memfd? How do we prevent exceptional cases? I am sure you guys already figured out the answers, so I don't plan to dig deeper until SNP host pages are accepted.

Quoted 3 lines

KVM will not allow SNP guests to map VMA-based memory as encrypted/private, full stop. Any invalidations initiated by mmu_notifiers will therefore apply only to shared memory.

Remind me. If I (as a SEV-SNP guest) flip the C-bit in my own x86 page table and write to some of the pages, am I generating encrypted dirty cache lines? I understand that the RMP table may say, hey it is "shared" but that's ok since I just don't need to pvalidate them, right?

Quoted 4 lines

That approach doesn't work for SEV/SEV-ES because KVM can't prevent the guest from accessing memory as encrypted, i.e. KVM needs the #NPF due to RMP violation to intercept attempts to convert a GFN from shared to private.

#15 lore raw
Quoted 23 lines · Mingwei Zhang
Quoted 22 lines · Sean Christopherson
Quoted 19 lines · Mingwei Zhang
Quoted 8 lines · Ashish

For SNP + gmem, where the HVA ranges covered by the MMU notifiers are not acting on encrypted pages, we are ignoring MMU invalidation notifiers for SNP guests as part of the SNP host patches being posted upstream and instead relying on gmem own invalidation stuff to clean them up on a per-folio basis.

Thanks,
Ashish

oh, I have no question about that. This series only applies to SEV/SEV-ES type of VMs.

For SNP + guest_memfd, I don't see the implementation details, but I doubt you can ignore mmu_notifiers if the request does cover some encrypted memory in error cases or corner cases. Does the SNP enforce the usage of guest_memfd? How do we prevent exceptional cases? I am sure you guys already figured out the answers, so I don't plan to dig deeper until SNP host pages are accepted.

Quoted 7 lines
Quoted 3 lines

KVM will not allow SNP guests to map VMA-based memory as encrypted/private, full stop. Any invalidations initiated by mmu_notifiers will therefore apply only to shared memory.

Remind me. If I (as a SEV-SNP guest) flip the C-bit in my own x86 page table and write to some of the pages, am I generating encrypted dirty cache lines?

No. See Table 15-39. "RMP Memory Access Checks" in the APM (my attempts to copy it to plain test failed miserably).

For accesses with effective C-bit == 0, the page must be Hypervisor-Owned. For effective C-bit == 1, the page must be fully assigned to the guest. Violation of those rules generates #VMEXIT.

A missing Validated attribute causes a #VC, but that case has lower priority than the about checks.

#16 lore raw
Quoted 35 lines · Sean Christopherson
Quoted 23 lines · Mingwei Zhang
Quoted 22 lines · Sean Christopherson
Quoted 19 lines · Mingwei Zhang
Quoted 8 lines · Ashish

For SNP + gmem, where the HVA ranges covered by the MMU notifiers are not acting on encrypted pages, we are ignoring MMU invalidation notifiers for SNP guests as part of the SNP host patches being posted upstream and instead relying on gmem own invalidation stuff to clean them up on a per-folio basis.

Thanks,
Ashish

oh, I have no question about that. This series only applies to SEV/SEV-ES type of VMs.

For SNP + guest_memfd, I don't see the implementation details, but I doubt you can ignore mmu_notifiers if the request does cover some encrypted memory in error cases or corner cases. Does the SNP enforce the usage of guest_memfd? How do we prevent exceptional cases? I am sure you guys already figured out the answers, so I don't plan to dig deeper until SNP host pages are accepted.

Quoted 7 lines
Quoted 3 lines

KVM will not allow SNP guests to map VMA-based memory as encrypted/private, full stop. Any invalidations initiated by mmu_notifiers will therefore apply only to shared memory.

Remind me. If I (as a SEV-SNP guest) flip the C-bit in my own x86 page table and write to some of the pages, am I generating encrypted dirty cache lines?

No. See Table 15-39. "RMP Memory Access Checks" in the APM (my attempts to copy it to plain test failed miserably).

Quoted 6 lines

For accesses with effective C-bit == 0, the page must be Hypervisor-Owned. For effective C-bit == 1, the page must be fully assigned to the guest. Violation of those rules generates #VMEXIT.

A missing Validated attribute causes a #VC, but that case has lower priority than the about checks.

Thank you. RMP check seems to be mandatory on all memory accesses when SNP is active. Hope that property remain an invarient regardless of any future optimization.

Thanks.
-Mingwei