Tweak TLB flushing when VMX is running on Hyper-V

Use Hyper-V's HvCallFlushGuestPhysicalAddressSpace for local TLB flushes. This makes any KVM_REQ_TLB_FLUSH_CURRENT (such as on root alloc) visible to all CPUs which means we no longer need to do a KVM_REQ_TLB_FLUSH on CPU migration.

The goal is to avoid invept-global in KVM_REQ_TLB_FLUSH. Hyper-V uses a shadow page table for the nested hypervisor (KVM) and has to invalidate all EPT roots when invept-global is issued. This has a performance impact on all nested VMs. KVM issues KVM_REQ_TLB_FLUSH on CPU migration, and under load the performance hit causes vCPUs to use up more of their slice of CPU time, leading to more CPU migrations. This has a snowball effect and causes CPU usage spikes.

By issuing the hypercall we are now guaranteed that any root modification that requires a local TLB flush becomes visible to all CPUs. The same hypercall is already used in kvm_arch_flush_remote_tlbs and kvm_arch_flush_remote_tlbs_range. The KVM expectation is that roots are flushed locally on alloc and we achieve consistency on migration by flushing all roots - the new behavior of achieving consistency on alloc on Hyper-V is a superset of the expected guarantees. This makes the KVM_REQ_TLB_FLUSH on CPU migration no longer necessary on Hyper-V.

Coincidentally - we now match the behavior of SVM on Hyper-V.

Signed-off-by Jeremi Piotrowski <[email protected]>

---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/vmx/vmx.c          | 20 +++++++++++++++++---
 arch/x86/kvm/vmx/vmx_onhyperv.h |  6 ++++++
 arch/x86/kvm/x86.c              |  3 +++
 4 files changed, 27 insertions(+), 3 deletions(-)

arch/x86/include/asm/kvm_host.h +1 -0

index b4a391929cdb..d3acab19f425 100644
@@ -1077,6 +1077,7 @@ struct kvm_vcpu_arch {
 
 #if IS_ENABLED(CONFIG_HYPERV)
 	hpa_t hv_root_tdp;
+	bool hv_vmx_use_flush_guest_mapping;
 #endif
 };
 

arch/x86/kvm/vmx/vmx.c +0 -0

index 4953846cb30d..f537e0df56fc 100644

arch/x86/kvm/vmx/vmx.c +4 -2

+		 * flushes to be visible across all CPUs so no need to do again
+		 * on migration.
 		 */
-		kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
+		if (!vmx_hv_use_flush_guest_mapping(vcpu))
+			kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
 
 		/*
 		 * Linux uses per-cpu TSS and GDT, so set these when switching

@@ -3243,11 +3247,21 @@ void vmx_flush_tlb_current(struct kvm_vcpu *vcpu)
 	if (!VALID_PAGE(root_hpa))
 		return;
 
-	if (enable_ept)

arch/x86/kvm/vmx/vmx.c +2 -1

 		ept_sync_context(construct_eptp(vcpu, root_hpa,
 						mmu->root_role.level));
-	else
+	} else {
 		vpid_sync_context(vmx_get_current_vpid(vcpu));
+	}
 }
 
 void vmx_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t addr)

arch/x86/kvm/vmx/vmx_onhyperv.h +6 -0

index cdf8cbb69209..a5c64c90e49e 100644
@@ -119,6 +119,11 @@ static inline void evmcs_load(u64 phys_addr)
 }
 
 void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf);
+
+static inline bool vmx_hv_use_flush_guest_mapping(struct kvm_vcpu *vcpu)
+{
+	return vcpu->arch.hv_vmx_use_flush_guest_mapping;
+}
 #else /* !IS_ENABLED(CONFIG_HYPERV) */
 static __always_inline bool kvm_is_using_evmcs(void) { return false; }
 static __always_inline void evmcs_write64(unsigned long field, u64 value) {}

@@ -128,6 +133,7 @@ static __always_inline u64 evmcs_read64(unsigned long field) { return 0; }
 static __always_inline u32 evmcs_read32(unsigned long field) { return 0; }
 static __always_inline u16 evmcs_read16(unsigned long field) { return 0; }
 static inline void evmcs_load(u64 phys_addr) {}
+static inline bool vmx_hv_use_flush_guest_mapping(struct kvm_vcpu *vcpu) { return false; }
 #endif /* IS_ENABLED(CONFIG_HYPERV) */
 
 #endif /* __ARCH_X86_KVM_VMX_ONHYPERV_H__ */

arch/x86/kvm/x86.c +3 -0

index b58a74c1722d..cbde795096a6 100644
@@ -25,6 +25,7 @@
 #include "tss.h"
 #include "kvm_cache_regs.h"
 #include "kvm_emulate.h"
+#include "kvm_onhyperv.h"
 #include "mmu/page_track.h"
 #include "x86.h"
 #include "cpuid.h"

@@ -12390,6 +12391,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
 #if IS_ENABLED(CONFIG_HYPERV)
 	vcpu->arch.hv_root_tdp = INVALID_PAGE;
+	vcpu->arch.hv_vmx_use_flush_guest_mapping =
+		(kvm_x86_ops.flush_remote_tlbs == hv_flush_remote_tlbs);
 #endif
 
 	r = kvm_x86_call(vcpu_create)(vcpu);

Quoted 13 lines · Jeremi Piotrowski

Use Hyper-V's HvCallFlushGuestPhysicalAddressSpace for local TLB flushes. This makes any KVM_REQ_TLB_FLUSH_CURRENT (such as on root alloc) visible to all CPUs which means we no longer need to do a KVM_REQ_TLB_FLUSH on CPU migration.

The goal is to avoid invept-global in KVM_REQ_TLB_FLUSH. Hyper-V uses a shadow page table for the nested hypervisor (KVM) and has to invalidate all EPT roots when invept-global is issued. This has a performance impact on all nested VMs. KVM issues KVM_REQ_TLB_FLUSH on CPU migration, and under load the performance hit causes vCPUs to use up more of their slice of CPU time, leading to more CPU migrations. This has a snowball effect and causes CPU usage spikes.

Quoted 8 lines

By issuing the hypercall we are now guaranteed that any root modification that requires a local TLB flush becomes visible to all CPUs. The same hypercall is already used in kvm_arch_flush_remote_tlbs and kvm_arch_flush_remote_tlbs_range. The KVM expectation is that roots are flushed locally on alloc and we achieve consistency on migration by flushing all roots - the new behavior of achieving consistency on alloc on Hyper-V is a superset of the expected guarantees. This makes the KVM_REQ_TLB_FLUSH on CPU migration no longer necessary on Hyper-V.

Sounds reasonable overall, my only concern (not sure if valid or not) is that using the hypercall for local flushes is going to be more expensive than invept-context we do today and thus while the performance is improved for the scenario when vCPUs are migrating a lot, we will take a hit in other cases.

Quoted 28 lines

Coincidentally - we now match the behavior of SVM on Hyper-V.

Signed-off-by Jeremi Piotrowski <[email protected]>

---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/vmx/vmx.c          | 20 +++++++++++++++++---
 arch/x86/kvm/vmx/vmx_onhyperv.h |  6 ++++++
 arch/x86/kvm/x86.c              |  3 +++
 4 files changed, 27 insertions(+), 3 deletions(-)

arch/x86/include/asm/kvm_host.h +1 -0

index b4a391929cdb..d3acab19f425 100644
@@ -1077,6 +1077,7 @@ struct kvm_vcpu_arch {
 
 #if IS_ENABLED(CONFIG_HYPERV)
 	hpa_t hv_root_tdp;
+	bool hv_vmx_use_flush_guest_mapping;
 #endif
 };
 

arch/x86/kvm/vmx/vmx.c +0 -0

index 4953846cb30d..f537e0df56fc 100644

arch/x86/kvm/vmx/vmx.c void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu) · line 1488

Quoted 4 lines

arch/x86/kvm/vmx/vmx.c +1 -0

@@ -1485,8 +1485,12 @@ void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu)
 		/*
 		 * Flush all EPTP/VPID contexts, the new pCPU may have stale
 		 * TLB entries from its previous association with the vCPU.
+		 * Unless we are running on Hyper-V where we promotes local TLB

s,promotes,promote, or, as Sean doesn't like pronouns,

"... where local TLB flushes are promoted ..."

Quoted 13 lines

arch/x86/kvm/vmx/vmx.c +4 -1

+		 * flushes to be visible across all CPUs so no need to do again
+		 * on migration.
 		 */
-		kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
+		if (!vmx_hv_use_flush_guest_mapping(vcpu))
+			kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
 
 		/*
 		 * Linux uses per-cpu TSS and GDT, so set these when switching

@@ -3243,11 +3247,21 @@ void vmx_flush_tlb_current(struct kvm_vcpu *vcpu)
 	if (!VALID_PAGE(root_hpa))
 		return;
 

arch/x86/kvm/vmx/vmx.c void vmx_flush_tlb_current(struct kvm_vcpu *vcpu) · line 3259

Quoted 11 lines

arch/x86/kvm/vmx/vmx.c +10 -1

-	if (enable_ept)
+	if (enable_ept) {
+		/*
+		 * hyperv_flush_guest_mapping() has the semantics of
+		 * invept-single across all pCPUs. This makes root
+		 * modifications consistent across pCPUs, so an invept-global
+		 * on migration is no longer required.
+		 */
+		if (vmx_hv_use_flush_guest_mapping(vcpu))
+			return (void)WARN_ON_ONCE(hyperv_flush_guest_mapping(root_hpa));
+

HvCallFlushGuestPhysicalAddressSpace sounds like a heavy operation as it affects all processors. Is there any visible perfomance impact of this change when there are no migrations (e.g. with vCPU pinning)? Or do we believe that Hyper-V actually handles invept-context the exact same way?

Quoted 24 lines · Vitaly Kuznetsov

Quoted 21 lines · Jeremi Piotrowski

Use Hyper-V's HvCallFlushGuestPhysicalAddressSpace for local TLB flushes. This makes any KVM_REQ_TLB_FLUSH_CURRENT (such as on root alloc) visible to all CPUs which means we no longer need to do a KVM_REQ_TLB_FLUSH on CPU migration.

The goal is to avoid invept-global in KVM_REQ_TLB_FLUSH. Hyper-V uses a shadow page table for the nested hypervisor (KVM) and has to invalidate all EPT roots when invept-global is issued. This has a performance impact on all nested VMs. KVM issues KVM_REQ_TLB_FLUSH on CPU migration, and under load the performance hit causes vCPUs to use up more of their slice of CPU time, leading to more CPU migrations. This has a snowball effect and causes CPU usage spikes.

By issuing the hypercall we are now guaranteed that any root modification that requires a local TLB flush becomes visible to all CPUs. The same hypercall is already used in kvm_arch_flush_remote_tlbs and kvm_arch_flush_remote_tlbs_range. The KVM expectation is that roots are flushed locally on alloc and we achieve consistency on migration by flushing all roots - the new behavior of achieving consistency on alloc on Hyper-V is a superset of the expected guarantees. This makes the KVM_REQ_TLB_FLUSH on CPU migration no longer necessary on Hyper-V.

Quoted 6 lines

Sounds reasonable overall, my only concern (not sure if valid or not) is that using the hypercall for local flushes is going to be more expensive than invept-context we do today and thus while the performance is improved for the scenario when vCPUs are migrating a lot, we will take a hit in other cases.

Quoted 19 lines

arch/x86/kvm/vmx/vmx.c void vmx_flush_tlb_current(struct kvm_vcpu *vcpu) · line 3259

Quoted 13 lines

arch/x86/kvm/vmx/vmx.c +10 -1

 		return;
 
-	if (enable_ept)
+	if (enable_ept) {
+		/*
+		 * hyperv_flush_guest_mapping() has the semantics of
+		 * invept-single across all pCPUs. This makes root
+		 * modifications consistent across pCPUs, so an invept-global
+		 * on migration is no longer required.
+		 */
+		if (vmx_hv_use_flush_guest_mapping(vcpu))
+			return (void)WARN_ON_ONCE(hyperv_flush_guest_mapping(root_hpa));
+

HvCallFlushGuestPhysicalAddressSpace sounds like a heavy operation as it affects all processors. Is there any visible perfomance impact of this change when there are no migrations (e.g. with vCPU pinning)? Or do we believe that Hyper-V actually handles invept-context the exact same way?

I'm going to have to do some more investigation to answer that - do you have an idea of a workload that would be sensitive to tlb flushes that I could compare this on?

In terms of cost, Hyper-V needs to invalidate the VMs shadow page table for a root and do the tlb flush. The first part is CPU intensive but is the same in both cases (hypercall and invept-single). The tlb flush part will require a bit more work for the hypercall as it needs to happen on all cores, and the tlb will now be empty for that root.

My assumption is that these local tlb flushes are rather rare as they will only happen when:
- new root is allocated
- we need to switch to a special root

So not very frequent post vm boot (with or without pinning). And the effect of the tlb being empty for that root on other CPUs should be a neutral, as users of the root would have performed the same local flush at a later point in time (when using it).

All the other mmu updates use kvm_flush_remote_tlbs* which already go through the hypercall.

Quoted 3 lines · Jeremi Piotrowski

Quoted 2 lines · Vitaly Kuznetsov

Jeremi Piotrowski <[email protected]> writes:

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Use Hyper-V's HvCallFlushGuestPhysicalAddressSpace for local TLB flushes. This makes any KVM_REQ_TLB_FLUSH_CURRENT (such as on root alloc) visible to all CPUs which means we no longer need to do a KVM_REQ_TLB_FLUSH on CPU migration.

The goal is to avoid invept-global in KVM_REQ_TLB_FLUSH. Hyper-V uses a shadow page table for the nested hypervisor (KVM) and has to invalidate all EPT roots when invept-global is issued.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

This has a performance impact on all nested VMs. KVM issues KVM_REQ_TLB_FLUSH on CPU migration, and under load the performance hit causes vCPUs to use up more of their slice of CPU time, leading to more CPU migrations. This has a snowball effect and causes CPU usage spikes.

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

By issuing the hypercall we are now guaranteed that any root modification that requires a local TLB flush becomes visible to all CPUs. The same hypercall is already used in kvm_arch_flush_remote_tlbs and kvm_arch_flush_remote_tlbs_range. The KVM expectation is that roots are flushed locally on alloc and we achieve consistency on migration by flushing all roots - the new behavior of achieving consistency on alloc on Hyper-V is a superset of the expected guarantees. This makes the KVM_REQ_TLB_FLUSH on CPU migration no longer necessary on Hyper-V.

Quoted 7 lines

Quoted 6 lines

Sounds reasonable overall, my only concern (not sure if valid or not) is that using the hypercall for local flushes is going to be more expensive than invept-context we do today and thus while the performance is improved for the scenario when vCPUs are migrating a lot, we will take a hit in other cases.

Sorry for delayed reply!

....

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

return;

Quoted 7 lines

Quoted 7 lines

Review

Quoted 7 lines

(quoted diff) +6 -1

-	if (enable_ept)
+	if (enable_ept) {
+		/*
+		 * hyperv_flush_guest_mapping() has the semantics of
+		 * invept-single across all pCPUs. This makes root
+		 * modifications consistent across pCPUs, so an invept-global
+		 * on migration is no longer required.

Quoted 15 lines

Quoted 5 lines

HvCallFlushGuestPhysicalAddressSpace sounds like a heavy operation as it affects all processors. Is there any visible perfomance impact of this change when there are no migrations (e.g. with vCPU pinning)? Or do we believe that Hyper-V actually handles invept-context the exact same way?

I'm going to have to do some more investigation to answer that - do you have an idea of a workload that would be sensitive to tlb flushes that I could compare this on?

In terms of cost, Hyper-V needs to invalidate the VMs shadow page table for a root and do the tlb flush. The first part is CPU intensive but is the same in both cases (hypercall and invept-single). The tlb flush part will require a bit more work for the hypercall as it needs to happen on all cores, and the tlb will now be empty for that root.

Quoted 5 lines

My assumption is that these local tlb flushes are rather rare as they will only happen when:
- new root is allocated
- we need to switch to a special root

KVM's MMU is an amazing maze so I'd appreciate if someone more knowledgeble corrects me;t my understanding is that we call *_flush_tlb_current() from two places:

kvm_mmu_load() and this covers the two cases above. These should not be common under normal circumstances but can be frequent in some special cases, e.g. when running a nested setup. Given that we're already running on top of Hyper-V, this means 3+ level nesting which I don't believe anyone really cares about.

It'll take more work than the below, e.g. to have VMX's construct_eptp() pull the level and A/D bits from kvm_mmu_page (vendor code can get at the kvm_mmu_page with root_to_sp()), but for the core concept/skeleton, I think this is it?

arch/x86/kvm/mmu/mmu.c +19 -0

index 6e838cb6c9e1..298130445182 100644
@@ -3839,6 +3839,37 @@ void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu)
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_free_guest_mode_roots);
 
+struct kvm_tlb_flush_root {
+       struct kvm *kvm;
+       hpa_t root;
+};
+
+static void kvm_flush_tlb_root(void *__data)
+{
+       struct kvm_tlb_flush_root *data = __data;
+
+       kvm_x86_call(flush_tlb_root)(data->kvm, data->root);
+}
+
+void kvm_mmu_flush_all_tlbs_root(struct kvm *kvm, struct kvm_mmu_page *root)
+{
+       struct kvm_tlb_flush_root data = {
+               .kvm = kvm,
+               .root = __pa(root->spt),
+       };
+

arch/x86/kvm/mmu/mmu.c +4 -10

+}
+
 static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant,
                            u8 level)
 {

@@ -3852,7 +3883,8 @@ static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant,
        WARN_ON_ONCE(role.direct && role.has_4_byte_gpte);
 
        sp = kvm_mmu_get_shadow_page(vcpu, gfn, role);
-       ++sp->root_count;
+       if (!sp->root_count++)
+               kvm_mmu_flush_all_tlbs_root(vcpu->kvm, sp);
 
        return __pa(sp->spt);
 }

@@ -5961,15 +5993,6 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu)
        kvm_mmu_sync_roots(vcpu);
 
        kvm_mmu_load_pgd(vcpu);
-
-       /*
-        * Flush any TLB entries for the new root, the provenance of the root
-        * is unknown.  Even if KVM ensures there are no stale TLB entries
-        * for a freed root, in theory another hypervisor could have left
-        * stale entries.  Flushing on alloc also allows KVM to skip the TLB
-        * flush when freeing a root (see kvm_tdp_mmu_put_root()).
-        */
-       kvm_x86_call(flush_tlb_current)(vcpu);
 out:
        return r;
 }

arch/x86/kvm/mmu/mmu_internal.h +2 -0

index 65f3c89d7c5d..3cbf0d612f5e 100644
@@ -167,6 +167,8 @@ static inline bool is_mirror_sp(const struct kvm_mmu_page *sp)
        return sp->role.is_mirror;
 }
 
+void kvm_mmu_flush_all_tlbs_root(struct kvm *kvm, struct kvm_mmu_page *root);
+
 static inline void kvm_mmu_alloc_external_spt(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)
 {
        /*

arch/x86/kvm/mmu/tdp_mmu.c +1 -0

index 7f3d7229b2c1..3ff36d09b4fa 100644
@@ -302,6 +302,7 @@ void kvm_tdp_mmu_alloc_root(struct kvm_vcpu *vcpu, bool mirror)
         */
        refcount_set(&root->tdp_mmu_root_count, 2);
        list_add_rcu(&root->link, &kvm->arch.tdp_mmu_roots);
+       kvm_mmu_flush_all_tlbs_root(vcpu->kvm, root);
 
 out_spin_unlock:
        spin_unlock(&kvm->arch.tdp_mmu_pages_lock);

Quoted 31 lines · Sean Christopherson

It'll take more work than the below, e.g. to have VMX's construct_eptp() pull the level and A/D bits from kvm_mmu_page (vendor code can get at the kvm_mmu_page with root_to_sp()), but for the core concept/skeleton, I think this is it?

arch/x86/kvm/mmu/mmu.c +19 -0

index 6e838cb6c9e1..298130445182 100644
@@ -3839,6 +3839,37 @@ void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu)
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_free_guest_mode_roots);
 
+struct kvm_tlb_flush_root {
+       struct kvm *kvm;
+       hpa_t root;
+};
+
+static void kvm_flush_tlb_root(void *__data)
+{
+       struct kvm_tlb_flush_root *data = __data;
+
+       kvm_x86_call(flush_tlb_root)(data->kvm, data->root);
+}
+
+void kvm_mmu_flush_all_tlbs_root(struct kvm *kvm, struct kvm_mmu_page *root)
+{
+       struct kvm_tlb_flush_root data = {
+               .kvm = kvm,
+               .root = __pa(root->spt),
+       };
+

arch/x86/kvm/mmu/mmu.c void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) · line 3870

Quoted 10 lines

arch/x86/kvm/mmu/mmu.c +10 -0

+       /*
+        * Flush any TLB entries for the new root, the provenance of the root
+        * is unknown.  Even if KVM ensures there are no stale TLB entries
+        * for a freed root, in theory another hypervisor could have left
+        * stale entries.  Flushing on alloc also allows KVM to skip the TLB
+        * flush when freeing a root (see kvm_tdp_mmu_put_root()), and flushing
+        * TLBs on all CPUs allows KVM to elide TLB flushes when a vCPU is
+        * migrated to a different pCPU.
+        */
+       on_each_cpu(kvm_flush_tlb_root, &data, 1);

Would it make sense to complement this with e.g. a CPU mask tracking all the pCPUs where the VM has ever been seen running (+ a flush when a new one is added to it)?

I'm worried about the potential performance impact for a case when a huge host is running a lot of small VMs in 'partitioning' mode (i.e. when all vCPUs are pinned). Additionally, this may have a negative impact on RT use-cases where each unnecessary interruption can be seen problematic.

Quoted 47 lines · Vitaly Kuznetsov

Quoted 31 lines · Sean Christopherson

It'll take more work than the below, e.g. to have VMX's construct_eptp() pull the level and A/D bits from kvm_mmu_page (vendor code can get at the kvm_mmu_page with root_to_sp()), but for the core concept/skeleton, I think this is it?

arch/x86/kvm/mmu/mmu.c +19 -0

index 6e838cb6c9e1..298130445182 100644
@@ -3839,6 +3839,37 @@ void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu)
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_free_guest_mode_roots);
 
+struct kvm_tlb_flush_root {
+       struct kvm *kvm;
+       hpa_t root;
+};
+
+static void kvm_flush_tlb_root(void *__data)
+{
+       struct kvm_tlb_flush_root *data = __data;
+
+       kvm_x86_call(flush_tlb_root)(data->kvm, data->root);
+}
+
+void kvm_mmu_flush_all_tlbs_root(struct kvm *kvm, struct kvm_mmu_page *root)
+{
+       struct kvm_tlb_flush_root data = {
+               .kvm = kvm,
+               .root = __pa(root->spt),
+       };
+

arch/x86/kvm/mmu/mmu.c void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) · line 3870

Quoted 10 lines

arch/x86/kvm/mmu/mmu.c +10 -0

+       /*
+        * Flush any TLB entries for the new root, the provenance of the root
+        * is unknown.  Even if KVM ensures there are no stale TLB entries
+        * for a freed root, in theory another hypervisor could have left
+        * stale entries.  Flushing on alloc also allows KVM to skip the TLB
+        * flush when freeing a root (see kvm_tdp_mmu_put_root()), and flushing
+        * TLBs on all CPUs allows KVM to elide TLB flushes when a vCPU is
+        * migrated to a different pCPU.
+        */
+       on_each_cpu(kvm_flush_tlb_root, &data, 1);

Would it make sense to complement this with e.g. a CPU mask tracking all the pCPUs where the VM has ever been seen running (+ a flush when a new one is added to it)?

Quoted 5 lines

I'm worried about the potential performance impact for a case when a huge host is running a lot of small VMs in 'partitioning' mode (i.e. when all vCPUs are pinned). Additionally, this may have a negative impact on RT use-cases where each unnecessary interruption can be seen problematic.

Oof, right. And it's not even a VM-to-VM noisy neighbor problem, e.g. a few vCPUs using nested TDP could generate a lot of noist IRQs through a VM. Hrm.

So I think the basic idea is so flawed/garbage that even enhancing it with per-VM pCPU tracking wouldn't work. I do think you've got the right idea with a pCPU mask though, but instead of using a mask to scope IPIs, use it to elide TLB flushes.

With the TDP MMU, KVM can have at most 6 non-nested roots active at any given time:
SMM vs. non-SMM, 4-level vs. 5-level, L1 vs. L2. Allocating a cpumask for each TDP MMU root seems reasonable. Then on task migration, instead of doing a global INVEPT, only INVEPT the current and prev_roots (because getting a new root will trigger a flush in kvm_mmu_load()), and skip INVEPT on TDP MMU roots if the pCPU has already done a flush for the root.

Or we could do the optimized tracking for all roots. x86 supports at most 8192 CPUs, which means 1KiB per root. That doesn't seem at all painful given that each shadow pages consumes 4KiB...

Quoted 30 lines · Sean Christopherson

Quoted 28 lines · Vitaly Kuznetsov

Quoted 7 lines · Sean Christopherson

(quoted diff) +7 -0

+void kvm_mmu_flush_all_tlbs_root(struct kvm *kvm, struct kvm_mmu_page *root)
+{
+       struct kvm_tlb_flush_root data = {
+               .kvm = kvm,
+               .root = __pa(root->spt),
+       };
+

Review

Quoted 10 lines

(quoted diff) +10 -0

+       /*
+        * Flush any TLB entries for the new root, the provenance of the root
+        * is unknown.  Even if KVM ensures there are no stale TLB entries
+        * for a freed root, in theory another hypervisor could have left
+        * stale entries.  Flushing on alloc also allows KVM to skip the TLB
+        * flush when freeing a root (see kvm_tdp_mmu_put_root()), and flushing
+        * TLBs on all CPUs allows KVM to elide TLB flushes when a vCPU is
+        * migrated to a different pCPU.
+        */
+       on_each_cpu(kvm_flush_tlb_root, &data, 1);

Would it make sense to complement this with e.g. a CPU mask tracking all the pCPUs where the VM has ever been seen running (+ a flush when a new one is added to it)?

I'm worried about the potential performance impact for a case when a huge host is running a lot of small VMs in 'partitioning' mode (i.e. when all vCPUs are pinned). Additionally, this may have a negative impact on RT use-cases where each unnecessary interruption can be seen problematic.

Quoted 6 lines

Oof, right. And it's not even a VM-to-VM noisy neighbor problem, e.g. a few vCPUs using nested TDP could generate a lot of noist IRQs through a VM. Hrm.

So I think the basic idea is so flawed/garbage that even enhancing it with per-VM pCPU tracking wouldn't work. I do think you've got the right idea with a pCPU mask though, but instead of using a mask to scope IPIs, use it to elide TLB flushes.

Sorry for the delay in replying, I've been sidetracked a bit.

I like this idea more, not special casing the TLB flushing approach per hypervisor is preferable.

Quoted 7 lines

With the TDP MMU, KVM can have at most 6 non-nested roots active at any given time:
SMM vs. non-SMM, 4-level vs. 5-level, L1 vs. L2. Allocating a cpumask for each TDP MMU root seems reasonable. Then on task migration, instead of doing a global INVEPT, only INVEPT the current and prev_roots (because getting a new root will trigger a flush in kvm_mmu_load()), and skip INVEPT on TDP MMU roots if the pCPU has already done a flush for the root.

Just to make sure I follow: current+prev_roots do you mean literally those (i.e. cached prev roots) or all roots on kvm->arch.tdp_mmu_roots?

So this would mean: on pCPU migration, check if current mmu has is_tdp_mmu_active() and then perform the INVEPT-single over roots instead of INVEPT-global. Otherwise stick to the KVM_REQ_TLB_FLUSH.

Would there need to be a check for is_guest_mode(), or that the switch is coming from the vmx/nested.c? I suppose not because nested doesn't seem to use TDP MMU.

Quoted 3 lines

Or we could do the optimized tracking for all roots. x86 supports at most 8192 CPUs, which means 1KiB per root. That doesn't seem at all painful given that each shadow pages consumes 4KiB...

Similar question here: which all roots would need to be tracked+flushed for shadow paging? pae_roots?

Quoted 3 lines · Jeremi Piotrowski

Quoted 2 lines · Sean Christopherson

Quoted 1 line · Vitaly Kuznetsov

Sean Christopherson <[email protected]> writes:

Quoted 9 lines

Quoted 7 lines

Oof, right. And it's not even a VM-to-VM noisy neighbor problem, e.g. a few vCPUs using nested TDP could generate a lot of noist IRQs through a VM. Hrm.

So I think the basic idea is so flawed/garbage that even enhancing it with per-VM pCPU tracking wouldn't work. I do think you've got the right idea with a pCPU mask though, but instead of using a mask to scope IPIs, use it to elide TLB flushes.

Sorry for the delay in replying, I've been sidetracked a bit.

No worries, I guarantee my delays will make your delays pale in comparison :-D

Quoted 11 lines

I like this idea more, not special casing the TLB flushing approach per hypervisor is preferable.

Quoted 7 lines

With the TDP MMU, KVM can have at most 6 non-nested roots active at any given time:
SMM vs. non-SMM, 4-level vs. 5-level, L1 vs. L2. Allocating a cpumask for each TDP MMU root seems reasonable. Then on task migration, instead of doing a global INVEPT, only INVEPT the current and prev_roots (because getting a new root will trigger a flush in kvm_mmu_load()), and skip INVEPT on TDP MMU roots if the pCPU has already done a flush for the root.

Quoted 2 lines

Just to make sure I follow: current+prev_roots do you mean literally those (i.e. cached prev roots) or all roots on kvm->arch.tdp_mmu_roots?

The former, i.e. "root" and all "prev_roots" entries in a kvm_mmu structure.

Quoted 3 lines

So this would mean: on pCPU migration, check if current mmu has is_tdp_mmu_active() and then perform the INVEPT-single over roots instead of INVEPT-global. Otherwise stick to the KVM_REQ_TLB_FLUSH.

No, KVM would still need to ensure shadow roots are flushed as well, because KVM doesn't flush TLBs when switching to a previous root (see fast_pgd_switch()). More at the bottom.

Quoted 3 lines

Would there need to be a check for is_guest_mode(), or that the switch is coming from the vmx/nested.c? I suppose not because nested doesn't seem to use TDP MMU.

Nested can use the TDP MMU, though there's practically no code in KVM that explicitly deals with this scenario. If L1 is using legacy shadow paging, i.e. is NOT using EPT/NPT, then KVM will use the TDP MMU to map L2 (with kvm_mmu_page_role.guest_mode=1 to differentiate from the L1 TDP MMU).

Quoted 6 lines

Quoted 3 lines

Or we could do the optimized tracking for all roots. x86 supports at most 8192 CPUs, which means 1KiB per root. That doesn't seem at all painful given that each shadow pages consumes 4KiB...

Similar question here: which all roots would need to be tracked+flushed for shadow paging? pae_roots?

Same general answer, "root" and all "prev_roots" entries. KVM uses up to two "struct kvm_mmu" instances to actually map memory into the guest: root_mmu and guest_mmu. The third instance, nested_mmu, is used to model gva->gpa translations for L2, i.e. is used only to walk L2 stage-1 page tables, and is never used to map memory into the guest, i.e. can't have entries in hardware TLBs.

The basic gist is to add a cpumask in each root, and then elide TLB flushes on pCPU migration if KVM has flushed the root at least once. Patch 5/5 in the attached set of patches provides a *very* rough sketch. Hopefully its enough to convey the core idea.

Patches 1-4 compile, but are otherwise untested. I'll post patches 1-3 as a small series once their tested, as those cleanups are worth doing irrespective of any optimizations we make to pCPU migration.

On Tue, Aug 05, 2025 at 04:42:46PM -0700, Sean Christopherson wrote:
I started working on extending patch 5, wanted to post it here to make sure I'm on the right track.

It works in testing so far and shows promising performance - it gets rid of all the pathological cases I saw before.

I haven't checked whether I broke SVM yet, and I need figure out a way to always keep the cpumask "offstack" so that we don't blow up every struct kvm_mmu_page instance with an inline cpumask - it needs to stay optional.

I also came across kvm_mmu_is_dummy_root(), that check is included in root_to_sp(). Can you think of any other checks that we might need to handle?

arch/x86/kvm/mmu/mmu.c +1 -0

+
 	return __pa(sp->spt);
 }

arch/x86/kvm/vmx/vmx.c +5 -3

+		return;
+
 	vmx_flush_tlb_ept_root(root_hpa);
 }
 
-static void vmx_flush_ept_on_pcpu_migration(struct kvm_mmu *mmu)
+static void vmx_flush_ept_on_pcpu_migration(struct kvm_mmu *mmu, int cpu)
 {
 	int i;
 
-	__vmx_flush_ept_on_pcpu_migration(mmu->root.hpa);
+	__vmx_flush_ept_on_pcpu_migration(mmu->root.hpa, cpu);
 
 	for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++)
-		__vmx_flush_ept_on_pcpu_migration(mmu->prev_roots[i].hpa);
+		__vmx_flush_ept_on_pcpu_migration(mmu->prev_roots[i].hpa, cpu);
 }
 
 void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu)

arch/x86/kvm/vmx/x86_ops.h +1 -0

index b4596f651232..4406d53e6ebe 100644
@@ -84,6 +84,7 @@ void vmx_flush_tlb_all(struct kvm_vcpu *vcpu);
 void vmx_flush_tlb_current(struct kvm_vcpu *vcpu);
 void vmx_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t addr);
 void vmx_flush_tlb_guest(struct kvm_vcpu *vcpu);
+void vmx_alloc_root_cpu_mask(struct kvm_mmu_page *root);
 void vmx_set_interrupt_shadow(struct kvm_vcpu *vcpu, int mask);
 u32 vmx_get_interrupt_shadow(struct kvm_vcpu *vcpu);
 void vmx_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall);

Signature · 1 line

2.39.5