Skip to contentSkip to conversation

Make ASIDs static for SVM

PATCH In review Yosry Ahmed 2 participants 14 messages Mar 13, 2025 – Mar 20, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Make ASIDs static for SVM

This series changes SVM to use a single ASID per-VM, instead of using dynamic generation-based ASIDs per-vCPU. Dynamic ASIDs were added for CPUs without FLUSHBYASID to avoid full TLB flushes, but as Sean said, FLUSHBYASID was added in 2010, and the case for this is no longer as strong [1].

Furthermore, having different ASIDs for different vCPUs is not required. ASIDs are local to physical CPUs. The only requirement is to make sure the ASID is flushed before a differnet vCPU runs on the same physical CPU (see below). Furthermore, SEV VMs have been using with a single ASID per-VM anyway (required for different reasons).

A new ASID is currently allocated in 3 cases:
(a) Once when the vCPU is initialized.
(b) When the vCPU moves to a new physical CPU.
(c) On TLB flushes when FLUSHBYASID is not available.

Case (a) is trivial, instead the ASID is allocated for VM creation. Case (b) is handled by flushing the ASID instead of assigning a new one. Case (c) is handled by doing a full TLB flush (i.e. TLB_CONTROL_FLUSH_ALL_ASID) instead of assinging a new ASID. This is a bit aggressive, but FLUSHBYASID is available in all modern CPUs.

The series is organized as follows:
- Patch 1 generalizes the VPID allocation code in VMX to be vendor-neutral, to reuse for SVM.
- Patches 2-3 do some refactoring and cleanups.
- Patches 4-5 address cases (b) and (c) above.
- Patch 6 moves to single ASID per-VM.
- Patch 7 performs some minimal unification between SVM and SEV code. More unification can be done. In particular, SEV can use the generalized kvm_tlb_tags to allocate ASIDs, and can stop tracking the ASID separately in struct kvm_sev_info. However, I didn't have enough SEV knowledge (or testability) to do this.

The performance impact does not seem to be that bad. To test this series, I ran 3 benchmarks in an SVM guest on a Milan machine:
- netperf
- cpuid_rate [2]
- A simple program doing mmap() and munmap() of 100M for 100 iterations, to trigger MMU syncs and TLB flushes when using the shadow MMU.

The benchmarks were ran with and without the patches for 5 iterations each, and also with and without NPT and FLUSBYASID to emulate old hardware. In all cases, there was either no difference or a 1-2% performance hit for the old hardware case. The performance hit could be larger for specific workloads, but niche performance-sensitive workloads should not be running on very old hardware.

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/kvm/[email protected]/

Yosry Ahmed (7):
KVM: VMX: Generalize VPID allocation to be vendor-neutral
KVM: SVM: Use cached local variable in init_vmcb()
KVM: SVM: Add helpers to set/clear ASID flush
KVM: SVM: Flush everything if FLUSHBYASID is not available
KVM: SVM: Flush the ASID when running on a new CPU
KVM: SVM: Use a single ASID per VM
KVM: SVM: Share more code between pre_sev_run() and pre_svm_run()

 arch/x86/include/asm/svm.h |  5 ---
 arch/x86/kvm/svm/nested.c  |  4 +-
 arch/x86/kvm/svm/sev.c     | 26 +++++-------
 arch/x86/kvm/svm/svm.c     | 87 ++++++++++++++++++++------------------
 arch/x86/kvm/svm/svm.h     | 28 ++++++++----
 arch/x86/kvm/vmx/nested.c  |  4 +-
 arch/x86/kvm/vmx/vmx.c     | 38 +++--------------
 arch/x86/kvm/vmx/vmx.h     |  4 +-
 arch/x86/kvm/x86.c         | 58 +++++++++++++++++++++++++
 arch/x86/kvm/x86.h         | 13 ++++++
 10 files changed, 161 insertions(+), 106 deletions(-)
Signature · 1 line
2.49.0.rc1.451.g8f38331e32-goog
#1 lore raw

PATCH 1/7 KVM: VMX: Generalize VPID allocation to be vendor-neutral

Generalize the VMX VPID allocation code and make move it to common code in preparation for sharing with SVM. Create a generic struct kvm_tlb_tags, representing a factory for VPIDs (or ASIDs later), and use one for VPIDs.

Most of the functionality remains the same, with the following differences:
- The enable_vpid checks are moved to the callers for allocate_vpid() and free_vpid(), as they are specific to VMX.
- The bitmap allocation is now dynamic (which will be required for SVM), so it is initialized and cleaned up in vmx_hardware_{setup/unsetup}().
- The range of valid TLB tags is expressed in terms of min/max instead of the number of tags to support SVM use cases.

---
 arch/x86/kvm/vmx/nested.c |  4 +--
 arch/x86/kvm/vmx/vmx.c    | 38 +++++--------------------
 arch/x86/kvm/vmx/vmx.h    |  4 +--
 arch/x86/kvm/x86.c        | 58 +++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/x86.h        | 13 +++++++++
 5 files changed, 82 insertions(+), 35 deletions(-)
arch/x86/kvm/vmx/nested.c +2 -2
index d06e50d9c0e79..b017bd2eb2382 100644
@@ -343,7 +343,7 @@ static void free_nested(struct kvm_vcpu *vcpu)
 	vmx->nested.vmxon = false;
 	vmx->nested.smm.vmxon = false;
 	vmx->nested.vmxon_ptr = INVALID_GPA;
-	free_vpid(vmx->nested.vpid02);
+	kvm_tlb_tags_free(&vmx_vpids, vmx->nested.vpid02);
 	vmx->nested.posted_intr_nv = -1;
 	vmx->nested.current_vmptr = INVALID_GPA;
 	if (enable_shadow_vmcs) {
@@ -5333,7 +5333,7 @@ static int enter_vmx_operation(struct kvm_vcpu *vcpu)
 		     HRTIMER_MODE_ABS_PINNED);
 	vmx->nested.preemption_timer.function = vmx_preemption_timer_fn;
 
-	vmx->nested.vpid02 = allocate_vpid();
+	vmx->nested.vpid02 = enable_vpid ? kvm_tlb_tags_alloc(&vmx_vpids) : 0;
 
 	vmx->nested.vmcs02_initialized = false;
 	vmx->nested.vmxon = true;
arch/x86/kvm/vmx/vmx.c +7 -31
index b70ed72c1783d..f7ce75842fa26 100644
@@ -496,8 +496,7 @@ DEFINE_PER_CPU(struct vmcs *, current_vmcs);
  */
 static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu);
 
-static DECLARE_BITMAP(vmx_vpid_bitmap, VMX_NR_VPIDS);
-static DEFINE_SPINLOCK(vmx_vpid_lock);
+struct kvm_tlb_tags vmx_vpids;
 
 struct vmcs_config vmcs_config __ro_after_init;
 struct vmx_capability vmx_capability __ro_after_init;
@@ -3972,31 +3971,6 @@ static void seg_setup(int seg)
 	vmcs_write32(sf->ar_bytes, ar);
 }
 
-int allocate_vpid(void)
-{
-	int vpid;
-
-	if (!enable_vpid)
-		return 0;
-	spin_lock(&vmx_vpid_lock);
-	vpid = find_first_zero_bit(vmx_vpid_bitmap, VMX_NR_VPIDS);
-	if (vpid < VMX_NR_VPIDS)
-		__set_bit(vpid, vmx_vpid_bitmap);
-	else
-		vpid = 0;
-	spin_unlock(&vmx_vpid_lock);
-	return vpid;
-}
-
-void free_vpid(int vpid)
-{
-	if (!enable_vpid || vpid == 0)
-		return;
-	spin_lock(&vmx_vpid_lock);
-	__clear_bit(vpid, vmx_vpid_bitmap);
-	spin_unlock(&vmx_vpid_lock);
-}
-
 static void vmx_msr_bitmap_l01_changed(struct vcpu_vmx *vmx)
 {
 	/*
@@ -7559,7 +7533,7 @@ void vmx_vcpu_free(struct kvm_vcpu *vcpu)
 
 	if (enable_pml)
 		vmx_destroy_pml_buffer(vmx);
-	free_vpid(vmx->vpid);
+	kvm_tlb_tags_free(&vmx_vpids, vmx->vpid);
 	nested_vmx_free_vcpu(vcpu);
 	free_loaded_vmcs(vmx->loaded_vmcs);
 	free_page((unsigned long)vmx->ve_info);
@@ -7578,7 +7552,7 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu)
 
 	err = -ENOMEM;
 
-	vmx->vpid = allocate_vpid();
+	vmx->vpid = enable_vpid ? kvm_tlb_tags_alloc(&vmx_vpids) : 0;
 
 	/*
 	 * If PML is turned on, failure on enabling PML just results in failure
@@ -7681,7 +7655,7 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu)
 free_pml:
 	vmx_destroy_pml_buffer(vmx);
 free_vpid:
-	free_vpid(vmx->vpid);
+	kvm_tlb_tags_free(&vmx_vpids, vmx->vpid);
 	return err;
 }
 
@@ -8373,6 +8347,7 @@ void vmx_hardware_unsetup(void)
 		nested_vmx_hardware_unsetup();
 
 	free_kvm_area();
+	kvm_tlb_tags_destroy(&vmx_vpids);
 }
 
 void vmx_vm_destroy(struct kvm *kvm)
@@ -8591,7 +8566,8 @@ __init int vmx_hardware_setup(void)
 	kvm_caps.has_bus_lock_exit = cpu_has_vmx_bus_lock_detection();
 	kvm_caps.has_notify_vmexit = cpu_has_notify_vmexit();
 
-	set_bit(0, vmx_vpid_bitmap); /* 0 is reserved for host */
+	/* VPID 0 is reserved for host, so min=1  */
+	kvm_tlb_tags_init(&vmx_vpids, 1, VMX_NR_VPIDS - 1);
 
 	if (enable_ept)
 		kvm_mmu_set_ept_masks(enable_ept_ad_bits,
arch/x86/kvm/vmx/vmx.h +2 -2
index 951e44dc9d0ea..9bece3ea63eaa 100644
@@ -376,10 +376,10 @@ struct kvm_vmx {
 	u64 *pid_table;
 };
 
+extern struct kvm_tlb_tags vmx_vpids;
+
 void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu,
 			struct loaded_vmcs *buddy);
-int allocate_vpid(void);
-void free_vpid(int vpid);
 void vmx_set_constant_host_state(struct vcpu_vmx *vmx);
 void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu);
 void vmx_set_host_fs_gs(struct vmcs_host_state *host, u16 fs_sel, u16 gs_sel,
arch/x86/kvm/x86.c +58 -0

64 lines not on this page · raw message

arch/x86/kvm/x86.h +13 -0
index 9dc32a4090761..9f84e933d189b 100644
@@ -652,4 +652,17 @@ int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, unsigned long nr,
 
 int kvm_emulate_hypercall(struct kvm_vcpu *vcpu);
 
+struct kvm_tlb_tags {
+	spinlock_t	lock;
+	unsigned long	*bitmap;
+	unsigned int	min;
+	unsigned int	max;
+};
+
+int kvm_tlb_tags_init(struct kvm_tlb_tags *tlb_tags, unsigned int min,
+		      unsigned int max);
+void kvm_tlb_tags_destroy(struct kvm_tlb_tags *tlb_tags);
+unsigned int kvm_tlb_tags_alloc(struct kvm_tlb_tags *tlb_tags);
+void kvm_tlb_tags_free(struct kvm_tlb_tags *tlb_tags, unsigned int tag);
+
 #endif
Signature · 1 line
2.49.0.rc1.451.g8f38331e32-goog
#2 lore raw

PATCH 2/7 KVM: SVM: Use cached local variable in init_vmcb()

In reply to Yosry Ahmed

svm->vmcb->control is already cached in the 'control' local variable, so use that.

---
 arch/x86/kvm/svm/svm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/svm.c +2 -2
index 8abeab91d329d..28a6d2c0f250f 100644
@@ -1367,12 +1367,12 @@ static void init_vmcb(struct kvm_vcpu *vcpu)
 		avic_init_vmcb(svm, vmcb);
 
 	if (vnmi)
-		svm->vmcb->control.int_ctl |= V_NMI_ENABLE_MASK;
+		control->int_ctl |= V_NMI_ENABLE_MASK;
 
 	if (vgif) {
 		svm_clr_intercept(svm, INTERCEPT_STGI);
 		svm_clr_intercept(svm, INTERCEPT_CLGI);
-		svm->vmcb->control.int_ctl |= V_GIF_ENABLE_MASK;
+		control->int_ctl |= V_GIF_ENABLE_MASK;
 	}
 
 	if (sev_guest(vcpu->kvm))
Signature · 1 line
2.49.0.rc1.451.g8f38331e32-goog
#3 lore raw

PATCH 3/7 KVM: SVM: Add helpers to set/clear ASID flush

In reply to Yosry Ahmed

Incoming changes will add more code paths that set tlb_ctl to TLB_CONTROL_FLUSH_ASID, and will eliminate the use of TLB_CONTROL_FLUSH_ALL_ASID except as fallback when FLUSHBYASID is not available. Introduce set/clear helpers to set tlb_ctl to TLB_CONTROL_FLUSH_ASID or TLB_CONTROL_DO_NOTHING.

Opportunistically move the TLB_CONTROL_* definitions to arch/x86/kvm/svm/svm.h as they are not used outside of arch/x86/kvm/svm/.

---
 arch/x86/include/asm/svm.h |  5 -----
 arch/x86/kvm/svm/nested.c  |  2 +-
 arch/x86/kvm/svm/sev.c     |  2 +-
 arch/x86/kvm/svm/svm.c     |  4 ++--
 arch/x86/kvm/svm/svm.h     | 15 +++++++++++++++
 5 files changed, 19 insertions(+), 9 deletions(-)
arch/x86/include/asm/svm.h +0 -5
index 9b7fa99ae9513..a97da63562eb3 100644
@@ -171,11 +171,6 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
 };
 
 
-#define TLB_CONTROL_DO_NOTHING 0
-#define TLB_CONTROL_FLUSH_ALL_ASID 1
-#define TLB_CONTROL_FLUSH_ASID 3
-#define TLB_CONTROL_FLUSH_ASID_LOCAL 7
-
 #define V_TPR_MASK 0x0f
 
 #define V_IRQ_SHIFT 8
arch/x86/kvm/svm/nested.c +1 -1
index 834b67672d50f..3bff948bc5752 100644
@@ -681,7 +681,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	/* Done at vmrun: asid.  */
 
 	/* Also overwritten later if necessary.  */
-	vmcb02->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+	svm_vmcb_clear_flush_asid(vmcb02);
 
 	/* nested_cr3.  */
 	if (nested_npt_enabled(svm))
arch/x86/kvm/svm/sev.c +1 -1
index 0bc708ee27887..b393674733969 100644
@@ -3479,7 +3479,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
 		return 0;
 
 	sd->sev_vmcbs[asid] = svm->vmcb;
-	svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+	svm_vmcb_set_flush_asid(svm->vmcb);
 	vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
 	return 0;
 }
arch/x86/kvm/svm/svm.c +2 -2
index 28a6d2c0f250f..8c90686a33f44 100644
@@ -4006,7 +4006,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)
 	 * VM-Exit (via kvm_mmu_reset_context()).
 	 */
 	if (static_cpu_has(X86_FEATURE_FLUSHBYASID))
-		svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+		svm_vmcb_set_flush_asid(svm->vmcb);
 	else
 		svm->current_vmcb->asid_generation--;
 }
@@ -4373,7 +4373,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
 		svm->nested.nested_run_pending = 0;
 	}
 
-	svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+	svm_vmcb_clear_flush_asid(svm->vmcb);
 	vmcb_mark_all_clean(svm->vmcb);
 
 	/* if exit due to PF check for async PF */
arch/x86/kvm/svm/svm.h +15 -0
index d4490eaed55dd..9fd5b249b9c19 100644
@@ -638,6 +638,21 @@ void svm_set_x2apic_msr_interception(struct vcpu_svm *svm, bool disable);
 void svm_complete_interrupt_delivery(struct kvm_vcpu *vcpu, int delivery_mode,
 				     int trig_mode, int vec);
 
+#define TLB_CONTROL_DO_NOTHING 0
+#define TLB_CONTROL_FLUSH_ALL_ASID 1
+#define TLB_CONTROL_FLUSH_ASID 3
+#define TLB_CONTROL_FLUSH_ASID_LOCAL 7
+
+static inline void svm_vmcb_set_flush_asid(struct vmcb *vmcb)
+{
+	vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+}
+
+static inline void svm_vmcb_clear_flush_asid(struct vmcb *vmcb)
+{
+	vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+}
+
 /* nested.c */
 
 #define NESTED_EXIT_HOST	0	/* Exit handled on host level */
Signature · 1 line
2.49.0.rc1.451.g8f38331e32-goog
#4 lore raw

PATCH 4/7 KVM: SVM: Flush everything if FLUSHBYASID is not available

In reply to Yosry Ahmed

Currently, if FLUSHBYASID is not available when performing a TLB flush, the fallback is decrementing the ASID generation to trigger allocating a new ASID. In preparation for using a static ASID per VM, just fallback to flushing everything if FLUSHBYASID is not available. This is probably worse from a performance perspective, but FLUSHBYASID has been around for ~15 years and it's not worth carrying the complexity.

The fallback logic is moved within svm_vmcb_set_flush_asid(), as more callers will be added and will need the fallback as well.

---
 arch/x86/kvm/svm/svm.c | 5 +----
 arch/x86/kvm/svm/svm.h | 5 ++++-
 2 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/kvm/svm/svm.c +1 -4
index 8c90686a33f44..e5064fbefb822 100644
@@ -4005,10 +4005,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)
 	 * unconditionally does a TLB flush on both nested VM-Enter and nested
 	 * VM-Exit (via kvm_mmu_reset_context()).
 	 */
-	if (static_cpu_has(X86_FEATURE_FLUSHBYASID))
-		svm_vmcb_set_flush_asid(svm->vmcb);
-	else
-		svm->current_vmcb->asid_generation--;
+	svm_vmcb_set_flush_asid(svm->vmcb);
 }
 
 static void svm_flush_tlb_current(struct kvm_vcpu *vcpu)
arch/x86/kvm/svm/svm.h +4 -1
index 9fd5b249b9c19..0f6426809e1b9 100644
@@ -645,7 +645,10 @@ void svm_complete_interrupt_delivery(struct kvm_vcpu *vcpu, int delivery_mode,
 
 static inline void svm_vmcb_set_flush_asid(struct vmcb *vmcb)
 {
-	vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+	if (static_cpu_has(X86_FEATURE_FLUSHBYASID))
+		vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+	else
+		vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
 }
 
 static inline void svm_vmcb_clear_flush_asid(struct vmcb *vmcb)
Signature · 1 line
2.49.0.rc1.451.g8f38331e32-goog
#5 lore raw

PATCH 5/7 KVM: SVM: Flush the ASID when running on a new CPU

In reply to Yosry Ahmed

Currently, when a vCPU is migrated to a new physical CPU, the ASID generation is reset to trigger allocating a new ASID. In preparation for using a static ASID per VM, just flush the ASID in this case (falling back to flushing everything if FLUSBYASID is not available).

---
 arch/x86/kvm/svm/svm.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/kvm/svm/svm.c +4 -4
index e5064fbefb822..b8a3fc81fc9c8 100644
@@ -3626,12 +3626,12 @@ static int pre_svm_run(struct kvm_vcpu *vcpu)
 	struct vcpu_svm *svm = to_svm(vcpu);
 
 	/*
-	 * If the previous vmrun of the vmcb occurred on a different physical
-	 * cpu, then mark the vmcb dirty and assign a new asid.  Hardware's
-	 * vmcb clean bits are per logical CPU, as are KVM's asid assignments.
+	 * If the previous VMRUN of the VMCB occurred on a different physical
+	 * CPU, then mark the VMCB dirty and flush the ASID.  Hardware's
+	 * VMCB clean bits are per logical CPU, as are KVM's ASID assignments.
 	 */
 	if (unlikely(svm->current_vmcb->cpu != vcpu->cpu)) {
-		svm->current_vmcb->asid_generation = 0;
+		svm_vmcb_set_flush_asid(svm->vmcb);
 		vmcb_mark_all_dirty(svm->vmcb);
 		svm->current_vmcb->cpu = vcpu->cpu;
         }
Signature · 1 line
2.49.0.rc1.451.g8f38331e32-goog
#6 lore raw

PATCH 6/7 KVM: SVM: Use a single ASID per VM

In reply to Yosry Ahmed

The ASID generation and dynamic ASID allocation logic is now only used by initialization the generation to 0 to trigger a new ASID allocation per-vCPU on the first VMRUN, so the ASID is more-or-less static per-vCPU.

Moreover, having a unique ASID per-vCPU is not required. ASIDs are local to each physical CPU, and the ASID is flushed when a vCPU is migrated to a new physical CPU anyway. SEV VMs have been using a single ASID per VM already for other reasons.

Use a static ASID per VM and drop the dynamic ASID allocation logic. The ASID is allocated during vCPU reset (SEV allocates its own ASID), and the ASID is always flushed on first use in case it was used by another VM previously.

On VMRUN, WARN if the ASID in the VMCB does not match the VM's ASID, and update it accordingly. Also, flush the ASID on every VMRUN if the VM failed to allocate a unique ASID. This would probably wreck performance if it happens, but it should be an edge case as most AMD CPUs have over 32k ASIDs.

---
 arch/x86/kvm/svm/nested.c |  2 ++
 arch/x86/kvm/svm/sev.c    |  7 +++--
 arch/x86/kvm/svm/svm.c    | 60 +++++++++++++++++++++------------------
 arch/x86/kvm/svm/svm.h    |  8 +-----
 4 files changed, 40 insertions(+), 37 deletions(-)
arch/x86/kvm/svm/nested.c +2 -0
index 3bff948bc5752..d6a07644c8734 100644
@@ -643,6 +643,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	struct kvm_vcpu *vcpu = &svm->vcpu;
 	struct vmcb *vmcb01 = svm->vmcb01.ptr;
 	struct vmcb *vmcb02 = svm->nested.vmcb02.ptr;
+	struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);
 	u32 pause_count12;
 	u32 pause_thresh12;
 
@@ -677,6 +678,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	vmcb02->control.nested_ctl = vmcb01->control.nested_ctl;
 	vmcb02->control.iopm_base_pa = vmcb01->control.iopm_base_pa;
 	vmcb02->control.msrpm_base_pa = vmcb01->control.msrpm_base_pa;
+	vmcb02->control.asid = kvm_svm->asid;
 
 	/* Done at vmrun: asid.  */
 
arch/x86/kvm/svm/sev.c +5 -2
index b393674733969..1ee04d6b9356b 100644
@@ -3465,8 +3465,10 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
 	if (sev_es_guest(kvm) && !VALID_PAGE(svm->vmcb->control.vmsa_pa))
 		return -EINVAL;
 
-	/* Assign the asid allocated with this SEV guest */
-	svm->asid = asid;
+	if (WARN_ON_ONCE(svm->vmcb->control.asid != asid)) {
+		svm->vmcb->control.asid = asid;
+		vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
+	}
 
 	/*
 	 * Flush guest TLB:
@@ -4509,6 +4511,7 @@ static void sev_es_init_vmcb(struct vcpu_svm *svm)
 void sev_init_vmcb(struct vcpu_svm *svm)
 {
 	svm->vmcb->control.nested_ctl |= SVM_NESTED_CTL_SEV_ENABLE;
+	svm->vmcb->control.asid = sev_get_asid(svm->vcpu.kvm);
 	clr_exception_intercept(svm, UD_VECTOR);
 
 	/*
arch/x86/kvm/svm/svm.c +32 -28

151 lines not on this page · raw message

arch/x86/kvm/svm/svm.h +1 -7
index 0f6426809e1b9..4c6664ba4048d 100644
@@ -119,6 +119,7 @@ struct kvm_svm {
 
 	/* Struct members for AVIC */
 	u32 avic_vm_id;
+	unsigned int asid;
 	struct page *avic_logical_id_table_page;
 	struct page *avic_physical_id_table_page;
 	struct hlist_node hnode;
@@ -132,7 +133,6 @@ struct kvm_vmcb_info {
 	struct vmcb *ptr;
 	unsigned long pa;
 	int cpu;
-	uint64_t asid_generation;
 };
 
 struct vmcb_save_area_cached {
@@ -247,7 +247,6 @@ struct vcpu_svm {
 	struct vmcb *vmcb;
 	struct kvm_vmcb_info vmcb01;
 	struct kvm_vmcb_info *current_vmcb;
-	u32 asid;
 	u32 sysenter_esp_hi;
 	u32 sysenter_eip_hi;
 	uint64_t tsc_aux;
@@ -330,11 +329,6 @@ struct vcpu_svm {
 };
 
 struct svm_cpu_data {
-	u64 asid_generation;
-	u32 max_asid;
-	u32 next_asid;
-	u32 min_asid;
-
 	struct vmcb *save_area;
 	unsigned long save_area_pa;
 
Signature · 1 line
2.49.0.rc1.451.g8f38331e32-goog
#7 lore raw

Re: [PATCH 6/7] KVM: SVM: Use a single ASID per VM

arch/x86/kvm/svm/svm.h struct kvm_svm { · line 122

Quoted 9 lines
arch/x86/kvm/svm/svm.h +1 -0
index 0f6426809e1b9..4c6664ba4048d 100644
@@ -119,6 +119,7 @@ struct kvm_svm {
 
 	/* Struct members for AVIC */
 	u32 avic_vm_id;
+	unsigned int asid;

I couldn't have put this in a worse place if I was trying.

#8 lore raw

Re: [PATCH 6/7] KVM: SVM: Use a single ASID per VM

In reply to Yosry Ahmed
Quoted 24 lines · Yosry Ahmed

The ASID generation and dynamic ASID allocation logic is now only used by initialization the generation to 0 to trigger a new ASID allocation per-vCPU on the first VMRUN, so the ASID is more-or-less static per-vCPU.

Moreover, having a unique ASID per-vCPU is not required. ASIDs are local to each physical CPU, and the ASID is flushed when a vCPU is migrated to a new physical CPU anyway. SEV VMs have been using a single ASID per VM already for other reasons.

Use a static ASID per VM and drop the dynamic ASID allocation logic. The ASID is allocated during vCPU reset (SEV allocates its own ASID), and the ASID is always flushed on first use in case it was used by another VM previously.

On VMRUN, WARN if the ASID in the VMCB does not match the VM's ASID, and update it accordingly. Also, flush the ASID on every VMRUN if the VM failed to allocate a unique ASID. This would probably wreck performance if it happens, but it should be an edge case as most AMD CPUs have over 32k ASIDs.

---

[..]

Quoted 10 lines
arch/x86/kvm/svm/svm.c +1 -1
@@ -3622,7 +3613,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 
 static int pre_svm_run(struct kvm_vcpu *vcpu)
 {
-	struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu);
+	struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);
 	struct vcpu_svm *svm = to_svm(vcpu);
 
 	/*

arch/x86/kvm/svm/svm.c static int pre_svm_run(struct kvm_vcpu *vcpu) · line 3635

Quoted 9 lines
arch/x86/kvm/svm/svm.c +3 -3
@@ -3639,9 +3630,15 @@ static int pre_svm_run(struct kvm_vcpu *vcpu)
 	if (sev_guest(vcpu->kvm))
 		return pre_sev_run(svm, vcpu->cpu);
 
-	/* FIXME: handle wraparound of asid_generation */
-	if (svm->current_vmcb->asid_generation != sd->asid_generation)
-		new_asid(svm, sd);
+	/* Flush the ASID on every VMRUN if kvm_svm->asid allocation failed */
+	if (unlikely(!kvm_svm->asid))
+		svm_vmcb_set_flush_asid(svm->vmcb);

This is wrong. I thought we can handle ASID allocation failures by just reusing ASID=0 and flushing it on every VMRUN, but using ASID=0 is illegal according to the APM. Also, in this case we also need to flush the ASID on every VM-exit, which I failed to do here.

There are two ways to handle running out of ASIDs:

(a) Failing to create the VM. This will impose a virtual limit on the number of VMs that can be run concurrently. The number of ASIDs was quite high on the CPUs I checked (2^15 IIRC), so it's probably not an issue, but I am not sure if this is considered breaking userspace.

(b) Designating a specific ASID value as the "fallback ASID". This value would be used by any VMs created after running out of ASIDs, and we flush it on every VMRUN, similar to what I am trying to do here for ASID=0.

Any thoughts on which way we should take? (a) is simpler if we can get away with it and all AMD CPUs have a sufficiently large number of ASIDs.

#9 lore raw

Re: [PATCH 6/7] KVM: SVM: Use a single ASID per VM

Quoted 52 lines · Yosry Ahmed
Quoted 24 lines · Yosry Ahmed

The ASID generation and dynamic ASID allocation logic is now only used by initialization the generation to 0 to trigger a new ASID allocation per-vCPU on the first VMRUN, so the ASID is more-or-less static per-vCPU.

Moreover, having a unique ASID per-vCPU is not required. ASIDs are local to each physical CPU, and the ASID is flushed when a vCPU is migrated to a new physical CPU anyway. SEV VMs have been using a single ASID per VM already for other reasons.

Use a static ASID per VM and drop the dynamic ASID allocation logic. The ASID is allocated during vCPU reset (SEV allocates its own ASID), and the ASID is always flushed on first use in case it was used by another VM previously.

On VMRUN, WARN if the ASID in the VMCB does not match the VM's ASID, and update it accordingly. Also, flush the ASID on every VMRUN if the VM failed to allocate a unique ASID. This would probably wreck performance if it happens, but it should be an edge case as most AMD CPUs have over 32k ASIDs.

---

[..]

Quoted 10 lines
arch/x86/kvm/svm/svm.c +1 -1
@@ -3622,7 +3613,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 
 static int pre_svm_run(struct kvm_vcpu *vcpu)
 {
-     struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu);
+     struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);
      struct vcpu_svm *svm = to_svm(vcpu);
 
      /*

arch/x86/kvm/svm/svm.c static int pre_svm_run(struct kvm_vcpu *vcpu) · line 3635

Quoted 9 lines
arch/x86/kvm/svm/svm.c +3 -3
@@ -3639,9 +3630,15 @@ static int pre_svm_run(struct kvm_vcpu *vcpu)
      if (sev_guest(vcpu->kvm))
              return pre_sev_run(svm, vcpu->cpu);
 
-     /* FIXME: handle wraparound of asid_generation */
-     if (svm->current_vmcb->asid_generation != sd->asid_generation)
-             new_asid(svm, sd);
+     /* Flush the ASID on every VMRUN if kvm_svm->asid allocation failed */
+     if (unlikely(!kvm_svm->asid))
+             svm_vmcb_set_flush_asid(svm->vmcb);

This is wrong. I thought we can handle ASID allocation failures by just reusing ASID=0 and flushing it on every VMRUN, but using ASID=0 is illegal according to the APM. Also, in this case we also need to flush the ASID on every VM-exit, which I failed to do here.

Quoted 6 lines

There are two ways to handle running out of ASIDs:

(a) Failing to create the VM. This will impose a virtual limit on the number of VMs that can be run concurrently. The number of ASIDs was quite high on the CPUs I checked (2^15 IIRC), so it's probably not an issue, but I am not sure if this is considered breaking userspace.

I'm pretty sure AMD had only 6 bits of ASID through at least Family 12H. At some point, VMCB ASID bits must have become decoupled from physical TLB tag bits. 15 TLB tag bits is inconceivable!

Quoted 8 lines

(b) Designating a specific ASID value as the "fallback ASID". This value would be used by any VMs created after running out of ASIDs, and we flush it on every VMRUN, similar to what I am trying to do here for ASID=0.

Any thoughts on which way we should take? (a) is simpler if we can get away with it and all AMD CPUs have a sufficiently large number of ASIDs.

#10 lore raw

Re: [PATCH 6/7] KVM: SVM: Use a single ASID per VM

Quoted 55 lines · Jim Mattson
Quoted 54 lines · Yosry Ahmed
Quoted 24 lines · Yosry Ahmed

The ASID generation and dynamic ASID allocation logic is now only used by initialization the generation to 0 to trigger a new ASID allocation per-vCPU on the first VMRUN, so the ASID is more-or-less static per-vCPU.

Moreover, having a unique ASID per-vCPU is not required. ASIDs are local to each physical CPU, and the ASID is flushed when a vCPU is migrated to a new physical CPU anyway. SEV VMs have been using a single ASID per VM already for other reasons.

Use a static ASID per VM and drop the dynamic ASID allocation logic. The ASID is allocated during vCPU reset (SEV allocates its own ASID), and the ASID is always flushed on first use in case it was used by another VM previously.

On VMRUN, WARN if the ASID in the VMCB does not match the VM's ASID, and update it accordingly. Also, flush the ASID on every VMRUN if the VM failed to allocate a unique ASID. This would probably wreck performance if it happens, but it should be an edge case as most AMD CPUs have over 32k ASIDs.

---

[..]

Quoted 10 lines
arch/x86/kvm/svm/svm.c +1 -1
@@ -3622,7 +3613,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 
 static int pre_svm_run(struct kvm_vcpu *vcpu)
 {
-     struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu);
+     struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);
      struct vcpu_svm *svm = to_svm(vcpu);
 
      /*

arch/x86/kvm/svm/svm.c static int pre_svm_run(struct kvm_vcpu *vcpu) · line 3635

Quoted 9 lines
arch/x86/kvm/svm/svm.c +3 -3
@@ -3639,9 +3630,15 @@ static int pre_svm_run(struct kvm_vcpu *vcpu)
      if (sev_guest(vcpu->kvm))
              return pre_sev_run(svm, vcpu->cpu);
 
-     /* FIXME: handle wraparound of asid_generation */
-     if (svm->current_vmcb->asid_generation != sd->asid_generation)
-             new_asid(svm, sd);
+     /* Flush the ASID on every VMRUN if kvm_svm->asid allocation failed */
+     if (unlikely(!kvm_svm->asid))
+             svm_vmcb_set_flush_asid(svm->vmcb);

This is wrong. I thought we can handle ASID allocation failures by just reusing ASID=0 and flushing it on every VMRUN, but using ASID=0 is illegal according to the APM. Also, in this case we also need to flush the ASID on every VM-exit, which I failed to do here.

There are two ways to handle running out of ASIDs:

Quoted 8 lines
Quoted 4 lines

(a) Failing to create the VM. This will impose a virtual limit on the number of VMs that can be run concurrently. The number of ASIDs was quite high on the CPUs I checked (2^15 IIRC), so it's probably not an issue, but I am not sure if this is considered breaking userspace.

I'm pretty sure AMD had only 6 bits of ASID through at least Family 12H. At some point, VMCB ASID bits must have become decoupled from physical TLB tag bits. 15 TLB tag bits is inconceivable!

I checked on Rome, Milan, Genoa, and Turin. For leaf 8000000a, the value of EBX is 00008000 for all of them. That's 32767 ASIDs for VMs (1 for host, and ignoring SEV). I don't have access to older hardware to check, but 6 bits would be 63 ASIDs for VMs. I imagine such a constraint may be less acceptable, so having a single fallback ASID may be the way to go?

Quoted 9 lines
Quoted 9 lines

(b) Designating a specific ASID value as the "fallback ASID". This value would be used by any VMs created after running out of ASIDs, and we flush it on every VMRUN, similar to what I am trying to do here for ASID=0.

Any thoughts on which way we should take? (a) is simpler if we can get away with it and all AMD CPUs have a sufficiently large number of ASIDs.

#11 lore raw

PATCH 7/7 KVM: SVM: Share more code between pre_sev_run() and pre_svm_run()

In reply to Yosry Ahmed

pre_svm_run() and pre_sev_run() now do some redundant work, and the control flow is not super clear. Specifically:
- Both functions check if the ASID in the VMCB is the expected one.
- Both functions check if the vCPU moved to a different physical CPU.
- Both functions issue an ASID TLB flush if needed.

Pass the ASID and whether or not SEV requires a TLB flush from pre_sev_run() to pre_svm_run(), and use the logic there instead. pre_sev_run() now only performs SEV-specific checks.

Note that pre_sev_run() used svm->vcpu.arch.last_vmentry_cpu to check if the vCPU moved to a different physical CPU, while pre_svm_run uses svm->current_vmcb->cpu. The former tracks the CPU per vCPU, while the latter tracks it per VMCB. For SEV, they both should be equivalent since there is a single VMCB per-vCPU (nested is not supported).

---
 arch/x86/kvm/svm/sev.c | 27 ++++++++++-----------------
 arch/x86/kvm/svm/svm.c | 10 ++++++----
 arch/x86/kvm/svm/svm.h |  2 +-
 3 files changed, 17 insertions(+), 22 deletions(-)
arch/x86/kvm/svm/sev.c +10 -17
index 1ee04d6b9356b..607139757f8ff 100644
@@ -3451,11 +3451,11 @@ void sev_es_unmap_ghcb(struct vcpu_svm *svm)
 	svm->sev_es.ghcb = NULL;
 }
 
-int pre_sev_run(struct vcpu_svm *svm, int cpu)
+int pre_sev_run(struct vcpu_svm *svm, unsigned int *asid, bool *need_flush)
 {
+	int cpu = svm->vcpu.cpu;
 	struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, cpu);
 	struct kvm *kvm = svm->vcpu.kvm;
-	unsigned int asid = sev_get_asid(kvm);
 
 	/*
 	 * Reject KVM_RUN if userspace attempts to run the vCPU with an invalid
@@ -3465,24 +3465,17 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
 	if (sev_es_guest(kvm) && !VALID_PAGE(svm->vmcb->control.vmsa_pa))
 		return -EINVAL;
 
-	if (WARN_ON_ONCE(svm->vmcb->control.asid != asid)) {
-		svm->vmcb->control.asid = asid;
-		vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
-	}
-
 	/*
-	 * Flush guest TLB:
-	 *
-	 * 1) when different VMCB for the same ASID is to be run on the same host CPU.
-	 * 2) or this VMCB was executed on different host CPU in previous VMRUNs.
+	 * Flush the guest TLB when a difference VMCB for the same ASID is to be
+	 * run on the same host CPU. The caller will also flush the TLB if the
+	 * VMCB was executed on a different host CPU in previous VMRUNs.
 	 */
-	if (sd->sev_vmcbs[asid] == svm->vmcb &&
-	    svm->vcpu.arch.last_vmentry_cpu == cpu)
-		return 0;
+	*asid = sev_get_asid(kvm);
+	if (sd->sev_vmcbs[*asid] != svm->vmcb) {
+		sd->sev_vmcbs[*asid] = svm->vmcb;
+		*need_flush = true;
+	}
 
-	sd->sev_vmcbs[asid] = svm->vmcb;
-	svm_vmcb_set_flush_asid(svm->vmcb);
-	vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
 	return 0;
 }
 
arch/x86/kvm/svm/svm.c +6 -4
index c5e2733fb856d..6b338d84e7b93 100644
@@ -3615,21 +3615,23 @@ static int pre_svm_run(struct kvm_vcpu *vcpu)
 {
 	struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);
 	struct vcpu_svm *svm = to_svm(vcpu);
+	unsigned int asid = kvm_svm->asid;
+	bool sev_need_flush = false;
+
+	if (sev_guest(vcpu->kvm) && pre_sev_run(svm, &asid, &sev_need_flush))
+		return -1;
 
 	/*
 	 * If the previous VMRUN of the VMCB occurred on a different physical
 	 * CPU, then mark the VMCB dirty and flush the ASID.  Hardware's
 	 * VMCB clean bits are per logical CPU, as are KVM's ASID assignments.
 	 */
-	if (unlikely(svm->current_vmcb->cpu != vcpu->cpu)) {
+	if (unlikely(sev_need_flush || svm->current_vmcb->cpu != vcpu->cpu)) {
 		svm_vmcb_set_flush_asid(svm->vmcb);
 		vmcb_mark_all_dirty(svm->vmcb);
 		svm->current_vmcb->cpu = vcpu->cpu;
         }
 
-	if (sev_guest(vcpu->kvm))
-		return pre_sev_run(svm, vcpu->cpu);
-
 	/* Flush the ASID on every VMRUN if kvm_svm->asid allocation failed */
 	if (unlikely(!kvm_svm->asid))
 		svm_vmcb_set_flush_asid(svm->vmcb);
arch/x86/kvm/svm/svm.h +1 -1
index 4c6664ba4048d..f25e99c79d07d 100644
@@ -754,7 +754,7 @@ void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu);
 
 /* sev.c */
 
-int pre_sev_run(struct vcpu_svm *svm, int cpu);
+int pre_sev_run(struct vcpu_svm *svm, unsigned int *asid, bool *need_flush);
 void sev_init_vmcb(struct vcpu_svm *svm);
 void sev_vcpu_after_set_cpuid(struct vcpu_svm *svm);
 int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in);
Signature · 1 line
2.49.0.rc1.451.g8f38331e32-goog
#12 lore raw

Re: [PATCH 7/7] KVM: SVM: Share more code between pre_sev_run() and pre_svm_run()

Quoted 95 lines · Yosry Ahmed

pre_svm_run() and pre_sev_run() now do some redundant work, and the control flow is not super clear. Specifically:
- Both functions check if the ASID in the VMCB is the expected one.
- Both functions check if the vCPU moved to a different physical CPU.
- Both functions issue an ASID TLB flush if needed.

Pass the ASID and whether or not SEV requires a TLB flush from pre_sev_run() to pre_svm_run(), and use the logic there instead. pre_sev_run() now only performs SEV-specific checks.

Note that pre_sev_run() used svm->vcpu.arch.last_vmentry_cpu to check if the vCPU moved to a different physical CPU, while pre_svm_run uses svm->current_vmcb->cpu. The former tracks the CPU per vCPU, while the latter tracks it per VMCB. For SEV, they both should be equivalent since there is a single VMCB per-vCPU (nested is not supported).

---
 arch/x86/kvm/svm/sev.c | 27 ++++++++++-----------------
 arch/x86/kvm/svm/svm.c | 10 ++++++----
 arch/x86/kvm/svm/svm.h |  2 +-
 3 files changed, 17 insertions(+), 22 deletions(-)
arch/x86/kvm/svm/sev.c +10 -17
index 1ee04d6b9356b..607139757f8ff 100644
@@ -3451,11 +3451,11 @@ void sev_es_unmap_ghcb(struct vcpu_svm *svm)
 	svm->sev_es.ghcb = NULL;
 }
 
-int pre_sev_run(struct vcpu_svm *svm, int cpu)
+int pre_sev_run(struct vcpu_svm *svm, unsigned int *asid, bool *need_flush)
 {
+	int cpu = svm->vcpu.cpu;
 	struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, cpu);
 	struct kvm *kvm = svm->vcpu.kvm;
-	unsigned int asid = sev_get_asid(kvm);
 
 	/*
 	 * Reject KVM_RUN if userspace attempts to run the vCPU with an invalid
@@ -3465,24 +3465,17 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
 	if (sev_es_guest(kvm) && !VALID_PAGE(svm->vmcb->control.vmsa_pa))
 		return -EINVAL;
 
-	if (WARN_ON_ONCE(svm->vmcb->control.asid != asid)) {
-		svm->vmcb->control.asid = asid;
-		vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
-	}
-
 	/*
-	 * Flush guest TLB:
-	 *
-	 * 1) when different VMCB for the same ASID is to be run on the same host CPU.
-	 * 2) or this VMCB was executed on different host CPU in previous VMRUNs.
+	 * Flush the guest TLB when a difference VMCB for the same ASID is to be
+	 * run on the same host CPU. The caller will also flush the TLB if the
+	 * VMCB was executed on a different host CPU in previous VMRUNs.
 	 */
-	if (sd->sev_vmcbs[asid] == svm->vmcb &&
-	    svm->vcpu.arch.last_vmentry_cpu == cpu)
-		return 0;
+	*asid = sev_get_asid(kvm);
+	if (sd->sev_vmcbs[*asid] != svm->vmcb) {
+		sd->sev_vmcbs[*asid] = svm->vmcb;
+		*need_flush = true;
+	}
 
-	sd->sev_vmcbs[asid] = svm->vmcb;
-	svm_vmcb_set_flush_asid(svm->vmcb);
-	vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
 	return 0;
 }
 
arch/x86/kvm/svm/svm.c +6 -1
index c5e2733fb856d..6b338d84e7b93 100644
@@ -3615,21 +3615,23 @@ static int pre_svm_run(struct kvm_vcpu *vcpu)
 {
 	struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);
 	struct vcpu_svm *svm = to_svm(vcpu);
+	unsigned int asid = kvm_svm->asid;
+	bool sev_need_flush = false;
+
+	if (sev_guest(vcpu->kvm) && pre_sev_run(svm, &asid, &sev_need_flush))
+		return -1;
 
 	/*
 	 * If the previous VMRUN of the VMCB occurred on a different physical
 	 * CPU, then mark the VMCB dirty and flush the ASID.  Hardware's
 	 * VMCB clean bits are per logical CPU, as are KVM's ASID assignments.
 	 */
-	if (unlikely(svm->current_vmcb->cpu != vcpu->cpu)) {
+	if (unlikely(sev_need_flush || svm->current_vmcb->cpu != vcpu->cpu)) {

arch/x86/kvm/svm/svm.c static int pre_svm_run(struct kvm_vcpu *vcpu) · line 3636

Quoted 10 lines
arch/x86/kvm/svm/svm.c +0 -3
 		svm_vmcb_set_flush_asid(svm->vmcb);
 		vmcb_mark_all_dirty(svm->vmcb);
 		svm->current_vmcb->cpu = vcpu->cpu;
         }
 
-	if (sev_guest(vcpu->kvm))
-		return pre_sev_run(svm, vcpu->cpu);
-
 	/* Flush the ASID on every VMRUN if kvm_svm->asid allocation failed */
 	if (unlikely(!kvm_svm->asid))

This should now check 'asid' instead of 'kvm_svm->asid'.

Same for the WARN below.

Quoted 17 lines
arch/x86/kvm/svm/svm.c +0 -0
 		svm_vmcb_set_flush_asid(svm->vmcb);
arch/x86/kvm/svm/svm.h +1 -1
index 4c6664ba4048d..f25e99c79d07d 100644
@@ -754,7 +754,7 @@ void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu);
 
 /* sev.c */
 
-int pre_sev_run(struct vcpu_svm *svm, int cpu);
+int pre_sev_run(struct vcpu_svm *svm, unsigned int *asid, bool *need_flush);
 void sev_init_vmcb(struct vcpu_svm *svm);
 void sev_vcpu_after_set_cpuid(struct vcpu_svm *svm);
 int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in);
Signature · 1 line
2.49.0.rc1.451.g8f38331e32-goog
#13 lore raw
In reply to Yosry Ahmed
Quoted 42 lines · Yosry Ahmed

This series changes SVM to use a single ASID per-VM, instead of using dynamic generation-based ASIDs per-vCPU. Dynamic ASIDs were added for CPUs without FLUSHBYASID to avoid full TLB flushes, but as Sean said, FLUSHBYASID was added in 2010, and the case for this is no longer as strong [1].

Furthermore, having different ASIDs for different vCPUs is not required. ASIDs are local to physical CPUs. The only requirement is to make sure the ASID is flushed before a differnet vCPU runs on the same physical CPU (see below). Furthermore, SEV VMs have been using with a single ASID per-VM anyway (required for different reasons).

A new ASID is currently allocated in 3 cases:
(a) Once when the vCPU is initialized.
(b) When the vCPU moves to a new physical CPU.
(c) On TLB flushes when FLUSHBYASID is not available.

Case (a) is trivial, instead the ASID is allocated for VM creation. Case (b) is handled by flushing the ASID instead of assigning a new one. Case (c) is handled by doing a full TLB flush (i.e. TLB_CONTROL_FLUSH_ALL_ASID) instead of assinging a new ASID. This is a bit aggressive, but FLUSHBYASID is available in all modern CPUs.

The series is organized as follows:
- Patch 1 generalizes the VPID allocation code in VMX to be vendor-neutral, to reuse for SVM.
- Patches 2-3 do some refactoring and cleanups.
- Patches 4-5 address cases (b) and (c) above.
- Patch 6 moves to single ASID per-VM.
- Patch 7 performs some minimal unification between SVM and SEV code. More unification can be done. In particular, SEV can use the generalized kvm_tlb_tags to allocate ASIDs, and can stop tracking the ASID separately in struct kvm_sev_info. However, I didn't have enough SEV knowledge (or testability) to do this.

The performance impact does not seem to be that bad. To test this series, I ran 3 benchmarks in an SVM guest on a Milan machine:
- netperf
- cpuid_rate [2]
- A simple program doing mmap() and munmap() of 100M for 100 iterations, to trigger MMU syncs and TLB flushes when using the shadow MMU.

Quoted 6 lines

The benchmarks were ran with and without the patches for 5 iterations each, and also with and without NPT and FLUSBYASID to emulate old hardware. In all cases, there was either no difference or a 1-2% performance hit for the old hardware case. The performance hit could be larger for specific workloads, but niche performance-sensitive workloads should not be running on very old hardware.

This series has several bugs. It allows a VM to use ASID 0 if we run out of space (which is not allowed by VMRUN), and it does not handle the case where multiple vCPUs of the same VM with the same ASID run on the same CPU (handled by SEV through svm_cpu_data.sev_vmcbs).

I also think it's useful to see how the nSVM TLB flushing looks like on top of this. So please hold off on reviewing this series, I will send a new combined series.

Quoted 28 lines

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/kvm/[email protected]/

Yosry Ahmed (7):
KVM: VMX: Generalize VPID allocation to be vendor-neutral
KVM: SVM: Use cached local variable in init_vmcb()
KVM: SVM: Add helpers to set/clear ASID flush
KVM: SVM: Flush everything if FLUSHBYASID is not available
KVM: SVM: Flush the ASID when running on a new CPU
KVM: SVM: Use a single ASID per VM
KVM: SVM: Share more code between pre_sev_run() and pre_svm_run()

 arch/x86/include/asm/svm.h |  5 ---
 arch/x86/kvm/svm/nested.c  |  4 +-
 arch/x86/kvm/svm/sev.c     | 26 +++++-------
 arch/x86/kvm/svm/svm.c     | 87 ++++++++++++++++++++------------------
 arch/x86/kvm/svm/svm.h     | 28 ++++++++----
 arch/x86/kvm/vmx/nested.c  |  4 +-
 arch/x86/kvm/vmx/vmx.c     | 38 +++--------------
 arch/x86/kvm/vmx/vmx.h     |  4 +-
 arch/x86/kvm/x86.c         | 58 +++++++++++++++++++++++++
 arch/x86/kvm/x86.h         | 13 ++++++
 10 files changed, 161 insertions(+), 106 deletions(-)
Signature · 1 line
2.49.0.rc1.451.g8f38331e32-goog