Skip to contentSkip to conversation

RISC-V: KVM: Avoid synchronous IPIs on VMID rollover

PATCH Superseded Wenlong Li 2 participants 2 messages Sep 10

Subsystems (3)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Wenlong Li

author
#0 lore raw

PATCH 1/1 RISC-V: KVM: Avoid synchronous IPIs on VMID rollover

The current VMID rollover path synchronously invokes HFENCE.GVMA on all online CPUs using on_each_cpu_mask(). This sends IPIs to remote CPUs and causes a vCPU running in guest mode on a targeted CPU to exit. The rollover path then waits for all CPUs to complete their local TLB flushes before VMID allocation can continue.

Make VMID rollover lazy to remove this cross-CPU synchronization.

Track active and reserved software VMIDs for each possible CPU. On rollover, preserve hardware VMIDs that may still have stale translations, clear the active VMID, and mark a local TLB flush pending for each CPU.

Do not send IPIs or otherwise force remote vCPUs to exit during rollover. Instead, when a CPU next activates a VMID before guest entry, perform the pending local HFENCE.GVMA and only then publish the new active VMID.

Perform VMID activation after preemption is disabled in the guest entry path. This keeps the per-CPU active VMID and pending flush state associated with the same CPU that subsequently enters guest mode.

Keep reserved hardware VMIDs unavailable to the bitmap allocator while stale translations associated with them may still exist. This prevents a hardware VMID from being reused prematurely after a generation rollover.

Represent a software VMID as a generation combined with a hardware VMID, and extract only the hardware VMID when programming HGATP or issuing VMID-specific fences.

Reserve hardware VMID 0. Disable VMID allocation when the hardware VMID space is too small to retain one reserved VMID per possible CPU while still leaving a VMID available for a new allocation.

Invalidate the per-CPU VMID bookkeeping when virtualization is disabled, so stale G-stage translations are flushed before the CPU next enters a guest.

This removes IPIs and remote CPU synchronization from the VMID rollover path, avoiding VM exits caused solely by VMID exhaustion on another CPU.

Tested on QEMU with two host CPUs and the VMID width temporarily forced to 2 bits. Repeated rollover, protection against premature VMID reuse, activation-versus-rollover stress, and vCPU migration were exercised with multiple concurrent guests without guest failures or host warnings.

Assisted-by YuanSheng:deepseek-v4-pro
---
 arch/riscv/include/asm/kvm_gstage.h |   5 +-
 arch/riscv/include/asm/kvm_vmid.h   |  26 ++-
 arch/riscv/kvm/main.c               |  12 +-
 arch/riscv/kvm/mmu.c                |   5 +-
 arch/riscv/kvm/tlb.c                |   9 +-
 arch/riscv/kvm/vcpu.c               |   4 +-
 arch/riscv/kvm/vcpu_sbi_replace.c   |   4 +-
 arch/riscv/kvm/vcpu_sbi_v01.c       |   6 +-
 arch/riscv/kvm/vmid.c               | 347 ++++++++++++++++++++++++----
 9 files changed, 348 insertions(+), 70 deletions(-)
arch/riscv/include/asm/kvm_gstage.h +3 -2
index aaf080ba1b77..7a295e106005 100644
@@ -8,7 +8,7 @@
 #define __RISCV_KVM_GSTAGE_H_
 
 #include <linux/kvm_types.h>
-
+#include <asm/kvm_vmid.h>
 struct kvm_gstage {
 	struct kvm *kvm;
 	unsigned long flags;
@@ -108,7 +108,8 @@ static inline void kvm_riscv_gstage_init(struct kvm_gstage *gstage, struct kvm *
 {
 	gstage->kvm = kvm;
 	gstage->flags = 0;
-	gstage->vmid = READ_ONCE(kvm->arch.vmid.vmid);
+	gstage->vmid = kvm_riscv_gstage_vmid_hwid(
+			atomic_long_read(&kvm->arch.vmid.id));
 	gstage->pgd = kvm->arch.pgd;
 	gstage->pgd_levels = kvm->arch.pgd_levels;
 }
arch/riscv/include/asm/kvm_vmid.h +22 -4
index db61b0525a8d..78811ab7dba0 100644
@@ -6,21 +6,39 @@
 #ifndef __RISCV_KVM_VMID_H_
 #define __RISCV_KVM_VMID_H_
 
+#include <linux/atomic.h>
 #include <linux/kvm_types.h>
 
 struct kvm_vmid {
 	/*
-	 * Writes to vmid_version and vmid happen with vmid_lock held
-	 * whereas reads happen without any lock held.
+	 * Software VMID:
+	 *
+	 *   [ generation | hardware VMID ]
+	 *
+	 * Only the low hardware VMID bits may be written to HGATP
+	 * or used as a hardware fence VMID.
 	 */
-	unsigned long vmid_version;
-	unsigned long vmid;
+	atomic_long_t id;
 };
 
 void __init kvm_riscv_gstage_vmid_detect(void);
 unsigned long kvm_riscv_gstage_vmid_bits(void);
+
+int __init kvm_riscv_gstage_vmid_alloc_init(void);
+void kvm_riscv_gstage_vmid_alloc_free(void);
+
 int kvm_riscv_gstage_vmid_init(struct kvm *kvm);
+
+unsigned long kvm_riscv_gstage_vmid_hwid(unsigned long vmid);
+
 bool kvm_riscv_gstage_vmid_ver_changed(struct kvm_vmid *vmid);
+
 void kvm_riscv_gstage_vmid_update(struct kvm_vcpu *vcpu);
 
+/*
+ * Invalidate the current CPU's fast-path VMID state while preserving
+ * the old identity conservatively in reserved_vmids.
+ */
+void kvm_riscv_gstage_vmid_cpu_invalidate(void);
+
 #endif
arch/riscv/kvm/main.c +11 -1
index 89568ccce01d..d6c6be4df61b 100644
@@ -16,7 +16,7 @@
 #include <asm/kvm_nacl.h>
 #include <asm/sbi.h>
 #include <asm/kvm_vcpu_vector.h>
-
+#include <asm/kvm_vmid.h>
 static DEFINE_PER_CPU(bool, kvm_riscv_virtualization_enabled);
 
 DEFINE_STATIC_KEY_FALSE(kvm_riscv_vsstage_tlb_no_gpa);
@@ -84,6 +84,7 @@ int kvm_arch_enable_virtualization_cpu(void)
 
 void kvm_arch_disable_virtualization_cpu(void)
 {
+	kvm_riscv_gstage_vmid_cpu_invalidate();
 	kvm_riscv_aia_disable();
 	kvm_riscv_csr_cleanup();
 	kvm_riscv_nacl_disable();
@@ -112,6 +113,7 @@ static int kvm_riscv_cpu_pm_notifier(struct notifier_block *self, unsigned long
 		 * is enabled on this CPU.
 		 */
 		if (__this_cpu_read(kvm_riscv_virtualization_enabled)) {
+			kvm_riscv_gstage_vmid_cpu_invalidate();
 			kvm_riscv_aia_pm_enter();
 			kvm_riscv_csr_cleanup();
 		}
@@ -130,6 +132,7 @@ static struct notifier_block kvm_riscv_cpu_pm_nb = {
 static void kvm_riscv_teardown(void)
 {
 	kvm_riscv_aia_exit();
+	kvm_riscv_gstage_vmid_alloc_free();
 	kvm_riscv_nacl_exit();
 	kvm_riscv_v_exit();
 	kvm_unregister_perf_callbacks();
@@ -181,8 +184,15 @@ static int __init riscv_kvm_init(void)
 
 	kvm_riscv_gstage_vmid_detect();
 
+	rc = kvm_riscv_gstage_vmid_alloc_init();
+	if (rc) {
+		kvm_riscv_nacl_exit();
+		return rc;
+	}
+
 	rc = kvm_riscv_aia_init();
 	if (rc && rc != -ENODEV) {
+		kvm_riscv_gstage_vmid_alloc_free();
 		kvm_riscv_nacl_exit();
 		return rc;
 	}
arch/riscv/kvm/mmu.c +3 -2
index 6035b5ec9503..db49fac8e9af 100644
@@ -15,7 +15,7 @@
 #include <linux/sched/signal.h>
 #include <asm/kvm_mmu.h>
 #include <asm/kvm_nacl.h>
-
+#include <asm/kvm_vmid.h>
 static bool __read_mostly eager_page_split = true;
 module_param(eager_page_split, bool, 0644);
 
@@ -799,10 +799,11 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm)
 void kvm_riscv_mmu_update_hgatp(struct kvm_vcpu *vcpu)
 {
 	struct kvm_arch *ka = &vcpu->kvm->arch;
+	unsigned long vmid = atomic_long_read(&ka->vmid.id);
 	unsigned long hgatp = kvm_riscv_gstage_mode(ka->pgd_levels)
 			      << HGATP_MODE_SHIFT;
 
-	hgatp |= (READ_ONCE(ka->vmid.vmid) << HGATP_VMID_SHIFT) & HGATP_VMID;
+	hgatp |= (kvm_riscv_gstage_vmid_hwid(vmid) << HGATP_VMID_SHIFT) & HGATP_VMID;
 	hgatp |= (ka->pgd_phys >> PAGE_SHIFT) & HGATP_PPN;
 
 	ncsr_write(CSR_HGATP, hgatp);
arch/riscv/kvm/tlb.c +5 -4
index 2ae34632cdcb..36678aeb64ef 100644
@@ -224,7 +224,7 @@ void kvm_riscv_local_tlb_sanitize(struct kvm_vcpu *vcpu)
 	 * entries by VMID whenever underlying Host CPU changes for a VCPU.
 	 */
 
-	vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid);
+	vmid = kvm_riscv_gstage_vmid_hwid(atomic_long_read(&vcpu->kvm->arch.vmid.id));
 	kvm_riscv_local_hfence_gvma_vmid_all(vmid);
 
 	/*
@@ -244,7 +244,7 @@ void kvm_riscv_fence_i_process(struct kvm_vcpu *vcpu)
 void kvm_riscv_tlb_flush_process(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vmid *v = &vcpu->kvm->arch.vmid;
-	unsigned long vmid = READ_ONCE(v->vmid);
+	unsigned long vmid = kvm_riscv_gstage_vmid_hwid(atomic_long_read(&v->id));
 
 	if (kvm_riscv_nacl_available())
 		nacl_hfence_gvma_vmid_all(nacl_shmem(), vmid);
@@ -255,7 +255,7 @@ void kvm_riscv_tlb_flush_process(struct kvm_vcpu *vcpu)
 void kvm_riscv_hfence_vvma_all_process(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vmid *v = &vcpu->kvm->arch.vmid;
-	unsigned long vmid = READ_ONCE(v->vmid);
+	unsigned long vmid = kvm_riscv_gstage_vmid_hwid(atomic_long_read(&v->id));
 
 	if (kvm_riscv_nacl_available())
 		nacl_hfence_vvma_all(nacl_shmem(), vmid);
@@ -533,6 +533,7 @@ int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages)
 {
 	kvm_riscv_hfence_gvma_vmid_gpa(kvm, -1UL, 0,
 				       gfn << PAGE_SHIFT, nr_pages << PAGE_SHIFT,
-				       PAGE_SHIFT, READ_ONCE(kvm->arch.vmid.vmid));
+				       PAGE_SHIFT, kvm_riscv_gstage_vmid_hwid(
+						atomic_long_read(&kvm->arch.vmid.id)));
 	return 0;
 }
arch/riscv/kvm/vcpu.c +2 -2
index e062ca19f9d8..85542c1a5ec3 100644
@@ -943,14 +943,14 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 			continue;
 		ret = 1;
 
-		kvm_riscv_gstage_vmid_update(vcpu);
-
 		ret = kvm_riscv_check_vcpu_requests(vcpu);
 		if (ret <= 0)
 			continue;
 
 		preempt_disable();
 
+		kvm_riscv_gstage_vmid_update(vcpu);
+
 		/* Update AIA HW state before entering guest */
 		ret = kvm_riscv_vcpu_aia_update(vcpu);
 		if (ret <= 0) {
arch/riscv/kvm/vcpu_sbi_replace.c +2 -2
index 506a510b6bff..84ba5ce4131e 100644
@@ -104,7 +104,7 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run
 		kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_FENCE_I_SENT);
 		break;
 	case SBI_EXT_RFENCE_REMOTE_SFENCE_VMA:
-		vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid);
+		vmid = kvm_riscv_gstage_vmid_hwid(atomic_long_read(&vcpu->kvm->arch.vmid.id));
 		if ((cp->a2 == 0 && cp->a3 == 0) || cp->a3 == -1UL)
 			kvm_riscv_hfence_vvma_all(vcpu->kvm, hbase, hmask, vmid);
 		else
@@ -113,7 +113,7 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run
 		kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_SENT);
 		break;
 	case SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID:
-		vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid);
+		vmid = kvm_riscv_gstage_vmid_hwid(atomic_long_read(&vcpu->kvm->arch.vmid.id));
 		if ((cp->a2 == 0 && cp->a3 == 0) || cp->a3 == -1UL)
 			kvm_riscv_hfence_vvma_asid_all(vcpu->kvm, hbase, hmask,
 						       cp->a4, vmid);
arch/riscv/kvm/vcpu_sbi_v01.c +4 -2
index de544ea3f28d..e38794838ca4 100644
@@ -88,14 +88,16 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
 		if (cp->a7 == SBI_EXT_0_1_REMOTE_FENCE_I)
 			kvm_riscv_fence_i(vcpu->kvm, hbase, hmask);
 		else if (cp->a7 == SBI_EXT_0_1_REMOTE_SFENCE_VMA) {
-			vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid);
+			vmid = kvm_riscv_gstage_vmid_hwid(
+				   atomic_long_read(&vcpu->kvm->arch.vmid.id));
 			if (cp->a1 == 0 && cp->a2 == 0)
 				kvm_riscv_hfence_vvma_all(vcpu->kvm, hbase, hmask, vmid);
 			else
 				kvm_riscv_hfence_vvma_gva(vcpu->kvm, hbase, hmask, cp->a1,
 							  cp->a2, PAGE_SHIFT, vmid);
 		} else {
-			vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid);
+			vmid = kvm_riscv_gstage_vmid_hwid(
+				   atomic_long_read(&vcpu->kvm->arch.vmid.id));
 			if (cp->a1 == 0 && cp->a2 == 0)
 				kvm_riscv_hfence_vvma_asid_all(vcpu->kvm, hbase, hmask,
 							       cp->a3, vmid);
arch/riscv/kvm/vmid.c +296 -51

413 lines not on this page · raw message

Signature · 1 line
2.47.3
#1 lore raw
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Critical] Packing generation and VMID into atomic_long_t truncates the generation counter on 32-bit platforms, introducing a rapid ABA vulnerability.

Pre-existing issues:
- [Critical] VMID generation counter wraparound to 0 causes new VMs to share reserved hardware VMID 0, breaking cross-VM isolation. --

commit 41c24296ddc6bb7cc33a83183e982b0c76bf449d
Author: Wenlong Li <[email protected]>

RISC-V: KVM: Avoid synchronous IPIs on VMID rollover

Make VMID rollover lazy to remove cross-CPU synchronization and avoid synchronous IPIs. Track active and reserved software VMIDs for each possible CPU, preserving hardware VMIDs with stale translations, and deferring TLB flushes until the CPU next activates a VMID.

Quoted 16 lines
arch/riscv/include/asm/kvm_vmid.h +2 -2
index db61b0525a8d0..78811ab7dba07 100644
@@ -6,21 +6,39 @@
 #ifndef __RISCV_KVM_VMID_H_
 #define __RISCV_KVM_VMID_H_
 
+#include <linux/atomic.h>
 #include <linux/kvm_types.h>
 
 struct kvm_vmid {
 	/*
-	 * Writes to vmid_version and vmid happen with vmid_lock held
-	 * whereas reads happen without any lock held.
+	 * Software VMID:

arch/riscv/include/asm/kvm_vmid.h line 22

Quoted 10 lines
arch/riscv/include/asm/kvm_vmid.h +6 -2
+	 *
+	 *   [ generation | hardware VMID ]
+	 *
+	 * Only the low hardware VMID bits may be written to HGATP
+	 * or used as a hardware fence VMID.
 	 */
-	unsigned long vmid_version;
-	unsigned long vmid;
+	atomic_long_t id;
 };

[Severity: Critical]
Does packing the generation and VMID into a single atomic_long_t truncate the generation counter on 32-bit platforms, causing a rapid ABA regression?

On RV32 platforms, atomic_long_t is only 32 bits. Reserving the lower bits for the hardware VMID restricts the available bits for the generation counter significantly.

Can a dormant VM sleep while an unprivileged guest forces rapid VMID rollovers, causing the counter to wrap in roughly an hour? If the dormant VM wakes up with a perfectly matching old generation, it could bypass allocation and reuse its old hardware VMID that might already belong to another active VM.

Should this use atomic64_t to prevent this rapid wraparound?

Quoted 4 lines
arch/riscv/kvm/vmid.c +0 -0
index c15bdb1dd8bef..05819e4f7cb57 100644

[ ... ]

arch/riscv/kvm/vmid.c unsigned long kvm_riscv_gstage_vmid_bits(void) · line 124

Quoted 9 lines

int kvm_riscv_gstage_vmid_init(struct kvm *kvm)
{

arch/riscv/kvm/vmid.c +1 -3
-	/* Mark the initial VMID and VMID version invalid */
-	kvm->arch.vmid.vmid_version = 0;
-	kvm->arch.vmid.vmid = 0;
+	atomic_long_set(&kvm->arch.vmid.id, 0);
 
 	return 0;
 }

[Severity: Critical]
This is a pre-existing issue, but does initializing the VM id to 0 in kvm_riscv_gstage_vmid_init() cause collisions when the global generation counter eventually wraps around to exactly 0?

When vmid_generation wraps to 0 in new_vmid_locked():

generation = atomic_long_add_return(VMID_FIRST_VERSION,
&vmid_generation);

The vmid_gen_match() check will evaluate to true for any newly created VM because its id was initialized to 0:

    static bool vmid_gen_match(unsigned long vmid)
    {
        return !((vmid ^ atomic_long_read(&vmid_generation)) >> vmid_bits);
    }

This causes kvm_riscv_gstage_vmid_update() to skip the slow path allocation:

    if (!vmid_gen_match(vmid)) {
        vmid = new_vmid_locked(kvm_vmid);
        atomic_long_set(&kvm_vmid->id, vmid);
        vmid_changed = true;
    }

Will this result in multiple new VMs sharing the reserved hardware VMID 0 and inadvertently bypassing Stage-2 memory isolation?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1