RISC-V: KVM: Avoid synchronous IPIs on VMID rollover
Subsystems (3)
PATCH v2 1/1 RISC-V: KVM: Avoid synchronous IPIs on VMID rollover
The current VMID rollover path synchronously invokes HFENCE.GVMA on all online CPUs using on_each_cpu_mask(). This sends IPIs to remote CPUs and causes a vCPU running in guest mode on a targeted CPU to exit. The rollover path then waits for all CPUs to complete their local TLB flushes before VMID allocation can continue.
Make VMID rollover lazy to remove this cross-CPU synchronization.
Track active and reserved software VMIDs for each possible CPU. On rollover, preserve hardware VMIDs that may still have stale translations, clear the active VMID, and mark a local TLB flush pending for each CPU.
Do not send IPIs or otherwise force remote vCPUs to exit during rollover. Instead, when a CPU next activates a VMID before guest entry, perform the pending local HFENCE.GVMA and only then publish the new active VMID.
Perform VMID activation after preemption is disabled in the guest entry path. This keeps the per-CPU active VMID and pending flush state associated with the same CPU that subsequently enters guest mode.
Keep reserved hardware VMIDs unavailable to the bitmap allocator while stale translations associated with them may still exist. This prevents a hardware VMID from being reused prematurely after a generation rollover.
Represent a software VMID as a 64-bit value combining a generation with a hardware VMID, independent of XLEN. This keeps the generation space large on RV32 while hardware VMIDs and bitmap indices remain native unsigned long values. Extract only the hardware VMID when programming HGATP or issuing VMID-specific fences.
Reserve software VMID 0 as the unallocated identifier. When the generation counter wraps to 0, compute the next nonzero generation locally and publish it atomically, so generation 0 is never visible to lockless readers.
Reserve hardware VMID 0. Disable VMID allocation when the hardware VMID space is too small to retain one reserved VMID per possible CPU while still leaving a VMID available for a new allocation.
Invalidate the per-CPU VMID bookkeeping when virtualization is disabled, so stale G-stage translations are flushed before the CPU next enters a guest.
This removes IPIs and remote CPU synchronization from the VMID rollover path, avoiding VM exits caused solely by VMID exhaustion on another CPU.
Tested on QEMU with two host CPUs and the VMID width temporarily forced to 2 bits. Repeated rollover, protection against premature VMID reuse, activation-versus-rollover stress, and vCPU migration were exercised with multiple concurrent guests without guest failures or host warnings.
---
Changes in v2:
- Use a 64-bit software VMID namespace independent of XLEN.
- Keep hardware VMIDs and bitmap indices as unsigned long.
- Reserve software VMID 0 as the invalid identifier.
- Avoid exposing generation 0 to lockless readers during rollover.
arch/riscv/include/asm/kvm_gstage.h | 5 +- arch/riscv/include/asm/kvm_vmid.h | 29 ++- arch/riscv/kvm/main.c | 12 +- arch/riscv/kvm/mmu.c | 5 +- arch/riscv/kvm/tlb.c | 11 +- arch/riscv/kvm/vcpu.c | 4 +- arch/riscv/kvm/vcpu_sbi_replace.c | 4 +- arch/riscv/kvm/vcpu_sbi_v01.c | 4 +- arch/riscv/kvm/vmid.c | 352 ++++++++++++++++++++++++---- 9 files changed, 356 insertions(+), 70 deletions(-)
arch/riscv/include/asm/kvm_gstage.h +3 -2
@@ -8,7 +8,7 @@#define __RISCV_KVM_GSTAGE_H_#include <linux/kvm_types.h> - +#include <asm/kvm_vmid.h> struct kvm_gstage {struct kvm *kvm;unsigned long flags;
@@ -108,7 +108,8 @@ static inline void kvm_riscv_gstage_init(struct kvm_gstage *gstage, struct kvm * { gstage->kvm = kvm; gstage->flags =0 ; - gstage->vmid =READ_ONCE (kvm->arch.vmid.vmid); + gstage->vmid = +kvm_riscv_gstage_vmid_hwid (atomic64_read (&kvm->arch.vmid.id)); gstage->pgd = kvm->arch.pgd; gstage->pgd_levels = kvm->arch.pgd_levels; }
arch/riscv/include/asm/kvm_vmid.h +25 -4
@@ -6,21 +6,42 @@#ifndef __RISCV_KVM_VMID_H_#define __RISCV_KVM_VMID_H_ +#include <linux/atomic.h> #include <linux/kvm_types.h> struct kvm_vmid {/* -* Writes to vmid_version and vmid happen with vmid_lock held -* whereas reads happen without any lock held. +* Software VMID: +* +* [ generation | hardware VMID ] +* +* Keep the identity 64-bit on RV32 so that the generation does not +* wrap at the native word size. Zero denotes an unallocated VMID. +* +* Only the low hardware VMID bits may be written to HGATP +* or used as a hardware fence VMID. */ -unsigned long vmid_version; -unsigned long vmid; +atomic64_t id; };void __init kvm_riscv_gstage_vmid_detect (void );unsigned long kvm_riscv_gstage_vmid_bits (void ); + +int __init kvm_riscv_gstage_vmid_alloc_init (void ); +void kvm_riscv_gstage_vmid_alloc_free (void ); +int kvm_riscv_gstage_vmid_init (struct kvm *kvm); + +unsigned long kvm_riscv_gstage_vmid_hwid (u64 vmid); +bool kvm_riscv_gstage_vmid_ver_changed (struct kvm_vmid *vmid); +void kvm_riscv_gstage_vmid_update (struct kvm_vcpu *vcpu); +/* +* Invalidate the current CPU's fast-path VMID state while preserving +* the old identity conservatively in reserved_vmids. +*/ +void kvm_riscv_gstage_vmid_cpu_invalidate (void ); +#endif
arch/riscv/kvm/main.c +11 -1
@@ -16,7 +16,7 @@#include <asm/kvm_nacl.h> #include <asm/sbi.h> #include <asm/kvm_vcpu_vector.h> - +#include <asm/kvm_vmid.h> static DEFINE_PER_CPU(bool, kvm_riscv_virtualization_enabled);DEFINE_STATIC_KEY_FALSE (kvm_riscv_vsstage_tlb_no_gpa);
@@ -84,6 +84,7 @@ int kvm_arch_enable_virtualization_cpu(void)void kvm_arch_disable_virtualization_cpu (void ) { +kvm_riscv_gstage_vmid_cpu_invalidate ();kvm_riscv_aia_disable ();kvm_riscv_csr_cleanup ();kvm_riscv_nacl_disable ();
@@ -112,6 +113,7 @@ static int kvm_riscv_cpu_pm_notifier(struct notifier_block *self, unsigned long* is enabled on this CPU. */ if (__this_cpu_read (kvm_riscv_virtualization_enabled)) { +kvm_riscv_gstage_vmid_cpu_invalidate ();kvm_riscv_aia_pm_enter ();kvm_riscv_csr_cleanup (); }
@@ -130,6 +132,7 @@ static struct notifier_block kvm_riscv_cpu_pm_nb = {static void kvm_riscv_teardown (void ) {kvm_riscv_aia_exit (); +kvm_riscv_gstage_vmid_alloc_free ();kvm_riscv_nacl_exit ();kvm_riscv_v_exit ();kvm_unregister_perf_callbacks ();
@@ -181,8 +184,15 @@ static int __init riscv_kvm_init(void)kvm_riscv_gstage_vmid_detect (); + rc =kvm_riscv_gstage_vmid_alloc_init (); +if (rc) { +kvm_riscv_nacl_exit (); +return rc; + } + rc =kvm_riscv_aia_init ();if (rc && rc != -ENODEV ) { +kvm_riscv_gstage_vmid_alloc_free ();kvm_riscv_nacl_exit ();return rc; }
arch/riscv/kvm/mmu.c +3 -2
@@ -15,7 +15,7 @@#include <linux/sched/signal.h> #include <asm/kvm_mmu.h> #include <asm/kvm_nacl.h> - +#include <asm/kvm_vmid.h> static bool __read_mostly eager_page_split = true;module_param (eager_page_split, bool,0644 );
@@ -799,10 +799,11 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm)void kvm_riscv_mmu_update_hgatp (struct kvm_vcpu *vcpu) {struct kvm_arch *ka = &vcpu->kvm->arch; +u64 vmid =atomic64_read (&ka->vmid.id);unsigned long hgatp =kvm_riscv_gstage_mode (ka->pgd_levels) <<HGATP_MODE_SHIFT ; - hgatp |= (READ_ONCE (ka->vmid.vmid) <<HGATP_VMID_SHIFT ) &HGATP_VMID ; + hgatp |= (kvm_riscv_gstage_vmid_hwid (vmid) <<HGATP_VMID_SHIFT ) &HGATP_VMID ; hgatp |= (ka->pgd_phys >>PAGE_SHIFT ) &HGATP_PPN ;ncsr_write (CSR_HGATP , hgatp);
arch/riscv/kvm/tlb.c +7 -4
@@ -224,7 +224,7 @@ void kvm_riscv_local_tlb_sanitize(struct kvm_vcpu *vcpu)* entries by VMID whenever underlying Host CPU changes for a VCPU. */ - vmid =READ_ONCE (vcpu->kvm->arch.vmid.vmid); + vmid =kvm_riscv_gstage_vmid_hwid (atomic64_read (&vcpu->kvm->arch.vmid.id));kvm_riscv_local_hfence_gvma_vmid_all (vmid);/*
@@ -244,7 +244,7 @@ void kvm_riscv_fence_i_process(struct kvm_vcpu *vcpu)void kvm_riscv_tlb_flush_process (struct kvm_vcpu *vcpu) {struct kvm_vmid *v = &vcpu->kvm->arch.vmid; -unsigned long vmid =READ_ONCE (v->vmid); +unsigned long vmid =kvm_riscv_gstage_vmid_hwid (atomic64_read (&v->id));if (kvm_riscv_nacl_available ())nacl_hfence_gvma_vmid_all (nacl_shmem (), vmid);
@@ -255,7 +255,7 @@ void kvm_riscv_tlb_flush_process(struct kvm_vcpu *vcpu)void kvm_riscv_hfence_vvma_all_process (struct kvm_vcpu *vcpu) {struct kvm_vmid *v = &vcpu->kvm->arch.vmid; -unsigned long vmid =READ_ONCE (v->vmid); +unsigned long vmid =kvm_riscv_gstage_vmid_hwid (atomic64_read (&v->id));if (kvm_riscv_nacl_available ())nacl_hfence_vvma_all (nacl_shmem (), vmid);
@@ -531,8 +531,11 @@ void kvm_riscv_hfence_vvma_all(struct kvm *kvm,int kvm_arch_flush_remote_tlbs_range (struct kvm *kvm,gfn_t gfn,u64 nr_pages) { +unsigned long vmid = +kvm_riscv_gstage_vmid_hwid (atomic64_read (&kvm->arch.vmid.id)); +kvm_riscv_hfence_gvma_vmid_gpa (kvm,-1UL ,0 , gfn <<PAGE_SHIFT , nr_pages <<PAGE_SHIFT , -PAGE_SHIFT ,READ_ONCE (kvm->arch.vmid.vmid)); +PAGE_SHIFT , vmid);return 0 ; }
arch/riscv/kvm/vcpu.c +2 -2
@@ -943,14 +943,14 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)continue ; ret =1 ; -kvm_riscv_gstage_vmid_update (vcpu); - ret =kvm_riscv_check_vcpu_requests (vcpu);if (ret <=0 )continue ;preempt_disable (); +kvm_riscv_gstage_vmid_update (vcpu); +/* Update AIA HW state before entering guest */ ret =kvm_riscv_vcpu_aia_update (vcpu);if (ret <=0 ) {
arch/riscv/kvm/vcpu_sbi_replace.c +2 -2
@@ -104,7 +104,7 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *runkvm_riscv_vcpu_pmu_incr_fw (vcpu,SBI_PMU_FW_FENCE_I_SENT );break ;case SBI_EXT_RFENCE_REMOTE_SFENCE_VMA : - vmid =READ_ONCE (vcpu->kvm->arch.vmid.vmid); + vmid =kvm_riscv_gstage_vmid_hwid (atomic64_read (&vcpu->kvm->arch.vmid.id));if ((cp->a2 ==0 && cp->a3 ==0 ) || cp->a3 ==-1UL )kvm_riscv_hfence_vvma_all (vcpu->kvm, hbase, hmask, vmid);else
@@ -113,7 +113,7 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *runkvm_riscv_vcpu_pmu_incr_fw (vcpu,SBI_PMU_FW_HFENCE_VVMA_SENT );break ;case SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID : - vmid =READ_ONCE (vcpu->kvm->arch.vmid.vmid); + vmid =kvm_riscv_gstage_vmid_hwid (atomic64_read (&vcpu->kvm->arch.vmid.id));if ((cp->a2 ==0 && cp->a3 ==0 ) || cp->a3 ==-1UL )kvm_riscv_hfence_vvma_asid_all (vcpu->kvm, hbase, hmask, cp->a4, vmid);
arch/riscv/kvm/vcpu_sbi_v01.c +2 -2
@@ -88,14 +88,14 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,if (cp->a7 ==SBI_EXT_0_1_REMOTE_FENCE_I )kvm_riscv_fence_i (vcpu->kvm, hbase, hmask);else if (cp->a7 ==SBI_EXT_0_1_REMOTE_SFENCE_VMA ) { - vmid =READ_ONCE (vcpu->kvm->arch.vmid.vmid); + vmid =kvm_riscv_gstage_vmid_hwid (atomic64_read (&vcpu->kvm->arch.vmid.id));if (cp->a1 ==0 && cp->a2 ==0 )kvm_riscv_hfence_vvma_all (vcpu->kvm, hbase, hmask, vmid);else kvm_riscv_hfence_vvma_gva (vcpu->kvm, hbase, hmask, cp->a1, cp->a2,PAGE_SHIFT , vmid); }else { - vmid =READ_ONCE (vcpu->kvm->arch.vmid.vmid); + vmid =kvm_riscv_gstage_vmid_hwid (atomic64_read (&vcpu->kvm->arch.vmid.id));if (cp->a1 ==0 && cp->a2 ==0 )kvm_riscv_hfence_vvma_asid_all (vcpu->kvm, hbase, hmask, cp->a3, vmid);
arch/riscv/kvm/vmid.c +301 -51
418 lines not on this page · raw message
Signature · 1 line
2.47.3
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Race condition during VMID rollover allows vCPU to enter guest with stale hardware VMID…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Race condition during VMID rollover allows vCPU to enter guest with stale hardware VMID, breaking memory isolation. --
commit f9683a31ea50a94b9a73a513d36b5718f5b167c1
Author: Wenlong Li <[email protected]>
RISC-V: KVM: Avoid synchronous IPIs on VMID rollover
This commit makes VMID rollover lazy to avoid synchronous cross-CPU IPIs on all online CPUs. It tracks active and reserved software VMIDs, performing pending local HFENCE.GVMAs on the next VMID activation before guest entry.
Quoted 8 lines
arch/riscv/kvm/vcpu.c +0
-0@@ -943,14 +943,14 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)continue ; ret =1 ;
arch/riscv/kvm/vcpu.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) · line 953
Quoted 10 lines
arch/riscv/kvm/vcpu.c +2
-2-kvm_riscv_gstage_vmid_update (vcpu); - ret =kvm_riscv_check_vcpu_requests (vcpu);if (ret <=0 )continue ;preempt_disable (); +kvm_riscv_gstage_vmid_update (vcpu); +
[Severity: Critical]
Does moving kvm_riscv_gstage_vmid_update() after kvm_riscv_check_vcpu_requests() create a window where a vCPU misses a pending request?
Later in kvm_arch_vcpu_ioctl_run(), the vCPU checks for pending requests:
if (kvm_riscv_gstage_vmid_ver_changed(&vcpu->kvm->arch.vmid) ||
kvm_request_pending(vcpu) ||
xfer_to_guest_mode_work_pending()) {
vcpu->mode = OUTSIDE_GUEST_MODE;If another vCPU updates the shared VMID generation locklessly, kvm_request_pending() can evaluate to false here if the KVM_REQ_UPDATE_HGATP request from the other vCPU has not yet been written. This would allow the vCPU to enter the guest with a stale hardware VMID.
Quoted 2 lines
arch/riscv/kvm/vcpu.c +0
-0/* Update AIA HW state before entering guest */ ret =kvm_riscv_vcpu_aia_update (vcpu);
[ ... ]
Quoted 5 lines
arch/riscv/kvm/vmid.c +0
-0@@ -201,31 +201,31 @@ void kvm_riscv_gstage_vmid_update(struct kvm_vcpu *vcpu)
[ ... ]
arch/riscv/kvm/vmid.c unsigned long kvm_riscv_gstage_vmid_bits(void) · line 328
Quoted 7 lines
arch/riscv/kvm/vmid.c +7
-0+/* +* Publish the CPU-local active identity only after the required +* local invalidation is complete. +*/ +atomic64_set (active, vmid); + +raw_spin_unlock_irqrestore (&vmid_lock, flags);
[Severity: Critical]
Is it safe to update the admission state and drop the lock here before issuing the HGATP update request?
If the current vCPU drops the lock, another vCPU can observe the new generation, update its own admission state, and bypass the KVM_REQ_UPDATE_HGATP check since the request has not yet been made. The second vCPU would then proceed to enter the guest.
arch/riscv/kvm/vmid.c unsigned long kvm_riscv_gstage_vmid_bits(void) · line 337
Quoted 9 lines
arch/riscv/kvm/vmid.c +8
-0+ +/* +* A new software VMID may contain a different hardware VMID, so +* request an HGATP update for every vCPU in the VM. +*/ +if (vmid_changed) { +kvm_for_each_vcpu (i, v, vcpu->kvm) +kvm_make_request (KVM_REQ_UPDATE_HGATP , v); }
[Severity: Critical]
Since this cross-vCPU request is issued locklessly, can this cause a regression where two VMs might share the same hardware VMID?
Because the second vCPU mentioned above advertises the new VMID in its admission state but still runs with the old hardware VMID in the guest, the old hardware VMID is no longer protected. It could be immediately reallocated to another VM, breaking spatial isolation.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1