RISC-V: KVM: Read HGEIP CSR on the correct cpu
Subsystems (3)
PATCH 1/1 RISC-V: KVM: Read HGEIP CSR on the correct cpu
From: Fangyu Yu <[email protected]>
When executing kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have migrated and the IMSIC VS-file have not been updated yet, currently the HGEIP CSR should be read from the imsic->vsfile_cpu ( the pCPU before migration ) via on_each_cpu_mask, but this will trigger an IPI call and repeated IPI within a period of time is expensive in a many-core systems.
Just let the vCPU execute and update the correct IMSIC VS-file via kvm_riscv_vcpu_aia_imsic_update may be a simple solution.
--- arch/riscv/kvm/aia_imsic.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
arch/riscv/kvm/aia_imsic.c +6 -2
@@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)*/ read_lock_irqsave (&imsic->vsfile_lock, flags); -if (imsic->vsfile_cpu >-1 ) - ret = !!(csr_read (CSR_HGEIP ) &BIT (imsic->vsfile_hgei)); +if (imsic->vsfile_cpu >-1 ) { +if (imsic->vsfile_cpu !=smp_processor_id ()) + ret = true; +else + ret = !!(csr_read (CSR_HGEIP ) &BIT (imsic->vsfile_hgei)); + }read_unlock_irqrestore (&imsic->vsfile_lock, flags);return ret;
Signature · 1 line
2.50.1
Quoted 24 lines · On Thu, Oct 16, 2025 at 9:27 AM <[email protected]> wrote:
From: Fangyu Yu <[email protected]>
When executing
kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have migrated and the IMSIC VS-file have not been updated yet, currently the HGEIP CSR should be read from theimsic->vsfile_cpu( the pCPU before migration ) viaon_each_cpu_mask, but this will trigger an IPI call and repeated IPI within a period of time is expensive in a many-core systems.Just let the vCPU execute and update the correct IMSIC VS-file via
kvm_riscv_vcpu_aia_imsic_updatemay be a simple solution.Fixes 4cec89db80ba ("RISC-V: KVM: Move HGEI[E|P] CSR access to IMSIC virtualization")Signed-off-by Fangyu Yu <[email protected]>--- arch/riscv/kvm/aia_imsic.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)arch/riscv/kvm/aia_imsic.c +0
-0
arch/riscv/kvm/aia_imsic.c bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu) · line 693
Quoted 7 lines
arch/riscv/kvm/aia_imsic.c +2
-2@@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)*/ read_lock_irqsave (&imsic->vsfile_lock, flags); -if (imsic->vsfile_cpu >-1 ) - ret = !!(csr_read (CSR_HGEIP ) &BIT (imsic->vsfile_hgei)); +if (imsic->vsfile_cpu >-1 ) { +if (imsic->vsfile_cpu !=smp_processor_id ())
Why not if (imsic->vsfile_cpu != vcpu->cpu)?
if pcpu changed, the kvm_sched_in() would update "vcpu->cpu by raw_smp_processor_id()".
The smp_processor_id() may involve DEBUG_PREEMPT code, and check_preemption_disabled() may cause a problem? Anyway, a full smp_processor_id() is not needed here, vcpu->cpu is enough.
If (imsic->vsfile_cpu != vcpu->cpu), then:
arch/riscv/kvm/aia_imsic.c bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu) · line 694
Quoted 1 line
arch/riscv/kvm/aia_imsic.c +1
-0+ ret = true;
Agree!
When a vCPU migrates to another pCPU, we could assume the interrupt has come in to prevent the missing interrupt.
To illustrate why ret = true, consider this example:
1. Guest WFI VM_exit.
2. Host Trap -> kvm_riscv_vcpu_exit() -> kvm_riscv_vcpu_virtual_insn() ->kvm_vcpu_halt() ->
kvm_vcpu_check_block() -> kvm_riscv_vcpu_aia_imsic_has_interrupt() loop:
for (;;) {
set_current_state(TASK_INTERRUPTIBLE);
if (kvm_vcpu_check_block(vcpu) < 0)
break;
waited = true;
schedule(); // if pcpu changed, the kvm_sched_in()
would update "vcpu->cpu = raw_smp_processor_id()".
} 3. Back to the outer loop in kvm/vcpu.c:
while (ret > 0) {
/* Check conditions before entering the guest */
ret = xfer_to_guest_mode_handle_work(vcpu);
if (ret)
continue;
ret = 1;
kvm_riscv_gstage_vmid_update(vcpu);
kvm_riscv_check_vcpu_requests(vcpu);
preempt_disable();
/* Update AIA HW state before entering guest */
ret = kvm_riscv_vcpu_aia_update(vcpu); // Updateimsic->vsfile_cpu with vcpu->cpu;
4. VM_enter Guest WFI
If there are no changes, then Guest WFI VM_exit again.
5. Host Trap again, but the vsfile_cpu & vcpu->cpu & raw_smp_processor_id() are synchronized.
Quoted 9 lines
arch/riscv/kvm/aia_imsic.c +3
-1+else + ret = !!(csr_read (CSR_HGEIP ) &BIT (imsic->vsfile_hgei)); + }read_unlock_irqrestore (&imsic->vsfile_lock, flags);return ret; --2.50.1
--
Best Regards
Guo Ren
Anup Patel
maintainerQuoted 24 lines · On Thu, Oct 16, 2025 at 6:57 AM <[email protected]> wrote:
From: Fangyu Yu <[email protected]>
When executing
kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have migrated and the IMSIC VS-file have not been updated yet, currently the HGEIP CSR should be read from theimsic->vsfile_cpu( the pCPU before migration ) viaon_each_cpu_mask, but this will trigger an IPI call and repeated IPI within a period of time is expensive in a many-core systems.Just let the vCPU execute and update the correct IMSIC VS-file via
kvm_riscv_vcpu_aia_imsic_updatemay be a simple solution.Fixes 4cec89db80ba ("RISC-V: KVM: Move HGEI[E|P] CSR access to IMSIC virtualization")Signed-off-by Fangyu Yu <[email protected]>--- arch/riscv/kvm/aia_imsic.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)arch/riscv/kvm/aia_imsic.c +0
-0
arch/riscv/kvm/aia_imsic.c bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu) · line 693
Quoted 7 lines
arch/riscv/kvm/aia_imsic.c +2
-2@@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)*/ read_lock_irqsave (&imsic->vsfile_lock, flags); -if (imsic->vsfile_cpu >-1 ) - ret = !!(csr_read (CSR_HGEIP ) &BIT (imsic->vsfile_hgei)); +if (imsic->vsfile_cpu >-1 ) { +if (imsic->vsfile_cpu !=smp_processor_id ())
Good catch !!!
I agree with Guo Ren. We should use "vcpu->cpu" over here instead of smp_processor_id(). Also, I think we should add some comments for future reference. I will take care of this at the time of merging this patch.
Queued this as fixes for Linux-6.18
Thanks,
Anup
Quoted 10 lines
arch/riscv/kvm/aia_imsic.c +4
-1+ ret = true; + else + ret = !!(csr_read (CSR_HGEIP ) &BIT (imsic->vsfile_hgei)); + }read_unlock_irqrestore (&imsic->vsfile_lock, flags);return ret; --2.50.1
Quoted 36 lines · Anup Patel
Quoted 24 lines · On Thu, Oct 16, 2025 at 6:57 AM <[email protected]> wrote:
From: Fangyu Yu <[email protected]>
When executing
kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have migrated and the IMSIC VS-file have not been updated yet, currently the HGEIP CSR should be read from theimsic->vsfile_cpu( the pCPU before migration ) viaon_each_cpu_mask, but this will trigger an IPI call and repeated IPI within a period of time is expensive in a many-core systems.Just let the vCPU execute and update the correct IMSIC VS-file via
kvm_riscv_vcpu_aia_imsic_updatemay be a simple solution.Fixes 4cec89db80ba ("RISC-V: KVM: Move HGEI[E|P] CSR access to IMSIC virtualization")Signed-off-by Fangyu Yu <[email protected]>--- arch/riscv/kvm/aia_imsic.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)arch/riscv/kvm/aia_imsic.c +0
-0arch/riscv/kvm/aia_imsic.c bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu) · line 693
Quoted 7 lines
arch/riscv/kvm/aia_imsic.c +2
-2@@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)*/ read_lock_irqsave (&imsic->vsfile_lock, flags); -if (imsic->vsfile_cpu >-1 ) - ret = !!(csr_read (CSR_HGEIP ) &BIT (imsic->vsfile_hgei)); +if (imsic->vsfile_cpu >-1 ) { +if (imsic->vsfile_cpu !=smp_processor_id ())Good catch !!!
Quoted 6 lines
I agree with Guo Ren. We should use "
vcpu->cpu" over here instead ofsmp_processor_id(). Also, I think we should add some comments for future reference. I will take care of this at the time of merging this patch.Queued this as fixes for Linux-6.18
Thx Anup, then Fangyu needn't update v2.
This patch also reveals the idea of our CSR_HGEIP design - Per-CPU access. Although arm64 & x86 don't have a similar problem by using a shared memory address map, it would be a complex hw design. The RISC-V CSR_HGEIP design is much simpler for hw, and vCPU migration on the pCPUs could tolerate one additional VM_enter/exit cost for an idle vcpu.
Quoted 14 lines
Thanks,
AnupQuoted 10 lines
arch/riscv/kvm/aia_imsic.c +4
-1+ ret = true; + else + ret = !!(csr_read (CSR_HGEIP ) &BIT (imsic->vsfile_hgei)); + }read_unlock_irqrestore (&imsic->vsfile_lock, flags);return ret; --2.50.1
Signature · 2 lines
Best Regards Guo Ren