Skip to contentSkip to conversation

RISC-V: KVM: Read HGEIP CSR on the correct cpu

PATCH In review [email protected] 3 participants 4 messages Oct 16, 2025 – Oct 17, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RISC-V: KVM: Read HGEIP CSR on the correct cpu

From: Fangyu Yu <[email protected]>

When executing kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have migrated and the IMSIC VS-file have not been updated yet, currently the HGEIP CSR should be read from the imsic->vsfile_cpu ( the pCPU before migration ) via on_each_cpu_mask, but this will trigger an IPI call and repeated IPI within a period of time is expensive in a many-core systems.

Just let the vCPU execute and update the correct IMSIC VS-file via kvm_riscv_vcpu_aia_imsic_update may be a simple solution.

Fixes 4cec89db80ba ("RISC-V: KVM: Move HGEI[E|P] CSR access to IMSIC virtualization")
---
 arch/riscv/kvm/aia_imsic.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
arch/riscv/kvm/aia_imsic.c +6 -2
index fda0346f0ea1..168c02ad0a78 100644
@@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)
 	 */
 
 	read_lock_irqsave(&imsic->vsfile_lock, flags);
-	if (imsic->vsfile_cpu > -1)
-		ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
+	if (imsic->vsfile_cpu > -1) {
+		if (imsic->vsfile_cpu != smp_processor_id())
+			ret = true;
+		else
+			ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
+	}
 	read_unlock_irqrestore(&imsic->vsfile_lock, flags);
 
 	return ret;
Signature · 1 line
2.50.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 24 lines · On Thu, Oct 16, 2025 at 9:27 AM <[email protected]> wrote:

From: Fangyu Yu <[email protected]>

When executing kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have migrated and the IMSIC VS-file have not been updated yet, currently the HGEIP CSR should be read from the imsic->vsfile_cpu ( the pCPU before migration ) via on_each_cpu_mask, but this will trigger an IPI call and repeated IPI within a period of time is expensive in a many-core systems.

Just let the vCPU execute and update the correct IMSIC VS-file via kvm_riscv_vcpu_aia_imsic_update may be a simple solution.

Fixes 4cec89db80ba ("RISC-V: KVM: Move HGEI[E|P] CSR access to IMSIC virtualization")
---
 arch/riscv/kvm/aia_imsic.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
arch/riscv/kvm/aia_imsic.c +0 -0
index fda0346f0ea1..168c02ad0a78 100644

arch/riscv/kvm/aia_imsic.c bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu) · line 693

Quoted 7 lines
arch/riscv/kvm/aia_imsic.c +2 -2
@@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)
         */
 
        read_lock_irqsave(&imsic->vsfile_lock, flags);
-       if (imsic->vsfile_cpu > -1)
-               ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
+       if (imsic->vsfile_cpu > -1) {
+               if (imsic->vsfile_cpu != smp_processor_id())

Why not if (imsic->vsfile_cpu != vcpu->cpu)?
if pcpu changed, the kvm_sched_in() would update "vcpu->cpu by raw_smp_processor_id()".

The smp_processor_id() may involve DEBUG_PREEMPT code, and check_preemption_disabled() may cause a problem? Anyway, a full smp_processor_id() is not needed here, vcpu->cpu is enough.

If (imsic->vsfile_cpu != vcpu->cpu), then:

arch/riscv/kvm/aia_imsic.c bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu) · line 694

Quoted 1 line
arch/riscv/kvm/aia_imsic.c +1 -0
+                       ret = true;

Agree!

When a vCPU migrates to another pCPU, we could assume the interrupt has come in to prevent the missing interrupt.

To illustrate why ret = true, consider this example:

1. Guest WFI VM_exit.

2. Host Trap -> kvm_riscv_vcpu_exit() -> kvm_riscv_vcpu_virtual_insn() ->kvm_vcpu_halt() ->

    kvm_vcpu_check_block() -> kvm_riscv_vcpu_aia_imsic_has_interrupt() loop:
        for (;;) {
                set_current_state(TASK_INTERRUPTIBLE);

                if (kvm_vcpu_check_block(vcpu) < 0)
                        break;

                waited = true;
                schedule(); // if pcpu changed, the kvm_sched_in()
would update "vcpu->cpu = raw_smp_processor_id()".
        }

3. Back to the outer loop in kvm/vcpu.c:

        while (ret > 0) {
                /* Check conditions before entering the guest */
                ret = xfer_to_guest_mode_handle_work(vcpu);
                if (ret)
                        continue;
                ret = 1;

                kvm_riscv_gstage_vmid_update(vcpu);

                kvm_riscv_check_vcpu_requests(vcpu);

                preempt_disable();

                /* Update AIA HW state before entering guest */
                ret = kvm_riscv_vcpu_aia_update(vcpu); // Update

imsic->vsfile_cpu with vcpu->cpu;

4. VM_enter Guest WFI
If there are no changes, then Guest WFI VM_exit again.

5. Host Trap again, but the vsfile_cpu & vcpu->cpu & raw_smp_processor_id() are synchronized.

Quoted 9 lines
arch/riscv/kvm/aia_imsic.c +3 -1
+               else
+                       ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
+       }
        read_unlock_irqrestore(&imsic->vsfile_lock, flags);
 
        return ret;
--

2.50.1

--
Best Regards
Guo Ren

maintainer
#2 lore raw
In reply to [email protected]
Quoted 24 lines · On Thu, Oct 16, 2025 at 6:57 AM <[email protected]> wrote:

From: Fangyu Yu <[email protected]>

When executing kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have migrated and the IMSIC VS-file have not been updated yet, currently the HGEIP CSR should be read from the imsic->vsfile_cpu ( the pCPU before migration ) via on_each_cpu_mask, but this will trigger an IPI call and repeated IPI within a period of time is expensive in a many-core systems.

Just let the vCPU execute and update the correct IMSIC VS-file via kvm_riscv_vcpu_aia_imsic_update may be a simple solution.

Fixes 4cec89db80ba ("RISC-V: KVM: Move HGEI[E|P] CSR access to IMSIC virtualization")
---
 arch/riscv/kvm/aia_imsic.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
arch/riscv/kvm/aia_imsic.c +0 -0
index fda0346f0ea1..168c02ad0a78 100644

arch/riscv/kvm/aia_imsic.c bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu) · line 693

Quoted 7 lines
arch/riscv/kvm/aia_imsic.c +2 -2
@@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)
         */
 
        read_lock_irqsave(&imsic->vsfile_lock, flags);
-       if (imsic->vsfile_cpu > -1)
-               ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
+       if (imsic->vsfile_cpu > -1) {
+               if (imsic->vsfile_cpu != smp_processor_id())

Good catch !!!

I agree with Guo Ren. We should use "vcpu->cpu" over here instead of smp_processor_id(). Also, I think we should add some comments for future reference. I will take care of this at the time of merging this patch.

Queued this as fixes for Linux-6.18

Thanks,
Anup

Quoted 10 lines
arch/riscv/kvm/aia_imsic.c +4 -1
+                       ret = true;
+               else
+                       ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
+       }
        read_unlock_irqrestore(&imsic->vsfile_lock, flags);
 
        return ret;
--

2.50.1

#3 lore raw
Quoted 36 lines · Anup Patel
Quoted 24 lines · On Thu, Oct 16, 2025 at 6:57 AM <[email protected]> wrote:

From: Fangyu Yu <[email protected]>

When executing kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have migrated and the IMSIC VS-file have not been updated yet, currently the HGEIP CSR should be read from the imsic->vsfile_cpu ( the pCPU before migration ) via on_each_cpu_mask, but this will trigger an IPI call and repeated IPI within a period of time is expensive in a many-core systems.

Just let the vCPU execute and update the correct IMSIC VS-file via kvm_riscv_vcpu_aia_imsic_update may be a simple solution.

Fixes 4cec89db80ba ("RISC-V: KVM: Move HGEI[E|P] CSR access to IMSIC virtualization")
---
 arch/riscv/kvm/aia_imsic.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
arch/riscv/kvm/aia_imsic.c +0 -0
index fda0346f0ea1..168c02ad0a78 100644

arch/riscv/kvm/aia_imsic.c bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu) · line 693

Quoted 7 lines
arch/riscv/kvm/aia_imsic.c +2 -2
@@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)
         */
 
        read_lock_irqsave(&imsic->vsfile_lock, flags);
-       if (imsic->vsfile_cpu > -1)
-               ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
+       if (imsic->vsfile_cpu > -1) {
+               if (imsic->vsfile_cpu != smp_processor_id())

Good catch !!!

Quoted 6 lines

I agree with Guo Ren. We should use "vcpu->cpu" over here instead of smp_processor_id(). Also, I think we should add some comments for future reference. I will take care of this at the time of merging this patch.

Queued this as fixes for Linux-6.18

Thx Anup, then Fangyu needn't update v2.

This patch also reveals the idea of our CSR_HGEIP design - Per-CPU access. Although arm64 & x86 don't have a similar problem by using a shared memory address map, it would be a complex hw design. The RISC-V CSR_HGEIP design is much simpler for hw, and vCPU migration on the pCPUs could tolerate one additional VM_enter/exit cost for an idle vcpu.

Quoted 14 lines

Thanks,
Anup

Quoted 10 lines
arch/riscv/kvm/aia_imsic.c +4 -1
+                       ret = true;
+               else
+                       ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
+       }
        read_unlock_irqrestore(&imsic->vsfile_lock, flags);
 
        return ret;
--

2.50.1

Signature · 2 lines
Best Regards
 Guo Ren