RISC-V KVM: guest local sfence.vma may miss stale VS-stage TLB after vCPU migration
Subsystems (3)
Hi all,
I would like to ask whether RISC-V KVM has a gap around guest sfence.vma and VS-stage TLB invalidation when a vCPU migrates across host CPUs.
On bare metal, Linux uses mm_cpumask in __flush_tlb_range() to decide whether to send a local or remote sfence.vma. That works because the cpumask reflects physical CPUs. In a guest, however, the kernel only sees vCPUs, so from the guest point of view a task may never "migrate" even though the underlying vCPU moved to a different host CPU. In other words, guest mm_cpumask is effectively a vCPU mask, not a host CPU mask.
There is also a related switch_mm()/set_mm_asid() aspect. With the ASID allocator enabled, RISC-V Linux does not flush on every context switch. A local_flush_tlb_all() is only done when a deferred ASID rollover flush is pending for the current CPU. That is fine on bare metal, because previous shootdowns are also based on physical CPUs. In a guest, however, the earlier shootdown may have been only a guest-local sfence.vma and may have missed an old host CPU where the same vCPU ran previously.
A simplified sequence is:
1. A guest task with mm A runs on vcpu0 while vcpu0 is scheduled on host CPU1. CPU1 fills a VS-stage TLB entry for that mm/ASID.
2. vcpu0 migrates to host CPU0. KVM's vCPU migration sanitization is local to the current host CPU, so it does not invalidate the VS-stage TLB entry that was left on host CPU1.
3. The guest later updates mm A's page table from vcpu0, for example due to COW. Since the guest still sees mm A as having run only on guest CPU0, __flush_tlb_range() can choose a local sfence.vma. That local sfence.vma only affects host CPU0, where vcpu0 is currently running, so the stale VS-stage entry on host CPU1 remains.
4. Later, the task is scheduled again while vcpu0 is running on host CPU1. If the ASID is still valid and no deferred ASID rollover flush is pending, set_mm_asid() may not issue any flush. The access on host CPU1 can then hit the old VS-stage TLB entry.
We hit this on a XiangShan RISC-V system. A bash task in the guest was scheduled on vcpu0 while that vCPU was running on host CPU1. A read-only page was prefetched and cached in CPU1's VS-stage TLB. Later the same vCPU migrated to host CPU0, the guest triggered COW and updated the mapping, and the guest still issued only a local sfence.vma. The stale VS-stage entry on CPU1 was not invalidated. After running for some time, the task later accessed that mapping while running on host CPU1 again and hit the stale translation. The data in that page was a pointer from the GOT, so it dereferenced to NULL and eventually caused a segmentation fault.
I checked current upstream RISC-V KVM (v7.0.2) and I do not see HSTATUS.VTVM being enabled, so guest sfence.vma is not trapped. I found kvm_riscv_local_tlb_sanitize() on host CPU migration, which handles G-stage VMID entries on the current host CPU, but I do not think it generally solves the stale VS-stage TLB case described above.
I also noticed the vendor-specific kvm_riscv_vsstage_tlb_no_gpa path in current upstream. It looks like a workaround for a particular implementation issue on Andes AX66, where the VS-stage TLB does not cache guest physical address and VMID, so the normal VMID/GPA-based invalidation model is insufficient. In that sense, it does not seem to be a generic solution for guest sfence.vma coherence across vCPU migration.
I am wondering what the right direction should be. Enabling HSTATUS.VTVM and trapping guest sfence.vma would allow KVM to translate guest local sfence.vma into the required host-side hfence.vvma, but that may be too heavy for the common case. Another possible direction may be to make the current Andes-specific vCPU migration flush logic more generic, so implementations that need VS-stage TLB sanitization on vCPU migration can opt in through a common mechanism. A third possibility may be to define a lightweight SBI interface for guest local sfence.vma, so Linux could use it in __flush_tlb_range() for the local case when running under KVM, and KVM could then decide whether a local sfence.vma is sufficient or whether host-side hfence.vvma is also needed.
My questions are:
1. Is the behavior above expected on current RISC-V KVM?
2. If not, what would be the preferred fix direction?
3. Should this be handled by trapping guest sfence.vma with VTVM, by generalizing the existing vCPU-migration VS-stage TLB flush logic, or by adding a lighter SBI-mediated path for guest local sfence.vma?
4. Or is there another intended mechanism to keep VS-stage TLB state coherent across vCPU migration?
If needed, I can send a reproducer and the exact hardware/kernel details.
Thanks,
Yaxing Guo
PATCH 1/1 riscv: KVM: Flush VS-stage stale entries
From: "Guo Ren (Alibaba DAMO Academy)" <[email protected]>
When a vCPU migrates to a different host CPU, the new CPU may retain stale VS-stage TLB entries (GVA → GPA) left by a previous run of the same guest. Issue an HFENCE.VVMA (all ASIDs, all addresses) inside kvm_riscv_mmu_update_hgatp() whenever the vCPU is loaded on a CPU different from the one it last exited on.
---
Does this patch solve your problem? I would prefer to issue a full HFENCE.VVMA whenever a new VMID is installed on a pCPU.
--- arch/riscv/include/asm/kvm_mmu.h | 2 +- arch/riscv/kvm/mmu.c | 5 ++++- arch/riscv/kvm/vcpu.c | 4 ++-- 3 files changed, 7 insertions(+), 4 deletions(-)
arch/riscv/include/asm/kvm_mmu.h +1 -1
@@ -16,6 +16,6 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, struct kvm_gstage_mapping *out_map);int kvm_riscv_mmu_alloc_pgd (struct kvm *kvm);void kvm_riscv_mmu_free_pgd (struct kvm *kvm); -void kvm_riscv_mmu_update_hgatp (struct kvm_vcpu *vcpu); +void kvm_riscv_mmu_update_hgatp (struct kvm_vcpu *vcpu,int cpu);#endif
arch/riscv/kvm/mmu.c +4 -1
@@ -705,7 +705,7 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm)free_pages ((unsigned long )pgd,get_order (kvm_riscv_gstage_pgd_size)); } -void kvm_riscv_mmu_update_hgatp (struct kvm_vcpu *vcpu) +void kvm_riscv_mmu_update_hgatp (struct kvm_vcpu *vcpu,int cpu) {struct kvm_arch *ka = &vcpu->kvm->arch;unsigned long hgatp =kvm_riscv_gstage_mode (ka->pgd_levels)
@@ -718,4 +718,7 @@ void kvm_riscv_mmu_update_hgatp(struct kvm_vcpu *vcpu)if (!kvm_riscv_gstage_vmid_bits ())kvm_riscv_local_hfence_gvma_all (); + +if (vcpu->arch.last_exit_cpu != cpu) + asm volatile(HFENCE_VVMA(zero, zero) : : : "memory"); }
arch/riscv/kvm/vcpu.c +2 -2
@@ -593,7 +593,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)csr_write (CSR_VSATP , csr->vsatp); } -kvm_riscv_mmu_update_hgatp (vcpu); +kvm_riscv_mmu_update_hgatp (vcpu, cpu);kvm_riscv_vcpu_aia_load (vcpu, cpu);
@@ -686,7 +686,7 @@ static int kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu)kvm_riscv_reset_vcpu (vcpu, true);if (kvm_check_request (KVM_REQ_UPDATE_HGATP , vcpu)) -kvm_riscv_mmu_update_hgatp (vcpu); +kvm_riscv_mmu_update_hgatp (vcpu, vcpu->cpu);if (kvm_check_request (KVM_REQ_FENCE_I , vcpu))kvm_riscv_fence_i_process (vcpu);
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH] riscv: KVM: Flush VS-stage stale entries
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] Direct execution of the raw HFENCE.VVMA instruction in kvm_riscv_mmu_update_hgatp() bypasses the Nested Acceleration (NACL) shared memory mechanism, breaking nested virtualization.
Pre-existing issues:
- [Critical] The VS-stage TLB flush condition last_exit_cpu != cpu is incomplete, skipping necessary flushes during VMID rollover, vmid_bits == 0 configurations, and same-VM vCPU interleaving on the same physical CPU. --
--- Patch [1]: [PATCH] riscv: KVM: Flush VS-stage stale entries --- commit f1e080e98f0df465d2cf4d51c2026f9bf9cd7c71
Author: Guo Ren (Alibaba DAMO Academy) <[email protected]>
riscv: KVM: Flush VS-stage stale entries
When a vCPU migrates to a different host CPU, the new CPU may retain stale VS-stage TLB entries (GVA -> GPA) left by a previous run of the same guest. This patch issues an HFENCE.VVMA inside kvm_riscv_mmu_update_hgatp() whenever the vCPU is loaded on a CPU different from the one it last exited on.
arch/riscv/kvm/mmu.c void kvm_riscv_mmu_update_hgatp(struct kvm_vcpu *vcpu) · line 723
Quoted 10 lines
arch/riscv/kvm/mmu.c +2
-0@@ -719,4 +719,7 @@ void kvm_riscv_mmu_update_hgatp(struct kvm_vcpu *vcpu)if (!kvm_riscv_gstage_vmid_bits ())kvm_riscv_local_hfence_gvma_all (); + +if (vcpu->arch.last_exit_cpu != cpu)
[Severity: Critical]
This is a pre-existing issue, but does this condition properly flush VS-stage TLB entries during VMID rollover, configurations where vmid_bits is 0, or when interleaving different VMs on the same physical CPU?
If a vCPU switches VMs on the same physical CPU, last_exit_cpu will still equal cpu and skip the flush. Since HFENCE.GVMA issued during rollover or VM switch does not guarantee flushing of VS-stage entries, could the incoming vCPU inherit the previous VM's stale VS-stage mappings?
arch/riscv/kvm/mmu.c void kvm_riscv_mmu_update_hgatp(struct kvm_vcpu *vcpu) · line 723
Quoted 1 line
arch/riscv/kvm/mmu.c +1
-0+ asm volatile(HFENCE_VVMA(zero, zero) : : : "memory");
[Severity: High]
Does executing the raw HFENCE.VVMA instruction bypass the Nested Acceleration (NACL) shared memory mechanism?
When executing as an L1 guest with NACL enabled, ncsr_write(CSR_HGATP, hgatp) only updates KVM's shared memory page with L0, not the physical CSR. Since the direct HFENCE.VVMA instruction is executed immediately after on the physical core, does it flush the TLB using the old (stale) VMID still present in the physical CSR?
Additionally, if no flush request is enqueued into the NACL shared memory via nacl_hfence_vvma_all(), will the L0 hypervisor fail to perform the flush for the new VMID before switching to the L2 guest?
Quoted 1 line
arch/riscv/kvm/mmu.c +0
-0}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Anup Patel
maintainerQuoted 21 lines · Yaxing Guo
Hi all,
I would like to ask whether RISC-V KVM has a gap around guest sfence.vma and VS-stage TLB invalidation when a vCPU migrates across host CPUs.
On bare metal, Linux uses
mm_cpumaskin__flush_tlb_range()to decide whether to send a local or remote sfence.vma. That works because the cpumask reflects physical CPUs. In a guest, however, the kernel only sees vCPUs, so from the guest point of view a task may never "migrate" even though the underlying vCPU moved to a different host CPU. In other words, guestmm_cpumaskis effectively a vCPU mask, not a host CPU mask.There is also a related switch_mm()/set_mm_asid() aspect. With the ASID allocator enabled, RISC-V Linux does not flush on every context switch. A
local_flush_tlb_all()is only done when a deferred ASID rollover flush is pending for the current CPU. That is fine on bare metal, because previous shootdowns are also based on physical CPUs. In a guest, however, the earlier shootdown may have been only a guest-local sfence.vma and may have missed an old host CPU where the same vCPU ran previously.
Quoted 7 lines
A simplified sequence is:
1. A guest task with mm A runs on vcpu0 while vcpu0 is scheduled on host CPU1. CPU1 fills a VS-stage TLB entry for that mm/ASID.
2. vcpu0 migrates to host CPU0. KVM's vCPU migration sanitization is local to the current host CPU, so it does not invalidate the VS-stage TLB entry that was left on host CPU1.
When vcpu0 migrates to host CPU0, the host CPU0 may or may not have VS-stage TLB entry so VCPU migration sanitization does the right thing.
In the future, vcpu0 may again move back to host CPU1 containing stale VS-stage TLB entries left in step1 above so the VCPU migration sanitization will cleanup these stale TLB enteries.
Quoted 5 lines
3. The guest later updates mm A's page table from vcpu0, for example due to COW. Since the guest still sees mm A as having run only on guest CPU0,
__flush_tlb_range()can choose a local sfence.vma. That local sfence.vma only affects host CPU0, where vcpu0 is currently running, so the stale VS-stage entry on host CPU1 remains.
No need to worry about stale VS-stage entry on host CPU1 because these will age-out or same guest may again move to host CPU1 resulting in VCPU migration sanitization.
Quoted 4 lines
4. Later, the task is scheduled again while vcpu0 is running on host CPU1. If the ASID is still valid and no deferred ASID rollover flush is pending,
set_mm_asid()may not issue any flush. The access on host CPU1 can then hit the old VS-stage TLB entry.
See above comments.
Quoted 10 lines
We hit this on a XiangShan RISC-V system. A bash task in the guest was scheduled on vcpu0 while that vCPU was running on host CPU1. A read-only page was prefetched and cached in CPU1's VS-stage TLB. Later the same vCPU migrated to host CPU0, the guest triggered COW and updated the mapping, and the guest still issued only a local sfence.vma. The stale VS-stage entry on CPU1 was not invalidated. After running for some time, the task later accessed that mapping while running on host CPU1 again and hit the stale translation. The data in that page was a pointer from the GOT, so it dereferenced to NULL and eventually caused a segmentation fault.
The above example is already taken care by TLB flushes in KVM RISC-V. This smells like some HW errata to me.
Quoted 14 lines
I checked current upstream RISC-V KVM (v7.0.2) and I do not see HSTATUS.VTVM being enabled, so guest sfence.vma is not trapped. I found
kvm_riscv_local_tlb_sanitize()on host CPU migration, which handles G-stage VMID entries on the current host CPU, but I do not think it generally solves the stale VS-stage TLB case described above.I also noticed the vendor-specific
kvm_riscv_vsstage_tlb_no_gpapath in current upstream. It looks like a workaround for a particular implementation issue on Andes AX66, where the VS-stage TLB does not cache guest physical address and VMID, so the normal VMID/GPA-based invalidation model is insufficient. In that sense, it does not seem to be a generic solution for guest sfence.vma coherence across vCPU migration.
Quoted 11 lines
I am wondering what the right direction should be. Enabling HSTATUS.VTVM and trapping guest sfence.vma would allow KVM to translate guest local sfence.vma into the required host-side hfence.vvma, but that may be too heavy for the common case. Another possible direction may be to make the current Andes-specific vCPU migration flush logic more generic, so implementations that need VS-stage TLB sanitization on vCPU migration can opt in through a common mechanism. A third possibility may be to define a lightweight SBI interface for guest local sfence.vma, so Linux could use it in
__flush_tlb_range()for the local case when running under KVM, and KVM could then decide whether a local sfence.vma is sufficient or whether host-side hfence.vvma is also needed.
Enabling HSTATUS.VTVM has a big impact on performance so certainly NACK from myside.
Quoted 3 lines
My questions are:
1. Is the behavior above expected on current RISC-V KVM?
Yes, it should work fine unless there is some HW bug.
Quoted 9 lines
2. If not, what would be the preferred fix direction?
3. Should this be handled by trapping guest sfence.vma with VTVM, by generalizing the existing vCPU-migration VS-stage TLB flush logic, or by adding a lighter SBI-mediated path for guest local sfence.vma?
4. Or is there another intended mechanism to keep VS-stage TLB state coherent across vCPU migration?If needed, I can send a reproducer and the exact hardware/kernel details.
Even if you share some reporducing code sequence, we still have don't have access to your HW so it won't help much.
Regards,
Anup
Quoted 43 lines · Anup Patel
Quoted 21 lines · Yaxing Guo
Hi all,
I would like to ask whether RISC-V KVM has a gap around guest sfence.vma and VS-stage TLB invalidation when a vCPU migrates across host CPUs.
On bare metal, Linux uses
mm_cpumaskin__flush_tlb_range()to decide whether to send a local or remote sfence.vma. That works because the cpumask reflects physical CPUs. In a guest, however, the kernel only sees vCPUs, so from the guest point of view a task may never "migrate" even though the underlying vCPU moved to a different host CPU. In other words, guestmm_cpumaskis effectively a vCPU mask, not a host CPU mask.There is also a related switch_mm()/set_mm_asid() aspect. With the ASID allocator enabled, RISC-V Linux does not flush on every context switch. A
local_flush_tlb_all()is only done when a deferred ASID rollover flush is pending for the current CPU. That is fine on bare metal, because previous shootdowns are also based on physical CPUs. In a guest, however, the earlier shootdown may have been only a guest-local sfence.vma and may have missed an old host CPU where the same vCPU ran previously.Quoted 7 lines
A simplified sequence is:
1. A guest task with mm A runs on vcpu0 while vcpu0 is scheduled on host CPU1. CPU1 fills a VS-stage TLB entry for that mm/ASID.
2. vcpu0 migrates to host CPU0. KVM's vCPU migration sanitization is local to the current host CPU, so it does not invalidate the VS-stage TLB entry that was left on host CPU1.When vcpu0 migrates to host CPU0, the host CPU0 may or may not have VS-stage TLB entry so VCPU migration sanitization does the right thing.
In the future, vcpu0 may again move back to host CPU1 containing stale VS-stage TLB entries left in step1 above so the VCPU migration sanitization will cleanup these stale TLB enteries.
Quoted 5 lines
3. The guest later updates mm A's page table from vcpu0, for example due to COW. Since the guest still sees mm A as having run only on guest CPU0,
__flush_tlb_range()can choose a local sfence.vma. That local sfence.vma only affects host CPU0, where vcpu0 is currently running, so the stale VS-stage entry on host CPU1 remains.
Quoted 3 lines
No need to worry about stale VS-stage entry on host CPU1 because these will age-out or same guest may again move to host CPU1 resulting in VCPU migration sanitization.
void kvm_riscv_local_tlb_sanitize(struct kvm_vcpu *vcpu)
{
unsigned long vmid;
if (!kvm_riscv_gstage_vmid_bits() ||
vcpu->arch.last_exit_cpu == vcpu->cpu)
return;You're right that kvm_riscv_local_tlb_sanitize() can correctly detect a CPU migration via the above check. This check runs before kvm_riscv_vcpu_enter_exit(), so last_exit_cpu still holds the previous CPU and the migration case is properly handled.
vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid);
kvm_riscv_local_hfence_gvma_vmid_all(vmid);The XiangShan team seems to read the specification as not requiring HFENCE.GVMA to invalidate VS-stage TLB entries. Could you please point me to the exact wording in the RISC-V Privileged Specification that indicates otherwise? I would really appreciate the reference.
Quoted 77 lines
Quoted 5 lines
4. Later, the task is scheduled again while vcpu0 is running on host CPU1. If the ASID is still valid and no deferred ASID rollover flush is pending,
set_mm_asid()may not issue any flush. The access on host CPU1 can then hit the old VS-stage TLB entry.See above comments.
Quoted 10 lines
We hit this on a XiangShan RISC-V system. A bash task in the guest was scheduled on vcpu0 while that vCPU was running on host CPU1. A read-only page was prefetched and cached in CPU1's VS-stage TLB. Later the same vCPU migrated to host CPU0, the guest triggered COW and updated the mapping, and the guest still issued only a local sfence.vma. The stale VS-stage entry on CPU1 was not invalidated. After running for some time, the task later accessed that mapping while running on host CPU1 again and hit the stale translation. The data in that page was a pointer from the GOT, so it dereferenced to NULL and eventually caused a segmentation fault.
The above example is already taken care by TLB flushes in KVM RISC-V. This smells like some HW errata to me.
Quoted 14 lines
I checked current upstream RISC-V KVM (v7.0.2) and I do not see HSTATUS.VTVM being enabled, so guest sfence.vma is not trapped. I found
kvm_riscv_local_tlb_sanitize()on host CPU migration, which handles G-stage VMID entries on the current host CPU, but I do not think it generally solves the stale VS-stage TLB case described above.I also noticed the vendor-specific
kvm_riscv_vsstage_tlb_no_gpapath in current upstream. It looks like a workaround for a particular implementation issue on Andes AX66, where the VS-stage TLB does not cache guest physical address and VMID, so the normal VMID/GPA-based invalidation model is insufficient. In that sense, it does not seem to be a generic solution for guest sfence.vma coherence across vCPU migration.Quoted 11 lines
I am wondering what the right direction should be. Enabling HSTATUS.VTVM and trapping guest sfence.vma would allow KVM to translate guest local sfence.vma into the required host-side hfence.vvma, but that may be too heavy for the common case. Another possible direction may be to make the current Andes-specific vCPU migration flush logic more generic, so implementations that need VS-stage TLB sanitization on vCPU migration can opt in through a common mechanism. A third possibility may be to define a lightweight SBI interface for guest local sfence.vma, so Linux could use it in
__flush_tlb_range()for the local case when running under KVM, and KVM could then decide whether a local sfence.vma is sufficient or whether host-side hfence.vvma is also needed.Enabling HSTATUS.VTVM has a big impact on performance so certainly NACK from myside.
Quoted 3 lines
My questions are:
1. Is the behavior above expected on current RISC-V KVM?Yes, it should work fine unless there is some HW bug.
Quoted 9 lines
2. If not, what would be the preferred fix direction?
3. Should this be handled by trapping guest sfence.vma with VTVM, by generalizing the existing vCPU-migration VS-stage TLB flush logic, or by adding a lighter SBI-mediated path for guest local sfence.vma?
4. Or is there another intended mechanism to keep VS-stage TLB state coherent across vCPU migration?If needed, I can send a reproducer and the exact hardware/kernel details.
Even if you share some reporducing code sequence, we still have don't have access to your HW so it won't help much.
Regards,
Anup_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 61 lines · Guo Ren
Quoted 43 lines · Anup Patel
Quoted 21 lines · Yaxing Guo
Hi all,
I would like to ask whether RISC-V KVM has a gap around guest sfence.vma and VS-stage TLB invalidation when a vCPU migrates across host CPUs.
On bare metal, Linux uses
mm_cpumaskin__flush_tlb_range()to decide whether to send a local or remote sfence.vma. That works because the cpumask reflects physical CPUs. In a guest, however, the kernel only sees vCPUs, so from the guest point of view a task may never "migrate" even though the underlying vCPU moved to a different host CPU. In other words, guestmm_cpumaskis effectively a vCPU mask, not a host CPU mask.There is also a related switch_mm()/set_mm_asid() aspect. With the ASID allocator enabled, RISC-V Linux does not flush on every context switch. A
local_flush_tlb_all()is only done when a deferred ASID rollover flush is pending for the current CPU. That is fine on bare metal, because previous shootdowns are also based on physical CPUs. In a guest, however, the earlier shootdown may have been only a guest-local sfence.vma and may have missed an old host CPU where the same vCPU ran previously.Quoted 7 lines
A simplified sequence is:
1. A guest task with mm A runs on vcpu0 while vcpu0 is scheduled on host CPU1. CPU1 fills a VS-stage TLB entry for that mm/ASID.
2. vcpu0 migrates to host CPU0. KVM's vCPU migration sanitization is local to the current host CPU, so it does not invalidate the VS-stage TLB entry that was left on host CPU1.When vcpu0 migrates to host CPU0, the host CPU0 may or may not have VS-stage TLB entry so VCPU migration sanitization does the right thing.
In the future, vcpu0 may again move back to host CPU1 containing stale VS-stage TLB entries left in step1 above so the VCPU migration sanitization will cleanup these stale TLB enteries.
Quoted 5 lines
3. The guest later updates mm A's page table from vcpu0, for example due to COW. Since the guest still sees mm A as having run only on guest CPU0,
__flush_tlb_range()can choose a local sfence.vma. That local sfence.vma only affects host CPU0, where vcpu0 is currently running, so the stale VS-stage entry on host CPU1 remains.Quoted 3 lines
No need to worry about stale VS-stage entry on host CPU1 because these will age-out or same guest may again move to host CPU1 resulting in VCPU migration sanitization.
void
kvm_riscv_local_tlb_sanitize(struct kvm_vcpu *vcpu){ unsigned long vmid; if (!kvm_riscv_gstage_vmid_bits() || vcpu->arch.last_exit_cpu == vcpu->cpu) return;You're right that
kvm_riscv_local_tlb_sanitize()can correctly detect a CPU migration via the above check. This check runs beforekvm_riscv_vcpu_enter_exit(), solast_exit_cpustill holds the previous CPU and the migration case is properly handled.
Quoted 7 lines
vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid); kvm_riscv_local_hfence_gvma_vmid_all(vmid);The XiangShan team seems to read the specification as not requiring HFENCE.GVMA to invalidate VS-stage TLB entries. Could you please point me to the exact wording in the RISC-V Privileged Specification that indicates otherwise? I would really appreciate the reference.
Oh, I found this in the specification:
Conceptually, an implementation might contain two address-translation
caches: one that maps guest virtual addresses to guest physical addresses, and another that maps guest physical addresses to supervisor physical addresses. HFENCE.GVMA need not flush the former cache, but it must flush
^^^^^^ ^^^^^
entries from the latter cache that match the HFENCE.GVMA’s address and
^^^^^^ ^^^^^VMID arguments.
This non-normative text describes two possible hardware designs, which cause HFENCE.GVMA to behave differently:
- In a two-cache design, stale VS-stage entries can remain after
HFENCE.GVMA.
- In a single-cache design (GVA → SPA), HFENCE.GVMA will also clean up the
corresponding stale VS-stage entries.
Quoted 83 lines
Quoted 71 lines
Quoted 6 lines
4. Later, the task is scheduled again while vcpu0 is running on host CPU1. If the ASID is still valid and no deferred ASID rollover flush is pending,
set_mm_asid()may not issue any flush. The access on host CPU1 can then hit the old VS-stage TLB entry.See above comments.
Quoted 10 lines
We hit this on a XiangShan RISC-V system. A bash task in the guest was scheduled on vcpu0 while that vCPU was running on host CPU1. A read-only page was prefetched and cached in CPU1's VS-stage TLB. Later the same vCPU migrated to host CPU0, the guest triggered COW and updated the mapping, and the guest still issued only a local sfence.vma. The stale VS-stage entry on CPU1 was not invalidated. After running for some time, the task later accessed that mapping while running on host CPU1 again and hit the stale translation. The data in that page was a pointer from the GOT, so it dereferenced to NULL and eventually caused a segmentation fault.
The above example is already taken care by TLB flushes in KVM RISC-V. This smells like some HW errata to me.
Quoted 14 lines
I checked current upstream RISC-V KVM (v7.0.2) and I do not see HSTATUS.VTVM being enabled, so guest sfence.vma is not trapped. I found
kvm_riscv_local_tlb_sanitize()on host CPU migration, which handles G-stage VMID entries on the current host CPU, but I do not think it generally solves the stale VS-stage TLB case described above.I also noticed the vendor-specific
kvm_riscv_vsstage_tlb_no_gpapath in current upstream. It looks like a workaround for a particular implementation issue on Andes AX66, where the VS-stage TLB does not cache guest physical address and VMID, so the normal VMID/GPA-based invalidation model is insufficient. In that sense, it does not seem to be a generic solution for guest sfence.vma coherence across vCPU migration.Quoted 11 lines
I am wondering what the right direction should be. Enabling HSTATUS.VTVM and trapping guest sfence.vma would allow KVM to translate guest local sfence.vma into the required host-side hfence.vvma, but that may be too heavy for the common case. Another possible direction may be to make the current Andes-specific vCPU migration flush logic more generic, so implementations that need VS-stage TLB sanitization on vCPU migration can opt in through a common mechanism. A third possibility may be to define a lightweight SBI interface for guest local sfence.vma, so Linux could use it in
__flush_tlb_range()for the local case when running under KVM, and KVM could then decide whether a local sfence.vma is sufficient or whether host-side hfence.vvma is also needed.Enabling HSTATUS.VTVM has a big impact on performance so certainly NACK from myside.
Quoted 3 lines
My questions are:
1. Is the behavior above expected on current RISC-V KVM?Yes, it should work fine unless there is some HW bug.
Quoted 9 lines
2. If not, what would be the preferred fix direction?
3. Should this be handled by trapping guest sfence.vma with VTVM, by generalizing the existing vCPU-migration VS-stage TLB flush logic, or by adding a lighter SBI-mediated path for guest local sfence.vma?
4. Or is there another intended mechanism to keep VS-stage TLB state coherent across vCPU migration?If needed, I can send a reproducer and the exact hardware/kernel details.
Even if you share some reporducing code sequence, we still have don't have access to your HW so it won't help much.
Quoted 7 lines
Regards,
Anup_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 30 lines · Anup Patel
Quoted 21 lines · Yaxing Guo
Hi all,
I would like to ask whether RISC-V KVM has a gap around guest sfence.vma and VS-stage TLB invalidation when a vCPU migrates across host CPUs.
On bare metal, Linux uses
mm_cpumaskin__flush_tlb_range()to decide whether to send a local or remote sfence.vma. That works because the cpumask reflects physical CPUs. In a guest, however, the kernel only sees vCPUs, so from the guest point of view a task may never "migrate" even though the underlying vCPU moved to a different host CPU. In other words, guestmm_cpumaskis effectively a vCPU mask, not a host CPU mask.There is also a related switch_mm()/set_mm_asid() aspect. With the ASID allocator enabled, RISC-V Linux does not flush on every context switch. A
local_flush_tlb_all()is only done when a deferred ASID rollover flush is pending for the current CPU. That is fine on bare metal, because previous shootdowns are also based on physical CPUs. In a guest, however, the earlier shootdown may have been only a guest-local sfence.vma and may have missed an old host CPU where the same vCPU ran previously.Quoted 7 lines
A simplified sequence is:
1. A guest task with mm A runs on vcpu0 while vcpu0 is scheduled on host CPU1. CPU1 fills a VS-stage TLB entry for that mm/ASID.
2. vcpu0 migrates to host CPU0. KVM's vCPU migration sanitization is local to the current host CPU, so it does not invalidate the VS-stage TLB entry that was left on host CPU1.
Quoted 7 lines
When vcpu0 migrates to host CPU0, the host CPU0 may or may not have VS-stage TLB entry so VCPU migration sanitization does the right thing.
In the future, vcpu0 may again move back to host CPU1 containing stale VS-stage TLB entries left in step1 above so the VCPU migration sanitization will cleanup these stale TLB enteries.
Sorry, perhaps I didn't describe it clearly enough. In short, the issue is: because vcpu0 migrated away, cpu1 still has stale TLB entries while cpu0 has the new mapping. After that, neither vcpu migrated again, but the process was scheduled from vcpu0 to vcpu1 (which is on cpu1), causing it to hit the stale TLB. Consider the following scenario:
1. vcpu0 and vcpu1 are both running on cpu1. At this point, vcpu0 is
on cpu1, and a process on it reads a page, which gets cached in the TLB (let's call this the "stale TLB entry").
2. vcpu0 migrates to cpu0, and then the process (on vcpu0, now on
cpu0) does a CoW (copy-on-write) and establishes a new mapping.
3. Neither vcpu0 nor vcpu1 migrates again. At this point, the process
gets scheduled inside the VM onto vcpu1 (which is on cpu1), reads the same page, and hits the stale TLB entry.
Quoted 11 lines
Quoted 5 lines
3. The guest later updates mm A's page table from vcpu0, for example due to COW. Since the guest still sees mm A as having run only on guest CPU0,
__flush_tlb_range()can choose a local sfence.vma. That local sfence.vma only affects host CPU0, where vcpu0 is currently running, so the stale VS-stage entry on host CPU1 remains.No need to worry about stale VS-stage entry on host CPU1 because these will age-out or same guest may again move to host CPU1 resulting in VCPU migration sanitization.
Quoted 1 line
4. Later, the task is scheduled again while vcpu0 is running on host CPU1.
Sorry, that was a typo on my part. 'Later, the task is scheduled again while vcpu0 is running on host CPU1' should say vcpu1 instead of vcpu0.
Quoted 70 lines
Quoted 3 lines
If the ASID is still valid and no deferred ASID rollover flush is pending,
set_mm_asid()may not issue any flush. The access on host CPU1 can then hit the old VS-stage TLB entry.See above comments.
Quoted 10 lines
We hit this on a XiangShan RISC-V system. A bash task in the guest was scheduled on vcpu0 while that vCPU was running on host CPU1. A read-only page was prefetched and cached in CPU1's VS-stage TLB. Later the same vCPU migrated to host CPU0, the guest triggered COW and updated the mapping, and the guest still issued only a local sfence.vma. The stale VS-stage entry on CPU1 was not invalidated. After running for some time, the task later accessed that mapping while running on host CPU1 again and hit the stale translation. The data in that page was a pointer from the GOT, so it dereferenced to NULL and eventually caused a segmentation fault.
The above example is already taken care by TLB flushes in KVM RISC-V. This smells like some HW errata to me.
Quoted 14 lines
I checked current upstream RISC-V KVM (v7.0.2) and I do not see HSTATUS.VTVM being enabled, so guest sfence.vma is not trapped. I found
kvm_riscv_local_tlb_sanitize()on host CPU migration, which handles G-stage VMID entries on the current host CPU, but I do not think it generally solves the stale VS-stage TLB case described above.I also noticed the vendor-specific
kvm_riscv_vsstage_tlb_no_gpapath in current upstream. It looks like a workaround for a particular implementation issue on Andes AX66, where the VS-stage TLB does not cache guest physical address and VMID, so the normal VMID/GPA-based invalidation model is insufficient. In that sense, it does not seem to be a generic solution for guest sfence.vma coherence across vCPU migration.Quoted 11 lines
I am wondering what the right direction should be. Enabling HSTATUS.VTVM and trapping guest sfence.vma would allow KVM to translate guest local sfence.vma into the required host-side hfence.vvma, but that may be too heavy for the common case. Another possible direction may be to make the current Andes-specific vCPU migration flush logic more generic, so implementations that need VS-stage TLB sanitization on vCPU migration can opt in through a common mechanism. A third possibility may be to define a lightweight SBI interface for guest local sfence.vma, so Linux could use it in
__flush_tlb_range()for the local case when running under KVM, and KVM could then decide whether a local sfence.vma is sufficient or whether host-side hfence.vvma is also needed.Enabling HSTATUS.VTVM has a big impact on performance so certainly NACK from myside.
Quoted 3 lines
My questions are:
1. Is the behavior above expected on current RISC-V KVM?Yes, it should work fine unless there is some HW bug.
Quoted 9 lines
2. If not, what would be the preferred fix direction?
3. Should this be handled by trapping guest sfence.vma with VTVM, by generalizing the existing vCPU-migration VS-stage TLB flush logic, or by adding a lighter SBI-mediated path for guest local sfence.vma?
4. Or is there another intended mechanism to keep VS-stage TLB state coherent across vCPU migration?If needed, I can send a reproducer and the exact hardware/kernel details.
Even if you share some reporducing code sequence, we still have don't have access to your HW so it won't help much.
Regards,
Anup
Signature · 3 lines
kvm-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kvm-riscv
+cc wangzhizun<[email protected]>
Quoted 134 lines · guoyaxing
Quoted 30 lines · Anup Patel
Quoted 21 lines · Yaxing Guo
Hi all,
I would like to ask whether RISC-V KVM has a gap around guest sfence.vma and VS-stage TLB invalidation when a vCPU migrates across host CPUs.
On bare metal, Linux uses
mm_cpumaskin__flush_tlb_range()to decide whether to send a local or remote sfence.vma. That works because the cpumask reflects physical CPUs. In a guest, however, the kernel only sees vCPUs, so from the guest point of view a task may never "migrate" even though the underlying vCPU moved to a different host CPU. In other words, guestmm_cpumaskis effectively a vCPU mask, not a host CPU mask.There is also a related switch_mm()/set_mm_asid() aspect. With the ASID allocator enabled, RISC-V Linux does not flush on every context switch. A
local_flush_tlb_all()is only done when a deferred ASID rollover flush is pending for the current CPU. That is fine on bare metal, because previous shootdowns are also based on physical CPUs. In a guest, however, the earlier shootdown may have been only a guest-local sfence.vma and may have missed an old host CPU where the same vCPU ran previously.Quoted 7 lines
A simplified sequence is:
1. A guest task with mm A runs on vcpu0 while vcpu0 is scheduled on host CPU1. CPU1 fills a VS-stage TLB entry for that mm/ASID.
2. vcpu0 migrates to host CPU0. KVM's vCPU migration sanitization is local to the current host CPU, so it does not invalidate the VS-stage TLB entry that was left on host CPU1.Quoted 7 lines
When vcpu0 migrates to host CPU0, the host CPU0 may or may not have VS-stage TLB entry so VCPU migration sanitization does the right thing.
In the future, vcpu0 may again move back to host CPU1 containing stale VS-stage TLB entries left in step1 above so the VCPU migration sanitization will cleanup these stale TLB enteries.
Sorry, perhaps I didn't describe it clearly enough. In short, the issue is: because vcpu0 migrated away, cpu1 still has stale TLB entries while cpu0 has the new mapping. After that, neither vcpu migrated again, but the process was scheduled from vcpu0 to vcpu1 (which is on cpu1), causing it to hit the stale TLB. Consider the following scenario:
1. vcpu0 and vcpu1 are both running on cpu1. At this point, vcpu0 is
on cpu1, and a process on it reads a page, which gets cached in the TLB (let's call this the "stale TLB entry").
2. vcpu0 migrates to cpu0, and then the process (on vcpu0, now on
cpu0) does a CoW (copy-on-write) and establishes a new mapping.
3. Neither vcpu0 nor vcpu1 migrates again. At this point, the process
gets scheduled inside the VM onto vcpu1 (which is on cpu1), reads the same page, and hits the stale TLB entry.Quoted 11 lines
Quoted 5 lines
3. The guest later updates mm A's page table from vcpu0, for example due to COW. Since the guest still sees mm A as having run only on guest CPU0,
__flush_tlb_range()can choose a local sfence.vma. That local sfence.vma only affects host CPU0, where vcpu0 is currently running, so the stale VS-stage entry on host CPU1 remains.No need to worry about stale VS-stage entry on host CPU1 because these will age-out or same guest may again move to host CPU1 resulting in VCPU migration sanitization.
Quoted 1 line
4. Later, the task is scheduled again while vcpu0 is running on host CPU1.
Sorry, that was a typo on my part. 'Later, the task is scheduled again while vcpu0 is running on host CPU1' should say vcpu1 instead of vcpu0.
Quoted 70 lines
Quoted 3 lines
If the ASID is still valid and no deferred ASID rollover flush is pending,
set_mm_asid()may not issue any flush. The access on host CPU1 can then hit the old VS-stage TLB entry.See above comments.
Quoted 10 lines
We hit this on a XiangShan RISC-V system. A bash task in the guest was scheduled on vcpu0 while that vCPU was running on host CPU1. A read-only page was prefetched and cached in CPU1's VS-stage TLB. Later the same vCPU migrated to host CPU0, the guest triggered COW and updated the mapping, and the guest still issued only a local sfence.vma. The stale VS-stage entry on CPU1 was not invalidated. After running for some time, the task later accessed that mapping while running on host CPU1 again and hit the stale translation. The data in that page was a pointer from the GOT, so it dereferenced to NULL and eventually caused a segmentation fault.
The above example is already taken care by TLB flushes in KVM RISC-V. This smells like some HW errata to me.
Quoted 14 lines
I checked current upstream RISC-V KVM (v7.0.2) and I do not see HSTATUS.VTVM being enabled, so guest sfence.vma is not trapped. I found
kvm_riscv_local_tlb_sanitize()on host CPU migration, which handles G-stage VMID entries on the current host CPU, but I do not think it generally solves the stale VS-stage TLB case described above.I also noticed the vendor-specific
kvm_riscv_vsstage_tlb_no_gpapath in current upstream. It looks like a workaround for a particular implementation issue on Andes AX66, where the VS-stage TLB does not cache guest physical address and VMID, so the normal VMID/GPA-based invalidation model is insufficient. In that sense, it does not seem to be a generic solution for guest sfence.vma coherence across vCPU migration.Quoted 11 lines
I am wondering what the right direction should be. Enabling HSTATUS.VTVM and trapping guest sfence.vma would allow KVM to translate guest local sfence.vma into the required host-side hfence.vvma, but that may be too heavy for the common case. Another possible direction may be to make the current Andes-specific vCPU migration flush logic more generic, so implementations that need VS-stage TLB sanitization on vCPU migration can opt in through a common mechanism. A third possibility may be to define a lightweight SBI interface for guest local sfence.vma, so Linux could use it in
__flush_tlb_range()for the local case when running under KVM, and KVM could then decide whether a local sfence.vma is sufficient or whether host- side hfence.vvma is also needed.Enabling HSTATUS.VTVM has a big impact on performance so certainly NACK from myside.
Quoted 3 lines
My questions are:
1. Is the behavior above expected on current RISC-V KVM?Yes, it should work fine unless there is some HW bug.
Quoted 9 lines
2. If not, what would be the preferred fix direction?
3. Should this be handled by trapping guest sfence.vma with VTVM, by generalizing the existing vCPU-migration VS-stage TLB flush logic, or by adding a lighter SBI-mediated path for guest local sfence.vma?
4. Or is there another intended mechanism to keep VS-stage TLB state coherent across vCPU migration?If needed, I can send a reproducer and the exact hardware/kernel details.
Even if you share some reporducing code sequence, we still have don't have access to your HW so it won't help much.
Regards,
Anup