LoongArch: KVM: Add PV TLB flush support
Subsystems (5)
v4 cover LoongArch: KVM: Add PV TLB flush support
From: Tao Cui <[email protected]>
This series implements paravirtualized TLB flush for LoongArch KVM guests.
In multi-vCPU KVM guests, remote TLB flushes broadcast IPIs to all target vCPUs, including those preempted by the host. Sending IPIs to preempted vCPUs causes unnecessary VM exits and grows with vCPU count, becoming severe in overcommitted deployments.
Reuse the existing steal-time shared memory page by adding a new KVM_VCPU_FLUSH_TLB flag to the preempted byte. On the guest side, skip IPIs to preempted vCPUs and set the flag via cmpxchg instead. On the host side, when re-entering a vCPU in kvm_update_stolen_time(), check and clear the flag; if set, drop the VPID to trigger a full TLB flush on the next VM entry. No new shared memory page, hypercall, or Kconfig option is needed.
The feature is advertised through the existing KVM PV feature
negotiation: the kernel exposes KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH as a VM-level capability, and userspace (QEMU) sets KVM_FEATURE_PV_TLB_FLUSH in the guest's CPUCFG_KVM_FEATURE mask after probing it. A corresponding QEMU patch ("target/loongarch: Enable PV TLB flush advertisement to the guest") is posted alongside this series.
- Host side: only trace a PV TLB flush request when one is observed.
- (Carried over) Host uses amand_db.w to atomically read and clear the preempted byte; selftest gained input validation and failure cleanup.
Testing: the PV TLB flush path itself is unchanged from v3, so the benchmark numbers below still hold. Note that, unlike v3, the feature must now be enabled by userspace: run a QEMU built with the companion patch (or with -cpu kvm-pv-tlb-flush=on) so the guest actually observes KVM_FEATURE_PV_TLB_FLUSH. Boot a 32-vCPU guest and run the selftest inside it with sleep-idle (PV helps) and busy-spin (PV cannot optimize) modes respectively:
qemu-system-loongarch64 \
-m 4G -smp 32 --cpu la464 --machine virt \
-bios .../QEMU_EFI.fd \
-kernel .../vmlinuz-...-pvtlb-v4+ \
-initrd /tmp/ramdisk_test.gz \
-serial mon:stdio \
-netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
-device virtio-net-pci,netdev=net0 \
-append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" -nographic# PV TLB flush enabled (idle threads sleep, vCPUs get preempted) guest# ./pv_tlb_flush_test 1 31 50000 0
# Baseline (idle threads busy-spin, all vCPUs stay active) guest# ./pv_tlb_flush_test 1 31 50000 1
With PV TLB flush (sleep idle): ~152,285 ns/flush
Without PV TLB flush (busy-spin): ~481,045 ns/flush
Improvement: ~68% latency reduction (~3.2x throughput increase)
Tao Cui (3):
LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
LoongArch: KVM: Implement guest-side PV TLB flush
KVM: selftests: loongarch: Add PV TLB flush performance test
arch/loongarch/include/asm/kvm_host.h | 1 + arch/loongarch/include/asm/kvm_para.h | 9 + arch/loongarch/include/asm/paravirt.h | 21 +++ arch/loongarch/include/uapi/asm/kvm.h | 1 + arch/loongarch/include/uapi/asm/kvm_para.h | 1 + arch/loongarch/kernel/paravirt.c | 60 ++++++ arch/loongarch/kernel/smp.c | 30 +++- arch/loongarch/kvm/trace.h | 15 ++ arch/loongarch/kvm/vcpu.c | 34 +++- arch/loongarch/kvm/vm.c | 3 + .../selftests/kvm/loongarch/pv_tlb_flush_test.c | 194 +++++++++++++++++++++ 11 files changed, 362 insertions(+), 7 deletions(-)
---
Changes in v4:
- Drop the "Preserve auto-enabled PV features on userspace override"
patch: forcing auto-enabled features back on is migration-unsafe, and for PV TLB flush (which cannot degrade gracefully) a missed flush would corrupt the guest. Enablement now follows the usual KVM model -- the kernel advertises KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH and userspace (QEMU) explicitly sets KVM_FEATURE_PV_TLB_FLUSH after probing it; a companion QEMU patch is posted alongside, and the feature can be kept off for an older destination
(-cpu kvm-pv-tlb-flush=off).
Changes in v3:
- Host side: replace amswap_db.w with amand_db.w to atomically read and clear only the preempted byte, preserving the pad bytes for future UAPI use. Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data.
- Host side: move the pv_auto_features OR operation before the compatibility check in kvm_loongarch_cpucfg_set_attr() so that userspace does not need updating for pure kernel-internal PV feature additions.
- Selftest: add input validation, error checking on pthread_create, and cleanup handling on failure.
Changes in v2:
- Host side: replace non-atomic unsafe_get_user + unsafe_put_user with atomic amswap_db.w inline assembly. This fixes two issues:
1) unsafe_put_user failure could skip the TLB flush entirely
2) non-atomic read+write race with guest-side try_cmpxchg could cause FLUSH_TLB requests to be lost
- Guest side: consolidate two separate READ_ONCE calls into a single READ_ONCE to eliminate a TOCTOU race where the host could clear preempted between the two reads. Also switch from byte-sized try_cmpxchg to 32-bit try_cmpxchg on the aligned word containing preempted.
Signature · 1 line
2.43.0
PATCH v4 1/3 LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
From: Tao Cui <[email protected]>
Implement paravirtualized TLB flush for LoongArch KVM guests using the existing steal-time shared memory page.
The mechanism uses the preempted byte in struct kvm_steal_time with an additional KVM_VCPU_FLUSH_TLB flag bit:
- When a guest vCPU needs remote TLB flush but the target vCPU is preempted (not running), it atomically sets KVM_VCPU_FLUSH_TLB in the target's steal-time preempted byte instead of sending an IPI.
- When the host re-enters the target vCPU (kvm_update_stolen_time()), it atomically reads and clears only the preempted byte via amand_db.w with mask ~0xff on the 32-bit aligned word. amand_db.w provides full barrier semantics and avoids the race with guest-side try_cmpxchg(). LoongArch is little-endian, so the preempted byte occupies bits [7:0]; this preserves pad[0..2] for future UAPI extension. If KVM_VCPU_FLUSH_TLB was set, the host drops the vCPU's VPID, which triggers a full TLB flush on the next VM entry via kvm_check_vpid().
- For non-preempted vCPUs, the guest falls back to normal IPI-based flush, avoiding unnecessary VM exits.
Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data when the cache line was last modified by a different core.
This significantly reduces TLB flush overhead in multi-vCPU workloads where target vCPUs are often idle/preempted.
--- arch/loongarch/include/asm/kvm_host.h | 1 + arch/loongarch/include/asm/kvm_para.h | 1 + arch/loongarch/include/uapi/asm/kvm.h | 1 + arch/loongarch/include/uapi/asm/kvm_para.h | 1 + arch/loongarch/kvm/trace.h | 15 ++++++++++ arch/loongarch/kvm/vcpu.c | 34 +++++++++++++++++++++- arch/loongarch/kvm/vm.c | 3 ++ 7 files changed, 55 insertions(+), 1 deletion(-)
arch/loongarch/include/asm/kvm_host.h +1 -0
@@ -168,6 +168,7 @@ enum emulation_result {#define LOONGARCH_PV_FEAT_MASK (BIT(KVM_FEATURE_IPI) | \ BIT(KVM_FEATURE_PREEMPT) | \ BIT(KVM_FEATURE_STEAL_TIME) | \ + BIT(KVM_FEATURE_PV_TLB_FLUSH) |\ BIT(KVM_FEATURE_USER_HCALL) | \ BIT(KVM_FEATURE_VIRT_EXTIOI))
arch/loongarch/include/asm/kvm_para.h +1 -0
@@ -41,6 +41,7 @@ struct kvm_steal_time {__u8 pad[47 ]; };#define KVM_VCPU_PREEMPTED (1 << 0) +#define KVM_VCPU_FLUSH_TLB (1 << 1)/* * Hypercall interface for KVM hypervisor
arch/loongarch/include/uapi/asm/kvm.h +1 -0
@@ -106,6 +106,7 @@ struct kvm_fpu {#define KVM_LOONGARCH_VM_FEAT_PTW 8#define KVM_LOONGARCH_VM_FEAT_MSGINT 9#define KVM_LOONGARCH_VM_FEAT_PV_PREEMPT 10 +#define KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH 11/* Device Control API on vcpu fd */ #define KVM_LOONGARCH_VCPU_CPUCFG 0
arch/loongarch/include/uapi/asm/kvm_para.h +1 -0
@@ -16,6 +16,7 @@#define KVM_FEATURE_IPI 1#define KVM_FEATURE_STEAL_TIME 2#define KVM_FEATURE_PREEMPT 3 +#define KVM_FEATURE_PV_TLB_FLUSH 4/* BIT 24 - 31 are features configurable by user space vmm */ #define KVM_FEATURE_VIRT_EXTIOI 24#define KVM_FEATURE_USER_HCALL 25
arch/loongarch/kvm/trace.h +15 -0
@@ -210,6 +210,21 @@ TRACE_EVENT(kvm_vpid_change,TP_printk ("VPID: 0x%08lx" , __entry->vpid) ); +TRACE_EVENT (kvm_pv_tlb_flush, + TP_PROTO(struct kvm_vcpu *vcpu, bool need_flush), +TP_ARGS (vcpu, need_flush), +TP_STRUCT__entry ( + __field(unsigned int, vcpu_id) +__field (bool, need_flush) + ), +TP_fast_assign ( + __entry->vcpu_id = vcpu->vcpu_id; + __entry->need_flush = need_flush; + ), +TP_printk ("vcpu %u need_flush %u" , __entry->vcpu_id, + __entry->need_flush) +); +#endif /* _TRACE_KVM_H */ #undef TRACE_INCLUDE_PATH
arch/loongarch/kvm/vcpu.c +33 -1
@@ -173,7 +173,39 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) } st = (struct kvm_steal_time __user *)ghc->hva; -if (kvm_guest_has_pv_feature (vcpu,KVM_FEATURE_PREEMPT )) { +if (kvm_guest_has_pv_feature (vcpu,KVM_FEATURE_PV_TLB_FLUSH )) { +u32 old =0 ; +int err =0 ; + +/* +* Prime the cache line with a normal load before the coherent +* atomic below; it was observed (when the line was last dirtied +* by another core) to be needed for amand_db.w to see a current +* value. amand_db.w overwrites `old` with the real pre-AND value, +* so this load contributes only its cache side-effect. +*/ +unsafe_get_user (old, (u32 __user *)&st->preempted, out); + +/* Atomically read and clear the preempted byte via amand_db.w. */ + asmvolatile ( +"1: amand_db.w %1, %3, %2 \n" +"2: \n" + _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) + : "+r" (err), "+&r" (old), +" +ZB " (*(u32 *)&st->preempted) + : "r" ((u32)~0xffu) + : "memory"); + + if (err) + goto out; + + vcpu->arch.st.preempted =0 ; + +if ((u8 )old &KVM_VCPU_FLUSH_TLB ) { + vcpu->arch.vpid =0 ;/* Drop vpid to flush TLB */ +trace_kvm_pv_tlb_flush (vcpu, true); + } + }else if (kvm_guest_has_pv_feature (vcpu,KVM_FEATURE_PREEMPT )) {unsafe_put_user (0 , &st->preempted, out); vcpu->arch.st.preempted =0 ; }
arch/loongarch/kvm/vm.c +3 -0
@@ -54,8 +54,10 @@ static void kvm_vm_init_features(struct kvm *kvm)if (kvm_pvtime_supported ()) { kvm->arch.pv_features |=BIT (KVM_FEATURE_PREEMPT ); kvm->arch.pv_features |=BIT (KVM_FEATURE_STEAL_TIME ); + kvm->arch.pv_features |=BIT (KVM_FEATURE_PV_TLB_FLUSH ); kvm->arch.kvm_features |=BIT (KVM_LOONGARCH_VM_FEAT_PV_PREEMPT ); kvm->arch.kvm_features |=BIT (KVM_LOONGARCH_VM_FEAT_PV_STEALTIME ); + kvm->arch.kvm_features |=BIT (KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH ); } }
@@ -158,6 +160,7 @@ static int kvm_vm_feature_has_attr(struct kvm *kvm, struct kvm_device_attr *attrcase KVM_LOONGARCH_VM_FEAT_PV_IPI :case KVM_LOONGARCH_VM_FEAT_PV_PREEMPT :case KVM_LOONGARCH_VM_FEAT_PV_STEALTIME : +case KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH :if (kvm_vm_support (&kvm->arch, attr->attr))return 0 ;return -ENXIO ;
Signature · 1 line
2.43.0
PATCH v4 2/3 LoongArch: KVM: Implement guest-side PV TLB flush
In reply to Tao CuiFrom: Tao Cui <[email protected]>
Add the guest-side implementation of PV TLB flush for LoongArch KVM guests, complementing the host-side support added in the previous commit.
When running as a KVM guest, remote TLB flushes are optimized by avoiding IPIs to preempted vCPUs:
- kvm_flush_tlb_mask() checks each target vCPU's steal-time preempted flag. If a vCPU is preempted, it atomically sets KVM_VCPU_FLUSH_TLB in the shared preempted byte via cmpxchg and removes that CPU from the IPI mask.
- Only non-preempted vCPUs receive IPIs via on_each_cpu_mask().
- When the host schedules a deferred-flush vCPU back in, it invalidates the VPID and flushes the TLB automatically.
All six SMP TLB flush functions (flush_tlb_all, flush_tlb_mm, flush_tlb_range, flush_tlb_kernel_range, flush_tlb_page, flush_tlb_one) are updated to use the PV path when the static key pv_tlb_flush_key is enabled.
The feature is gated by KVM_FEATURE_PV_TLB_FLUSH and requires KVM_FEATURE_STEAL_TIME (for the shared memory page). Per-CPU cpumask buffers are allocated via arch_initcall.
--- arch/loongarch/include/asm/kvm_para.h | 8 ++++ arch/loongarch/include/asm/paravirt.h | 21 ++++++++++ arch/loongarch/kernel/paravirt.c | 60 +++++++++++++++++++++++++++ arch/loongarch/kernel/smp.c | 30 +++++++++++--- 4 files changed, 113 insertions(+), 6 deletions(-)
arch/loongarch/include/asm/kvm_para.h +8 -0
@@ -187,4 +187,12 @@ static inline bool kvm_check_and_clear_guest_paused(void) return false; } +static inline bool kvm_pv_tlb_flush_supported (void ) +{ +unsigned int feat =kvm_arch_para_features (); + +return (feat & (1U <<KVM_FEATURE_PV_TLB_FLUSH )) && + (feat & (1U <<KVM_FEATURE_STEAL_TIME )); +} +#endif /* _ASM_LOONGARCH_KVM_PARA_H */
arch/loongarch/include/asm/paravirt.h +21 -0
@@ -5,15 +5,26 @@#ifdef CONFIG_PARAVIRT #include <linux/jump_label.h> +#include <linux/cpumask.h> +#include <linux/smp.h> DECLARE_STATIC_KEY_FALSE (virt_preempt_key);DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (pv_tlb_flush_key); DECLARE_PER_CPU(struct kvm_steal_time, steal_time);int __init pv_ipi_init (void );int __init pv_time_init (void );int __init pv_spinlock_init (void ); +void kvm_flush_tlb_mask (const struct cpumask *cpumask, +smp_call_func_t func,void *info); + +static inline bool pv_tlb_flush_enabled (void ) +{ +return static_branch_unlikely (&pv_tlb_flush_key); +} +#else static inline int pv_ipi_init (void )
@@ -31,5 +42,15 @@ static inline int pv_spinlock_init(void)return 0 ; } +static inline bool pv_tlb_flush_enabled (void ) +{ +return false; +} + +static inline void kvm_flush_tlb_mask (const struct cpumask *cpumask, +smp_call_func_t func,void *info) +{ +} +#endif // CONFIG_PARAVIRT #endif
arch/loongarch/kernel/paravirt.c +60 -0
73 lines not on this page · raw message
arch/loongarch/kernel/smp.c +24 -6
@@ -727,7 +727,10 @@ static void flush_tlb_all_ipi(void *info)void flush_tlb_all (void ) { -on_each_cpu (flush_tlb_all_ipi,NULL ,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (cpu_online_mask, flush_tlb_all_ipi,NULL ); +else +on_each_cpu (flush_tlb_all_ipi,NULL ,1 ); }static void flush_tlb_mm_ipi (void *mm)
@@ -743,7 +746,10 @@ void flush_tlb_mm(struct mm_struct *mm)preempt_disable ();if ((atomic_read (&mm->mm_users) !=1 ) || (current->mm != mm)) { -on_each_cpu_mask (mm_cpumask (mm), flush_tlb_mm_ipi, mm,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (mm_cpumask (mm), flush_tlb_mm_ipi, mm); +else +on_each_cpu_mask (mm_cpumask (mm), flush_tlb_mm_ipi, mm,1 ); }else {unsigned int cpu;
@@ -782,7 +788,10 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned l .addr2 = end, }; -on_each_cpu_mask (mm_cpumask (mm), flush_tlb_range_ipi, &fd,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (mm_cpumask (mm), flush_tlb_range_ipi, &fd); +else +on_each_cpu_mask (mm_cpumask (mm), flush_tlb_range_ipi, &fd,1 ); }else {unsigned int cpu;
@@ -809,7 +818,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) .addr2 = end, }; -on_each_cpu (flush_tlb_kernel_range_ipi, &fd,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (cpu_online_mask, flush_tlb_kernel_range_ipi, &fd); +else +on_each_cpu (flush_tlb_kernel_range_ipi, &fd,1 ); }static void flush_tlb_page_ipi (void *info)
@@ -828,7 +840,10 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long page) .addr1 = page, }; -on_each_cpu_mask (mm_cpumask (vma->vm_mm), flush_tlb_page_ipi, &fd,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (mm_cpumask (vma->vm_mm), flush_tlb_page_ipi, &fd); +else +on_each_cpu_mask (mm_cpumask (vma->vm_mm), flush_tlb_page_ipi, &fd,1 ); }else {unsigned int cpu;
@@ -851,6 +866,9 @@ static void flush_tlb_one_ipi(void *info)void flush_tlb_one (unsigned long vaddr) { -on_each_cpu (flush_tlb_one_ipi, (void *)vaddr,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (cpu_online_mask, flush_tlb_one_ipi, (void *)vaddr); +else +on_each_cpu (flush_tlb_one_ipi, (void *)vaddr,1 ); }EXPORT_SYMBOL (flush_tlb_one);
Signature · 1 line
2.43.0
PATCH v4 3/3 KVM: selftests: loongarch: Add PV TLB flush performance test
In reply to Tao CuiFrom: Tao Cui <[email protected]>
Add a multi-threaded benchmark to measure PV TLB flush performance inside LoongArch KVM guests.
The test spawns flusher threads that repeatedly mmap/munmap to trigger TLB shootdown IPIs, alongside idle threads that either sleep or busy-spin. With PV TLB flush enabled, IPIs to preempted vCPUs are replaced by deferred flags in the steal-time shared page.
Usage (inside guest):
./pv_tlb_flush_test <flushers> <idle> <iterations> <busy_idle> busy_idle=0: idle threads sleep (PV can skip IPIs to preempted vCPUs) busy_idle=1: idle threads spin (all vCPUs active, PV cannot optimize)
--- .../kvm/loongarch/pv_tlb_flush_test.c | 194 ++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +194 -0
@@ -0,0 +1,194 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* LoongArch PV TLB Flush Performance Test +* +* Measure the overhead of remote TLB flushes in a KVM guest by spawning +* flusher threads that repeatedly mmap/munmap (triggering TLB shootdown +* IPIs) alongside idle threads that either sleep or busy-spin. +* +* With PV TLB flush enabled, IPIs to preempted vCPUs are replaced by +* deferred flags in the steal-time shared page, reducing flush latency. +* +* Usage: +* Compile on LoongArch guest: +* gcc -O2 -static -pthread -o pv_tlb_flush_test pv_tlb_flush_test.c +* Run (inside KVM guest): +* ./pv_tlb_flush_test <flushers> <idle> <iterations> <busy_idle> +* Examples: +* ./pv_tlb_flush_test 1 31 50000 0 # 1 flusher, 31 sleep, PV helps +* ./pv_tlb_flush_test 1 31 50000 1 # 1 flusher, 31 busy-spin, no PV +* +* busy_idle=0: idle threads sleep, vCPUs get preempted, PV TLB flush +* can skip IPIs to them +* busy_idle=1: idle threads spin, all vCPUs stay active, PV TLB flush +* cannot optimize (baseline for comparison) +*/ +#define _GNU_SOURCE +#include <stdio.h> +#include <stdlib.h> +#include <sys/mman.h> +#include <string.h> +#include <time.h> +#include <unistd.h> +#include <sched.h> +#include <pthread.h> + +#define MEM_SIZE (2*1024*1024) +#define DEFAULT_ITERS 50000 +#define MAX_THREADS 64 + +static int nr_iters =DEFAULT_ITERS ; +static volatile int start_barrier; +static volatile int stop_flag; +static int busy_idle =0 ; + +struct thread_args { +int cpu; +unsigned long *result; +int *completed; +}; + +static inline unsigned long clock_ns (void ) { +struct timespec ts; +clock_gettime (CLOCK_MONOTONIC , &ts); +return (unsigned long )ts.tv_sec *1000000000UL + ts.tv_nsec; +} + +static void pin_cpu (int cpu) { +cpu_set_t set; +if (cpu <0 ) +return ; +CPU_ZERO (&set); +CPU_SET (cpu, &set); +sched_setaffinity (0 ,sizeof (set), &set); +} + +static void *idle_thread (void *arg) { +struct thread_args *ta = arg; +pin_cpu (ta->cpu); +while (!__atomic_load_n (&start_barrier, __ATOMIC_ACQUIRE)); +if (busy_idle) { +volatile long sink =0 ; +while (!__atomic_load_n (&stop_flag, __ATOMIC_ACQUIRE)) + sink++; + }else { +while (!__atomic_load_n (&stop_flag, __ATOMIC_ACQUIRE)) +usleep (1000 ); + } +return NULL ; +} + +static void *flush_thread (void *arg) { +struct thread_args *ta = arg; +unsigned long start, end; +int i; +size_t mem_size =MEM_SIZE ; +pin_cpu (ta->cpu); +while (!__atomic_load_n (&start_barrier, __ATOMIC_ACQUIRE)); + start =clock_ns (); +for (i =0 ; i < nr_iters && !__atomic_load_n (&stop_flag, __ATOMIC_ACQUIRE); i++) { +void *p =mmap (NULL , mem_size,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 ); +if (p ==MAP_FAILED )break ; +for (size_t off =0 ; off < mem_size; off +=65536 ) + ((volatile char *)p)[off] =0 ; +munmap (p, mem_size); + } + end =clock_ns (); + *ta->result = end - start; + *ta->completed = i; +return NULL ; +} + +int main (int argc,char **argv) { +int nr_flush =1 , nr_idle =3 , i, run; +int ncpus =sysconf (_SC_NPROCESSORS_ONLN); +if (argc >1 ) nr_flush =atoi (argv[1 ]); +if (argc >2 ) nr_idle =atoi (argv[2 ]); +if (argc >3 ) nr_iters =atoi (argv[3 ]); +if (argc >4 ) busy_idle =atoi (argv[4 ]); + +if (nr_flush <1 || nr_idle <0 || nr_flush + nr_idle >MAX_THREADS ) { +fprintf (stderr,"Usage: %s [flushers(1-%d)] [idle(0-%d)] [iters] [busy_idle]\n" , + argv[0 ],MAX_THREADS ,MAX_THREADS ); +return 1 ; + } +if (nr_iters <=0 ) { +fprintf (stderr,"Error: iterations must be positive\n" ); +return 1 ; + } + +printf ("=== TLB Flush Benchmark ===\n" ); +printf ("CPUs: %d Flushers: %d Idle: %d Iters: %d Mode: %s\n" , + ncpus, nr_flush, nr_idle, nr_iters, + busy_idle ?"busy-spin" :"sleep" ); + +for (run =0 ; run <3 ; run++) { +int total = nr_flush + nr_idle; +int do_pin = (total <= ncpus); +int created =0 ; +pthread_t threads[MAX_THREADS ]; +unsigned long results[MAX_THREADS ]; +int completed[MAX_THREADS ]; +struct thread_args args[MAX_THREADS ]; + start_barrier =0 ; stop_flag =0 ; + +for (i =0 ; i < nr_idle; i++) { + args[i].cpu = do_pin ? nr_flush + i :-1 ; + args[i].result =NULL ; + args[i].completed =NULL ; +if (pthread_create (&threads[i],NULL , idle_thread, &args[i])) { +perror ("pthread_create idle" ); + goto cleanup; + } + created++; + } +for (i =0 ; i < nr_flush; i++) { +int idx = nr_idle + i; + results[idx] =0 ; + completed[idx] =0 ; + args[idx].cpu = do_pin ? i :-1 ; + args[idx].result = &results[idx]; + args[idx].completed = &completed[idx]; +if (pthread_create (&threads[idx],NULL , flush_thread, &args[idx])) { +perror ("pthread_create flush" ); + goto cleanup; + } + created++; + } + +usleep (10000 ); +__atomic_store_n (&start_barrier,1 , __ATOMIC_RELEASE); +for (i =0 ; i < nr_flush; i++) +pthread_join (threads[nr_idle + i],NULL ); +__atomic_store_n (&stop_flag,1 , __ATOMIC_RELEASE); +for (i =0 ; i < nr_idle; i++) +pthread_join (threads[i],NULL ); + +unsigned long total_ns =0 ; +unsigned long total_done =0 ; +for (i =0 ; i < nr_flush; i++) { +int idx = nr_idle + i; +unsigned long done = completed[idx]; +if (done ==0 ) { +printf (" Run %d flusher %d: no iterations completed\n" , run, i); +continue ; + } +printf (" Run %d flusher %d: %lu ns/flush (%lu iters)\n" , + run, i, results[idx] / done, done); + total_ns += results[idx]; + total_done += done; + } +if (total_done >0 ) +printf (" Run %d Avg: %lu ns/flush\n" , run, total_ns / total_done); +continue ; + +cleanup: +__atomic_store_n (&start_barrier,1 , __ATOMIC_RELEASE); +__atomic_store_n (&stop_flag,1 , __ATOMIC_RELEASE); +for (i =0 ; i < created; i++) +pthread_join (threads[i],NULL ); +return 1 ; + } +return 0 ; +}
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v4 3/3] KVM: selftests: loongarch: Add PV TLB flush performance test
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] Missing Makefile integration for the new test file `pv_tlb_flush_test.c`. - [Medium] Signed…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Missing Makefile integration for the new test file pv_tlb_flush_test.c.
- [Medium] Signed integer overflow in thread bounds check leads to stack buffer overflow. --
commit 56616f9df639ca6715014c26c5864780ced6ff41
Author: Tao Cui <[email protected]>
KVM: selftests: loongarch: Add PV TLB flush performance test
Add a multi-threaded benchmark to measure PV TLB flush performance inside LoongArch KVM guests.
Quoted 12 lines
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +6
-0@@ -0,0 +1,194 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* LoongArch PV TLB Flush Performance Test +* +* Measure the overhead of remote TLB flushes in a KVM guest by spawning +* flusher threads that repeatedly mmap/munmap (triggering TLB shootdown
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c line 16
Quoted 10 lines
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +10
-0+* IPIs) alongside idle threads that either sleep or busy-spin. +* +* With PV TLB flush enabled, IPIs to preempted vCPUs are replaced by +* deferred flags in the steal-time shared page, reducing flush latency. +* +* Usage: +* Compile on LoongArch guest: +* gcc -O2 -static -pthread -o pv_tlb_flush_test pv_tlb_flush_test.c +* Run (inside KVM guest): +* ./pv_tlb_flush_test <flushers> <idle> <iterations> <busy_idle>
[Severity: Medium]
Since this test is provided as a standalone script and not added to the selftests Makefile (e.g., TEST_PROGS or TEST_GEN_PROGS), will it be missed during make install and automated test execution?
Also, is it intentional that this test bypasses the standard KVM selftests framework (kvm_util) which typically provisions VMs and runs guest payloads programmatically?
[ ... ]
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c line 111
Quoted 9 lines
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +9
-0+int main (int argc,char **argv) { +int nr_flush =1 , nr_idle =3 , i, run; +int ncpus =sysconf (_SC_NPROCESSORS_ONLN); +if (argc >1 ) nr_flush =atoi (argv[1 ]); +if (argc >2 ) nr_idle =atoi (argv[2 ]); +if (argc >3 ) nr_iters =atoi (argv[3 ]); +if (argc >4 ) busy_idle =atoi (argv[4 ]); + +if (nr_flush <1 || nr_idle <0 || nr_flush + nr_idle >MAX_THREADS ) {
[Severity: Medium]
Could this bounds check be bypassed if a user provides values that sum to a value exceeding the 32-bit signed integer limit (e.g., nr_flush=1 and nr_idle=2147483647)? The signed addition would overflow to a negative number.
If bypassed, would the program then loop nr_idle times and write out of bounds to the stack-allocated arrays like threads and args in the loop below?
[ ... ]
Quoted 11 lines
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +11
-0+pthread_t threads[MAX_THREADS ]; +unsigned long results[MAX_THREADS ]; +int completed[MAX_THREADS ]; +struct thread_args args[MAX_THREADS ]; + start_barrier =0 ; stop_flag =0 ; + +for (i =0 ; i < nr_idle; i++) { + args[i].cpu = do_pin ? nr_flush + i :-1 ; + args[i].result =NULL ; + args[i].completed =NULL ; +if (pthread_create (&threads[i],NULL , idle_thread, &args[i])) {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
[email protected]
AI reviewRe: [PATCH v4 1/3] LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
In reply to Tao CuiThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] KVM_FEATURE_PV_TLB_FLUSH is unconditionally enabled by default, breaking capability negotiation…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] KVM_FEATURE_PV_TLB_FLUSH is unconditionally enabled by default, breaking capability negotiation and live migration. --
commit f9022e883153b129074818c0b5b2ff53a144e2f5
Author: Tao Cui <[email protected]>
LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
This commit implements paravirtualized TLB flushes for LoongArch guests using the preempted byte in the steal-time shared memory page. It introduces the KVM_VCPU_FLUSH_TLB flag to avoid sending IPIs when the target vCPU is currently preempted.
[ ... ]
arch/loongarch/kvm/vm.c static void kvm_vm_init_features(struct kvm *kvm) · line 60
Quoted 12 lines
arch/loongarch/kvm/vm.c +2
-0@@ -54,8 +54,10 @@ static void kvm_vm_init_features(struct kvm *kvm)if (kvm_pvtime_supported ()) { kvm->arch.pv_features |=BIT (KVM_FEATURE_PREEMPT ); kvm->arch.pv_features |=BIT (KVM_FEATURE_STEAL_TIME ); + kvm->arch.pv_features |=BIT (KVM_FEATURE_PV_TLB_FLUSH ); kvm->arch.kvm_features |=BIT (KVM_LOONGARCH_VM_FEAT_PV_PREEMPT ); kvm->arch.kvm_features |=BIT (KVM_LOONGARCH_VM_FEAT_PV_STEALTIME ); + kvm->arch.kvm_features |=BIT (KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH );
[Severity: High]
Should this new guest-visible feature be disabled by default?
If a VM starts on a kernel with this feature enabled by default, the guest will detect and use PV TLB flushes. If that VM is later live-migrated to an older kernel that lacks PV TLB flush support, the destination host will silently ignore the KVM_VCPU_FLUSH_TLB flag in the steal-time structure.
Could this lead to the guest assuming its TLB was flushed when it wasn't, resulting in stale translations or memory corruption? Would it be safer to require an explicit userspace capability enablement before exposing this to the guest?
Quoted 2 lines
arch/loongarch/kvm/vm.c +0
-0} }
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Bibo Mao
maintainerRe: [PATCH v4 3/3] KVM: selftests: loongarch: Add PV TLB flush performance test
In reply to Tao CuiHi Tao,
Thanks for you patch.
Now LoongArch KVM is just workable and not used with wide scale, I am a little worried the stability of this PV feature.
Do you have real work load to show the benefits of the PV TLB flush, and which TLB APIs should be hooked.
Regards
Bibo Mao
Quoted 222 lines · Tao Cui
From: Tao Cui <[email protected]>
Add a multi-threaded benchmark to measure PV TLB flush performance inside LoongArch KVM guests.
The test spawns flusher threads that repeatedly mmap/munmap to trigger TLB shootdown IPIs, alongside idle threads that either sleep or busy-spin. With PV TLB flush enabled, IPIs to preempted vCPUs are replaced by deferred flags in the steal-time shared page.
Usage (inside guest):
./pv_tlb_flush_test <flushers> <idle> <iterations> <busy_idle> busy_idle=0: idle threads sleep (PV can skip IPIs to preempted vCPUs) busy_idle=1: idle threads spin (all vCPUs active, PV cannot optimize)Signed-off-by Tao Cui <[email protected]>--- .../kvm/loongarch/pv_tlb_flush_test.c | 194 ++++++++++++++++++ 1 file changed, 194 insertions(+)create mode 100644
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.ctools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +194
-0@@ -0,0 +1,194 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* LoongArch PV TLB Flush Performance Test +* +* Measure the overhead of remote TLB flushes in a KVM guest by spawning +* flusher threads that repeatedly mmap/munmap (triggering TLB shootdown +* IPIs) alongside idle threads that either sleep or busy-spin. +* +* With PV TLB flush enabled, IPIs to preempted vCPUs are replaced by +* deferred flags in the steal-time shared page, reducing flush latency. +* +* Usage: +* Compile on LoongArch guest: +* gcc -O2 -static -pthread -o pv_tlb_flush_test pv_tlb_flush_test.c +* Run (inside KVM guest): +* ./pv_tlb_flush_test <flushers> <idle> <iterations> <busy_idle> +* Examples: +* ./pv_tlb_flush_test 1 31 50000 0 # 1 flusher, 31 sleep, PV helps +* ./pv_tlb_flush_test 1 31 50000 1 # 1 flusher, 31 busy-spin, no PV +* +* busy_idle=0: idle threads sleep, vCPUs get preempted, PV TLB flush +* can skip IPIs to them +* busy_idle=1: idle threads spin, all vCPUs stay active, PV TLB flush +* cannot optimize (baseline for comparison) +*/ +#define _GNU_SOURCE +#include <stdio.h> +#include <stdlib.h> +#include <sys/mman.h> +#include <string.h> +#include <time.h> +#include <unistd.h> +#include <sched.h> +#include <pthread.h> + +#define MEM_SIZE (2*1024*1024) +#define DEFAULT_ITERS 50000 +#define MAX_THREADS 64 + +static int nr_iters =DEFAULT_ITERS ; +static volatile int start_barrier; +static volatile int stop_flag; +static int busy_idle =0 ; + +struct thread_args { +int cpu; +unsigned long *result; +int *completed; +}; + +static inline unsigned long clock_ns (void ) { +struct timespec ts; +clock_gettime (CLOCK_MONOTONIC , &ts); +return (unsigned long )ts.tv_sec *1000000000UL + ts.tv_nsec; +} + +static void pin_cpu (int cpu) { +cpu_set_t set; +if (cpu <0 ) +return ; +CPU_ZERO (&set); +CPU_SET (cpu, &set); +sched_setaffinity (0 ,sizeof (set), &set); +} + +static void *idle_thread (void *arg) { +struct thread_args *ta = arg; +pin_cpu (ta->cpu); +while (!__atomic_load_n (&start_barrier, __ATOMIC_ACQUIRE)); +if (busy_idle) { +volatile long sink =0 ; +while (!__atomic_load_n (&stop_flag, __ATOMIC_ACQUIRE)) + sink++; + }else { +while (!__atomic_load_n (&stop_flag, __ATOMIC_ACQUIRE)) +usleep (1000 ); + } +return NULL ; +} + +static void *flush_thread (void *arg) { +struct thread_args *ta = arg; +unsigned long start, end; +int i; +size_t mem_size =MEM_SIZE ; +pin_cpu (ta->cpu); +while (!__atomic_load_n (&start_barrier, __ATOMIC_ACQUIRE)); + start =clock_ns (); +for (i =0 ; i < nr_iters && !__atomic_load_n (&stop_flag, __ATOMIC_ACQUIRE); i++) { +void *p =mmap (NULL , mem_size,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 ); +if (p ==MAP_FAILED )break ; +for (size_t off =0 ; off < mem_size; off +=65536 ) + ((volatile char *)p)[off] =0 ; +munmap (p, mem_size); + } + end =clock_ns (); + *ta->result = end - start; + *ta->completed = i; +return NULL ; +} + +int main (int argc,char **argv) { +int nr_flush =1 , nr_idle =3 , i, run; +int ncpus =sysconf (_SC_NPROCESSORS_ONLN); +if (argc >1 ) nr_flush =atoi (argv[1 ]); +if (argc >2 ) nr_idle =atoi (argv[2 ]); +if (argc >3 ) nr_iters =atoi (argv[3 ]); +if (argc >4 ) busy_idle =atoi (argv[4 ]); + +if (nr_flush <1 || nr_idle <0 || nr_flush + nr_idle >MAX_THREADS ) { +fprintf (stderr,"Usage: %s [flushers(1-%d)] [idle(0-%d)] [iters] [busy_idle]\n" , + argv[0 ],MAX_THREADS ,MAX_THREADS ); +return 1 ; + } +if (nr_iters <=0 ) { +fprintf (stderr,"Error: iterations must be positive\n" ); +return 1 ; + } + +printf ("=== TLB Flush Benchmark ===\n" ); +printf ("CPUs: %d Flushers: %d Idle: %d Iters: %d Mode: %s\n" , + ncpus, nr_flush, nr_idle, nr_iters, + busy_idle ?"busy-spin" :"sleep" ); + +for (run =0 ; run <3 ; run++) { +int total = nr_flush + nr_idle; +int do_pin = (total <= ncpus); +int created =0 ; +pthread_t threads[MAX_THREADS ]; +unsigned long results[MAX_THREADS ]; +int completed[MAX_THREADS ]; +struct thread_args args[MAX_THREADS ]; + start_barrier =0 ; stop_flag =0 ; + +for (i =0 ; i < nr_idle; i++) { + args[i].cpu = do_pin ? nr_flush + i :-1 ; + args[i].result =NULL ; + args[i].completed =NULL ; +if (pthread_create (&threads[i],NULL , idle_thread, &args[i])) { +perror ("pthread_create idle" ); + goto cleanup; + } + created++; + } +for (i =0 ; i < nr_flush; i++) { +int idx = nr_idle + i; + results[idx] =0 ; + completed[idx] =0 ; + args[idx].cpu = do_pin ? i :-1 ; + args[idx].result = &results[idx]; + args[idx].completed = &completed[idx]; +if (pthread_create (&threads[idx],NULL , flush_thread, &args[idx])) { +perror ("pthread_create flush" ); + goto cleanup; + } + created++; + } + +usleep (10000 ); +__atomic_store_n (&start_barrier,1 , __ATOMIC_RELEASE); +for (i =0 ; i < nr_flush; i++) +pthread_join (threads[nr_idle + i],NULL ); +__atomic_store_n (&stop_flag,1 , __ATOMIC_RELEASE); +for (i =0 ; i < nr_idle; i++) +pthread_join (threads[i],NULL ); + +unsigned long total_ns =0 ; +unsigned long total_done =0 ; +for (i =0 ; i < nr_flush; i++) { +int idx = nr_idle + i; +unsigned long done = completed[idx]; +if (done ==0 ) { +printf (" Run %d flusher %d: no iterations completed\n" , run, i); +continue ; + } +printf (" Run %d flusher %d: %lu ns/flush (%lu iters)\n" , + run, i, results[idx] / done, done); + total_ns += results[idx]; + total_done += done; + } +if (total_done >0 ) +printf (" Run %d Avg: %lu ns/flush\n" , run, total_ns / total_done); +continue ; + +cleanup: +__atomic_store_n (&start_barrier,1 , __ATOMIC_RELEASE); +__atomic_store_n (&stop_flag,1 , __ATOMIC_RELEASE); +for (i =0 ; i < created; i++) +pthread_join (threads[i],NULL ); +return 1 ; + } +return 0 ; +}
Bibo Mao
maintainerRe: [PATCH v4 1/3] LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
In reply to Tao CuiQuoted 126 lines · Tao Cui
From: Tao Cui <[email protected]>
Implement paravirtualized TLB flush for LoongArch KVM guests using the existing steal-time shared memory page.
The mechanism uses the preempted byte in struct
kvm_steal_timewith an additionalKVM_VCPU_FLUSH_TLBflag bit:- When a guest vCPU needs remote TLB flush but the target vCPU is preempted (not running), it atomically sets
KVM_VCPU_FLUSH_TLBin the target's steal-time preempted byte instead of sending an IPI.
- When the host re-enters the target vCPU (kvm_update_stolen_time()), it atomically reads and clears only the preempted byte via amand_db.w with mask ~0xff on the 32-bit aligned word. amand_db.w provides full barrier semantics and avoids the race with guest-sidetry_cmpxchg(). LoongArch is little-endian, so the preempted byte occupies bits [7:0]; this preserves pad[0..2] for future UAPI extension. IfKVM_VCPU_FLUSH_TLBwas set, the host drops the vCPU's VPID, which triggers a full TLB flush on the next VM entry viakvm_check_vpid().
- For non-preempted vCPUs, the guest falls back to normal IPI-based flush, avoiding unnecessary VM exits.Issue a normal load (
unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data when the cache line was last modified by a different core.This significantly reduces TLB flush overhead in multi-vCPU workloads where target vCPUs are often idle/preempted.
Signed-off-by Tao Cui <[email protected]>--- arch/loongarch/include/asm/kvm_host.h | 1 + arch/loongarch/include/asm/kvm_para.h | 1 + arch/loongarch/include/uapi/asm/kvm.h | 1 + arch/loongarch/include/uapi/asm/kvm_para.h | 1 + arch/loongarch/kvm/trace.h | 15 ++++++++++ arch/loongarch/kvm/vcpu.c | 34 +++++++++++++++++++++- arch/loongarch/kvm/vm.c | 3 ++ 7 files changed, 55 insertions(+), 1 deletion(-)arch/loongarch/include/asm/kvm_host.h +1
-0@@ -168,6 +168,7 @@ enum emulation_result {#define LOONGARCH_PV_FEAT_MASK (BIT(KVM_FEATURE_IPI) | \ BIT(KVM_FEATURE_PREEMPT) | \ BIT(KVM_FEATURE_STEAL_TIME) | \ + BIT(KVM_FEATURE_PV_TLB_FLUSH) |\ BIT(KVM_FEATURE_USER_HCALL) | \ BIT(KVM_FEATURE_VIRT_EXTIOI))arch/loongarch/include/asm/kvm_para.h +1
-0@@ -41,6 +41,7 @@ struct kvm_steal_time {__u8 pad[47 ]; };#define KVM_VCPU_PREEMPTED (1 << 0) +#define KVM_VCPU_FLUSH_TLB (1 << 1)/* * Hypercall interface for KVM hypervisor arch/loongarch/include/uapi/asm/kvm.h +1
-0@@ -106,6 +106,7 @@ struct kvm_fpu {#define KVM_LOONGARCH_VM_FEAT_PTW 8#define KVM_LOONGARCH_VM_FEAT_MSGINT 9#define KVM_LOONGARCH_VM_FEAT_PV_PREEMPT 10 +#define KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH 11/* Device Control API on vcpu fd */ #define KVM_LOONGARCH_VCPU_CPUCFG 0arch/loongarch/include/uapi/asm/kvm_para.h +1
-0@@ -16,6 +16,7 @@#define KVM_FEATURE_IPI 1#define KVM_FEATURE_STEAL_TIME 2#define KVM_FEATURE_PREEMPT 3 +#define KVM_FEATURE_PV_TLB_FLUSH 4/* BIT 24 - 31 are features configurable by user space vmm */ #define KVM_FEATURE_VIRT_EXTIOI 24#define KVM_FEATURE_USER_HCALL 25arch/loongarch/kvm/trace.h +15
-0@@ -210,6 +210,21 @@ TRACE_EVENT(kvm_vpid_change,TP_printk ("VPID: 0x%08lx" , __entry->vpid) ); +TRACE_EVENT (kvm_pv_tlb_flush, + TP_PROTO(struct kvm_vcpu *vcpu, bool need_flush), +TP_ARGS (vcpu, need_flush), +TP_STRUCT__entry ( + __field(unsigned int, vcpu_id) +__field (bool, need_flush) + ), +TP_fast_assign ( + __entry->vcpu_id = vcpu->vcpu_id; + __entry->need_flush = need_flush; + ), +TP_printk ("vcpu %u need_flush %u" , __entry->vcpu_id, + __entry->need_flush) +); +#endif /* _TRACE_KVM_H */ #undef TRACE_INCLUDE_PATHarch/loongarch/kvm/vcpu.c +2
-1@@ -173,7 +173,39 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) } st = (struct kvm_steal_time __user *)ghc->hva; -if (kvm_guest_has_pv_feature (vcpu,KVM_FEATURE_PREEMPT )) { +if (kvm_guest_has_pv_feature (vcpu,KVM_FEATURE_PV_TLB_FLUSH )) { +u32 old =0 ;
arch/loongarch/kvm/vcpu.c static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · line 187
Quoted 10 lines
arch/loongarch/kvm/vcpu.c +10
-0+int err =0 ; + +/* +* Prime the cache line with a normal load before the coherent +* atomic below; it was observed (when the line was last dirtied +* by another core) to be needed for amand_db.w to see a current +* value. amand_db.w overwrites `old` with the real pre-AND value, +* so this load contributes only its cache side-effect. +*/ +unsafe_get_user (old, (u32 __user *)&st->preempted, out);
If there is a failure, it does nothing and go to out directly, however guest TLB remains stale without flush operation.
arch/loongarch/kvm/vcpu.c static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · line 200
Quoted 13 lines
arch/loongarch/kvm/vcpu.c +13
-0+ +/* Atomically read and clear the preempted byte via amand_db.w. */ + asmvolatile ( +"1: amand_db.w %1, %3, %2 \n" +"2: \n" + _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) + : "+r" (err), "+&r" (old), +" +ZB " (*(u32 *)&st->preempted) + : "r" ((u32)~0xffu) + : "memory"); + + if (err) + goto out;
Ditto, if there is error, guest TLB is not flushed and vCPU continues to run with preempted bit set. The next TLB flush will be skipped with this vCPU since its preempted bit is not clear.
Regards
Bibo Mao
Quoted 35 lines
arch/loongarch/kvm/vcpu.c +8
-0+ + vcpu->arch.st.preempted =0 ; + +if ((u8 )old &KVM_VCPU_FLUSH_TLB ) { + vcpu->arch.vpid =0 ;/* Drop vpid to flush TLB */ +trace_kvm_pv_tlb_flush (vcpu, true); + } + }else if (kvm_guest_has_pv_feature (vcpu,KVM_FEATURE_PREEMPT )) {unsafe_put_user (0 , &st->preempted, out); vcpu->arch.st.preempted =0 ; }arch/loongarch/kvm/vm.c +3
-0@@ -54,8 +54,10 @@ static void kvm_vm_init_features(struct kvm *kvm)if (kvm_pvtime_supported ()) { kvm->arch.pv_features |=BIT (KVM_FEATURE_PREEMPT ); kvm->arch.pv_features |=BIT (KVM_FEATURE_STEAL_TIME ); + kvm->arch.pv_features |=BIT (KVM_FEATURE_PV_TLB_FLUSH ); kvm->arch.kvm_features |=BIT (KVM_LOONGARCH_VM_FEAT_PV_PREEMPT ); kvm->arch.kvm_features |=BIT (KVM_LOONGARCH_VM_FEAT_PV_STEALTIME ); + kvm->arch.kvm_features |=BIT (KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH ); } }@@ -158,6 +160,7 @@ static int kvm_vm_feature_has_attr(struct kvm *kvm, struct kvm_device_attr *attrcase KVM_LOONGARCH_VM_FEAT_PV_IPI :case KVM_LOONGARCH_VM_FEAT_PV_PREEMPT :case KVM_LOONGARCH_VM_FEAT_PV_STEALTIME : +case KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH :if (kvm_vm_support (&kvm->arch, attr->attr))return 0 ;return -ENXIO ;
Bibo Mao
maintainerRe: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
In reply to Tao CuiQuoted 133 lines · Tao Cui
From: Tao Cui <[email protected]>
Add the guest-side implementation of PV TLB flush for LoongArch KVM guests, complementing the host-side support added in the previous commit.
When running as a KVM guest, remote TLB flushes are optimized by avoiding IPIs to preempted vCPUs:
-
kvm_flush_tlb_mask()checks each target vCPU's steal-time preempted flag. If a vCPU is preempted, it atomically setsKVM_VCPU_FLUSH_TLBin the shared preempted byte via cmpxchg and removes that CPU from the IPI mask.
- Only non-preempted vCPUs receive IPIs viaon_each_cpu_mask().
- When the host schedules a deferred-flush vCPU back in, it invalidates the VPID and flushes the TLB automatically.All six SMP TLB flush functions (
flush_tlb_all,flush_tlb_mm,flush_tlb_range,flush_tlb_kernel_range,flush_tlb_page,flush_tlb_one) are updated to use the PV path when the static keypv_tlb_flush_keyis enabled.The feature is gated by
KVM_FEATURE_PV_TLB_FLUSHand requiresKVM_FEATURE_STEAL_TIME(for the shared memory page). Per-CPU cpumask buffers are allocated viaarch_initcall.Signed-off-by Tao Cui <[email protected]>--- arch/loongarch/include/asm/kvm_para.h | 8 ++++ arch/loongarch/include/asm/paravirt.h | 21 ++++++++++ arch/loongarch/kernel/paravirt.c | 60 +++++++++++++++++++++++++++ arch/loongarch/kernel/smp.c | 30 +++++++++++--- 4 files changed, 113 insertions(+), 6 deletions(-)arch/loongarch/include/asm/kvm_para.h +8
-0@@ -187,4 +187,12 @@ static inline bool kvm_check_and_clear_guest_paused(void) return false; } +static inline bool kvm_pv_tlb_flush_supported (void ) +{ +unsigned int feat =kvm_arch_para_features (); + +return (feat & (1U <<KVM_FEATURE_PV_TLB_FLUSH )) && + (feat & (1U <<KVM_FEATURE_STEAL_TIME )); +} +#endif /* _ASM_LOONGARCH_KVM_PARA_H */ arch/loongarch/include/asm/paravirt.h +21
-0@@ -5,15 +5,26 @@#ifdef CONFIG_PARAVIRT #include <linux/jump_label.h> +#include <linux/cpumask.h> +#include <linux/smp.h> DECLARE_STATIC_KEY_FALSE (virt_preempt_key);DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (pv_tlb_flush_key); DECLARE_PER_CPU(struct kvm_steal_time, steal_time);int __init pv_ipi_init (void );int __init pv_time_init (void );int __init pv_spinlock_init (void ); +void kvm_flush_tlb_mask (const struct cpumask *cpumask, +smp_call_func_t func,void *info); + +static inline bool pv_tlb_flush_enabled (void ) +{ +return static_branch_unlikely (&pv_tlb_flush_key); +} +#else static inline int pv_ipi_init (void )@@ -31,5 +42,15 @@ static inline int pv_spinlock_init(void)return 0 ; } +static inline bool pv_tlb_flush_enabled (void ) +{ +return false; +} + +static inline void kvm_flush_tlb_mask (const struct cpumask *cpumask, +smp_call_func_t func,void *info) +{ +} +#endif // CONFIG_PARAVIRT #endif arch/loongarch/kernel/paravirt.c +20
-0@@ -12,7 +12,9 @@static int has_steal_clock;DEFINE_STATIC_KEY_FALSE (virt_preempt_key);DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (pv_tlb_flush_key); DEFINE_PER_CPU(struct kvm_steal_time, steal_time) __aligned(64); +static DEFINE_PER_CPU(cpumask_var_t, __pv_cpu_mask);static bool steal_acc = true;@@ -208,6 +210,64 @@ int __init pv_ipi_init(void)return 0 ; } +#ifdef CONFIG_SMP +void kvm_flush_tlb_mask (const struct cpumask *cpumask, +smp_call_func_t func,void *info) +{ +int cpu; +struct kvm_steal_time *src; +struct cpumask *flushmask; + +preempt_disable (); + flushmask =this_cpu_cpumask_var_ptr (__pv_cpu_mask); +cpumask_copy (flushmask, cpumask); + +/* +* We have to call flush only on online vCPUs. And +* queue flush_on_enter for pre-empted vCPUs +*/ +for_each_cpu (cpu, flushmask) { +u32 *ptr, old, new;
arch/loongarch/kernel/paravirt.c int __init pv_ipi_init(void) · line 240
Quoted 10 lines
arch/loongarch/kernel/paravirt.c +10
-0+ + src = &per_cpu (steal_time, cpu); + ptr = (u32 *)&src->preempted; + old =READ_ONCE (*ptr); +if (!((u8 )old &KVM_VCPU_PREEMPTED )) +continue ; + + new = old |KVM_VCPU_FLUSH_TLB ; +if (try_cmpxchg (ptr, &old, new)) +__cpumask_clear_cpu (cpu, flushmask);
is it possible that vCPU is running however with KVM_VCPU_PREEMPTED set?
arch/loongarch/kernel/paravirt.c int __init pv_ipi_init(void) · line 243
Quoted 3 lines
arch/loongarch/kernel/paravirt.c +3
-0+ } + +on_each_cpu_mask (flushmask, func, info,1 );
LoongArch VM does not support post interrupt now, IPI with vCPU will trap to hypervisor.
There is another PV TLB flush method using hypercall like PV IPI, which will make a request with wait flag such as KVM_REQ_TLB_FLUSH and kick vCPU.
With this PV TLB flush method, there is not IPI interrupt in VM, and wait is handled in hypervisor, the negative effect is that it flushes all TLBs.
Regards
Bibo Mao
Quoted 106 lines
arch/loongarch/kernel/paravirt.c +27
-0+ preempt_enable(); +} + +static int __init pv_tlb_flush_init (void ) +{ +int cpu; + +if (!kvm_pv_tlb_flush_supported ()) +return 0 ; + +for_each_possible_cpu (cpu) { +if (!zalloc_cpumask_var_node (per_cpu_ptr (&__pv_cpu_mask, cpu), +GFP_KERNEL ,cpu_to_node (cpu))) { +while (--cpu >=0 ) +free_cpumask_var (*per_cpu_ptr (&__pv_cpu_mask, cpu)); +return -ENOMEM ; + } + } + +static_branch_enable (&pv_tlb_flush_key); +pr_info ("KVM setup pv remote TLB flush\n" ); + +return 0 ; +} +arch_initcall (pv_tlb_flush_init); +#endif +static int pv_enable_steal_time (void ) {int cpu =smp_processor_id ();arch/loongarch/kernel/smp.c +24
-6@@ -727,7 +727,10 @@ static void flush_tlb_all_ipi(void *info)void flush_tlb_all (void ) { -on_each_cpu (flush_tlb_all_ipi,NULL ,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (cpu_online_mask, flush_tlb_all_ipi,NULL ); +else +on_each_cpu (flush_tlb_all_ipi,NULL ,1 ); }static void flush_tlb_mm_ipi (void *mm)@@ -743,7 +746,10 @@ void flush_tlb_mm(struct mm_struct *mm)preempt_disable ();if ((atomic_read (&mm->mm_users) !=1 ) || (current->mm != mm)) { -on_each_cpu_mask (mm_cpumask (mm), flush_tlb_mm_ipi, mm,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (mm_cpumask (mm), flush_tlb_mm_ipi, mm); +else +on_each_cpu_mask (mm_cpumask (mm), flush_tlb_mm_ipi, mm,1 ); }else {unsigned int cpu;@@ -782,7 +788,10 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned l .addr2 = end, }; -on_each_cpu_mask (mm_cpumask (mm), flush_tlb_range_ipi, &fd,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (mm_cpumask (mm), flush_tlb_range_ipi, &fd); +else +on_each_cpu_mask (mm_cpumask (mm), flush_tlb_range_ipi, &fd,1 ); }else {unsigned int cpu;@@ -809,7 +818,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) .addr2 = end, }; -on_each_cpu (flush_tlb_kernel_range_ipi, &fd,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (cpu_online_mask, flush_tlb_kernel_range_ipi, &fd); +else +on_each_cpu (flush_tlb_kernel_range_ipi, &fd,1 ); }static void flush_tlb_page_ipi (void *info)@@ -828,7 +840,10 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long page) .addr1 = page, }; -on_each_cpu_mask (mm_cpumask (vma->vm_mm), flush_tlb_page_ipi, &fd,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (mm_cpumask (vma->vm_mm), flush_tlb_page_ipi, &fd); +else +on_each_cpu_mask (mm_cpumask (vma->vm_mm), flush_tlb_page_ipi, &fd,1 ); }else {unsigned int cpu;@@ -851,6 +866,9 @@ static void flush_tlb_one_ipi(void *info)void flush_tlb_one (unsigned long vaddr) { -on_each_cpu (flush_tlb_one_ipi, (void *)vaddr,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (cpu_online_mask, flush_tlb_one_ipi, (void *)vaddr); +else +on_each_cpu (flush_tlb_one_ipi, (void *)vaddr,1 ); }EXPORT_SYMBOL (flush_tlb_one);
Bibo Mao
maintainerRe: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
In reply to Tao CuiQuoted 138 lines · Tao Cui
From: Tao Cui <[email protected]>
Add the guest-side implementation of PV TLB flush for LoongArch KVM guests, complementing the host-side support added in the previous commit.
When running as a KVM guest, remote TLB flushes are optimized by avoiding IPIs to preempted vCPUs:
-
kvm_flush_tlb_mask()checks each target vCPU's steal-time preempted flag. If a vCPU is preempted, it atomically setsKVM_VCPU_FLUSH_TLBin the shared preempted byte via cmpxchg and removes that CPU from the IPI mask.
- Only non-preempted vCPUs receive IPIs viaon_each_cpu_mask().
- When the host schedules a deferred-flush vCPU back in, it invalidates the VPID and flushes the TLB automatically.All six SMP TLB flush functions (
flush_tlb_all,flush_tlb_mm,flush_tlb_range,flush_tlb_kernel_range,flush_tlb_page,flush_tlb_one) are updated to use the PV path when the static keypv_tlb_flush_keyis enabled.The feature is gated by
KVM_FEATURE_PV_TLB_FLUSHand requiresKVM_FEATURE_STEAL_TIME(for the shared memory page). Per-CPU cpumask buffers are allocated viaarch_initcall.Signed-off-by Tao Cui <[email protected]>--- arch/loongarch/include/asm/kvm_para.h | 8 ++++ arch/loongarch/include/asm/paravirt.h | 21 ++++++++++ arch/loongarch/kernel/paravirt.c | 60 +++++++++++++++++++++++++++ arch/loongarch/kernel/smp.c | 30 +++++++++++--- 4 files changed, 113 insertions(+), 6 deletions(-)arch/loongarch/include/asm/kvm_para.h +8
-0@@ -187,4 +187,12 @@ static inline bool kvm_check_and_clear_guest_paused(void) return false; } +static inline bool kvm_pv_tlb_flush_supported (void ) +{ +unsigned int feat =kvm_arch_para_features (); + +return (feat & (1U <<KVM_FEATURE_PV_TLB_FLUSH )) && + (feat & (1U <<KVM_FEATURE_STEAL_TIME )); +} +#endif /* _ASM_LOONGARCH_KVM_PARA_H */ arch/loongarch/include/asm/paravirt.h +21
-0@@ -5,15 +5,26 @@#ifdef CONFIG_PARAVIRT #include <linux/jump_label.h> +#include <linux/cpumask.h> +#include <linux/smp.h> DECLARE_STATIC_KEY_FALSE (virt_preempt_key);DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (pv_tlb_flush_key); DECLARE_PER_CPU(struct kvm_steal_time, steal_time);int __init pv_ipi_init (void );int __init pv_time_init (void );int __init pv_spinlock_init (void ); +void kvm_flush_tlb_mask (const struct cpumask *cpumask, +smp_call_func_t func,void *info); + +static inline bool pv_tlb_flush_enabled (void ) +{ +return static_branch_unlikely (&pv_tlb_flush_key); +} +#else static inline int pv_ipi_init (void )@@ -31,5 +42,15 @@ static inline int pv_spinlock_init(void)return 0 ; } +static inline bool pv_tlb_flush_enabled (void ) +{ +return false; +} + +static inline void kvm_flush_tlb_mask (const struct cpumask *cpumask, +smp_call_func_t func,void *info) +{ +} +#endif // CONFIG_PARAVIRT #endif arch/loongarch/kernel/paravirt.c +25
-0@@ -12,7 +12,9 @@static int has_steal_clock;DEFINE_STATIC_KEY_FALSE (virt_preempt_key);DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (pv_tlb_flush_key); DEFINE_PER_CPU(struct kvm_steal_time, steal_time) __aligned(64); +static DEFINE_PER_CPU(cpumask_var_t, __pv_cpu_mask);static bool steal_acc = true;@@ -208,6 +210,64 @@ int __init pv_ipi_init(void)return 0 ; } +#ifdef CONFIG_SMP +void kvm_flush_tlb_mask (const struct cpumask *cpumask, +smp_call_func_t func,void *info) +{ +int cpu; +struct kvm_steal_time *src; +struct cpumask *flushmask; + +preempt_disable (); + flushmask =this_cpu_cpumask_var_ptr (__pv_cpu_mask); +cpumask_copy (flushmask, cpumask); + +/* +* We have to call flush only on online vCPUs. And +* queue flush_on_enter for pre-empted vCPUs +*/ +for_each_cpu (cpu, flushmask) { +u32 *ptr, old, new; + + src = &per_cpu (steal_time, cpu); + ptr = (u32 *)&src->preempted; + old =READ_ONCE (*ptr); +if (!((u8 )old &KVM_VCPU_PREEMPTED ))
arch/loongarch/kernel/paravirt.c int __init pv_ipi_init(void) · line 245
Quoted 10 lines
arch/loongarch/kernel/paravirt.c +10
-0+continue ; + + new = old |KVM_VCPU_FLUSH_TLB ; +if (try_cmpxchg (ptr, &old, new)) +__cpumask_clear_cpu (cpu, flushmask); + } + +on_each_cpu_mask (flushmask, func, info,1 ); +preempt_enable (); +}
FYI, another PV TLB flush method in arch/x86/hyperv/mmu.c on hyperV which is complicated and fine-gained.
Regards
Bibo Mao
Quoted 104 lines
arch/loongarch/kernel/paravirt.c +25
-0+ +static int __init pv_tlb_flush_init (void ) +{ +int cpu; + +if (!kvm_pv_tlb_flush_supported ()) +return 0 ; + +for_each_possible_cpu (cpu) { +if (!zalloc_cpumask_var_node (per_cpu_ptr (&__pv_cpu_mask, cpu), +GFP_KERNEL ,cpu_to_node (cpu))) { +while (--cpu >=0 ) +free_cpumask_var (*per_cpu_ptr (&__pv_cpu_mask, cpu)); +return -ENOMEM ; + } + } + +static_branch_enable (&pv_tlb_flush_key); +pr_info ("KVM setup pv remote TLB flush\n" ); + +return 0 ; +} +arch_initcall (pv_tlb_flush_init); +#endif +static int pv_enable_steal_time (void ) {int cpu =smp_processor_id ();arch/loongarch/kernel/smp.c +24
-6@@ -727,7 +727,10 @@ static void flush_tlb_all_ipi(void *info)void flush_tlb_all (void ) { -on_each_cpu (flush_tlb_all_ipi,NULL ,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (cpu_online_mask, flush_tlb_all_ipi,NULL ); +else +on_each_cpu (flush_tlb_all_ipi,NULL ,1 ); }static void flush_tlb_mm_ipi (void *mm)@@ -743,7 +746,10 @@ void flush_tlb_mm(struct mm_struct *mm)preempt_disable ();if ((atomic_read (&mm->mm_users) !=1 ) || (current->mm != mm)) { -on_each_cpu_mask (mm_cpumask (mm), flush_tlb_mm_ipi, mm,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (mm_cpumask (mm), flush_tlb_mm_ipi, mm); +else +on_each_cpu_mask (mm_cpumask (mm), flush_tlb_mm_ipi, mm,1 ); }else {unsigned int cpu;@@ -782,7 +788,10 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned l .addr2 = end, }; -on_each_cpu_mask (mm_cpumask (mm), flush_tlb_range_ipi, &fd,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (mm_cpumask (mm), flush_tlb_range_ipi, &fd); +else +on_each_cpu_mask (mm_cpumask (mm), flush_tlb_range_ipi, &fd,1 ); }else {unsigned int cpu;@@ -809,7 +818,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) .addr2 = end, }; -on_each_cpu (flush_tlb_kernel_range_ipi, &fd,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (cpu_online_mask, flush_tlb_kernel_range_ipi, &fd); +else +on_each_cpu (flush_tlb_kernel_range_ipi, &fd,1 ); }static void flush_tlb_page_ipi (void *info)@@ -828,7 +840,10 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long page) .addr1 = page, }; -on_each_cpu_mask (mm_cpumask (vma->vm_mm), flush_tlb_page_ipi, &fd,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (mm_cpumask (vma->vm_mm), flush_tlb_page_ipi, &fd); +else +on_each_cpu_mask (mm_cpumask (vma->vm_mm), flush_tlb_page_ipi, &fd,1 ); }else {unsigned int cpu;@@ -851,6 +866,9 @@ static void flush_tlb_one_ipi(void *info)void flush_tlb_one (unsigned long vaddr) { -on_each_cpu (flush_tlb_one_ipi, (void *)vaddr,1 ); +if (pv_tlb_flush_enabled ()) +kvm_flush_tlb_mask (cpu_online_mask, flush_tlb_one_ipi, (void *)vaddr); +else +on_each_cpu (flush_tlb_one_ipi, (void *)vaddr,1 ); }EXPORT_SYMBOL (flush_tlb_one);
Re: [PATCH v4 1/3] LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
In reply to Bibo MaoHi Bibo,
Thank you for the review.
You're right, and honestly I didn't think the
error path through carefully enough here.
The current handling leaves the preempted byte uncleared and skips the flush, which is exactly the cascade you described: a vCPU re-entered on the error path keeps running with the bit set, so later flushes keep deferring it and its TLB stays stale until some later entry happens to succeed. A stale TLB is a correctness issue, so this path needs conservative handling even though the fault itself is rare.
I'll drop the VPID on error so the TLB is flushed conservatively and the logic self-heals:
if (err) {
vcpu->arch.vpid = 0; /* be conservative: flush anyway */
goto out;
}The byte isn't cleared on the fault, but the TLB is clean this round; the next successful amand_db.w reads the still-set FLUSH_TLB bit and flushes again (redundantly but harmlessly), then clears the byte. The same covers the unsafe_get_user() priming-load failure.
I'll fix this in the next version.
Thanks,
Tao
Quoted 37 lines · Bibo Mao
Quoted 8 lines
arch/loongarch/kvm/vcpu.c +2
-1@@ -173,7 +173,39 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) } st = (struct kvm_steal_time __user *)ghc->hva; - if (kvm_guest_has_pv_feature(vcpu, KVM_FEATURE_PREEMPT)) { + if (kvm_guest_has_pv_feature(vcpu, KVM_FEATURE_PV_TLB_FLUSH)) { + u32 old = 0;arch/loongarch/kvm/vcpu.c static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · line 186
Quoted 10 lines
arch/loongarch/kvm/vcpu.c +10
-0+ int err = 0; + + /* + * Prime the cache line with a normal load before the coherent + * atomic below; it was observed (when the line was last dirtied + * by another core) to be needed for amand_db.w to see a current + * value. amand_db.w overwrites `old` with the real pre-AND value, + * so this load contributes only its cache side-effect. + */ + unsafe_get_user(old, (u32 __user *)&st->preempted, out);If there is a failure, it does nothing and go to out directly, however guest TLB remains stale without flush operation.
arch/loongarch/kvm/vcpu.c static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · line 200
Quoted 13 lines
arch/loongarch/kvm/vcpu.c +13
-0+ + /* Atomically read and clear the preempted byte via amand_db.w. */ + asm volatile( + "1: amand_db.w %1, %3, %2 \n" + "2: \n" + _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) + : "+r" (err), "+&r" (old), + "+ZB" (*(u32 *)&st->preempted) + : "r" ((u32)~0xffu) + : "memory"); + + if (err) + goto out;Ditto, if there is error, guest TLB is not flushed and vCPU continues to run with preempted bit set. The next TLB flush will be skipped with this vCPU since its preempted bit is not clear.
Regards
Bibo Mao
Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
In reply to Bibo MaoHi Bibo,
That's really kind of you. I'd really value the chance to work with you.
Quoted 3 lines · Bibo Mao
Quoted 2 lines
+
__cpumask_clear_cpu(cpu, flushmask);is it possible that vCPU is running however with
KVM_VCPU_PREEMPTEDset?
It shouldn't happen with a correct host: the bit is set on vcpu_put (preempt) and cleared on entry through kvm_update_stolen_time(), so a running vCPU always has preempted == 0. The guest-side race is closed by try_cmpxchg(). If the host clears the byte between the guest's READ_ONCE and the cmpxchg, the cmpxchg fails and the CPU stays in the IPI mask, so no flush is missed. The only thing that can break the invariant is the error path from patch 1, which the conservative handling there closes. Point taken though: the design's correctness rests entirely on the host honoring this, so I agree the patch-1 fix is necessary rather than optional.
Quoted 9 lines
Quoted 3 lines
+ }
+
+on_each_cpu_mask(flushmask, func, info, 1);LoongArch VM does not support post interrupt now, IPI with vCPU will trap to hypervisor.
There is another PV TLB flush method using hypercall like PV IPI, which will make a request with wait flag such as
KVM_REQ_TLB_FLUSHand kick vCPU.With this PV TLB flush method, there is not IPI interrupt in VM, and wait is handled in hypervisor, the negative effect is that it flushes all TLBs.
This is the point I most want to get right, and you're correct. Since LoongArch has no posted interrupts, an IPI to a *running* vCPU still traps, so the steal-time path only genuinely helps for preempted/idle vCPUs; for running ones it's no better (a little heavier) than a plain IPI. The hypercall approach (analogous to our PV IPI) would avoid in-guest IPIs entirely, at the cost of a full TLB flush and hypervisor-side wait handling.
Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.
Quoted 1 line
FYI, another PV TLB flush method in
arch/x86/hyperv/mmu.c...
Thanks for the pointer. The Hyper-V path is a pure hypercall: the guest hands the hypervisor the address space, an address-range list, and the target VP set, so the flush stays fine-grained instead of nuking the whole TLB. That's a good template if the numbers point us toward a hypercall-based design.
Thanks,
Tao
Re: [PATCH v4 3/3] KVM: selftests: loongarch: Add PV TLB flush performance test
In reply to Bibo MaoFrom: Tao Cui <[email protected]>
Thanks, and I share the stability concern. I don't want to land something that adds risk for a feature that isn't clearly earning its place yet.
Quoted 2 lines
Do you have real workload to show the benefits of the PV TLB flush, and which TLB APIs should be hooked.
Completely fair, and to be upfront I don't have a large-scale real-world workload to point to yet. So my plan is to build a set of local test scripts that simulate as many usage scenarios as possible: mmap/fork-heavy paths, kernel-range flushes, and an overcommitted setup where vCPU preemption is frequent. I'll report the IPI and TLB-flush overhead with and without the feature, broken down per flush API. The coverage is synthetic rather than a single flagship workload, but hopefully it can at least give us a rough, preliminary sense of whether the feature is worth pursuing.
Thanks,
Tao
Bibo Mao
maintainerRe: [PATCH v4 3/3] KVM: selftests: loongarch: Add PV TLB flush performance test
Quoted 8 lines · Tao Cui
From: Tao Cui <[email protected]>
Thanks, and I share the stability concern. I don't want to land something that adds risk for a feature that isn't clearly earning its place yet.
Quoted 2 lines
Do you have real workload to show the benefits of the PV TLB flush, and which TLB APIs should be hooked.
Quoted 9 lines
Completely fair, and to be upfront I don't have a large-scale real-world workload to point to yet. So my plan is to build a set of local test scripts that simulate as many usage scenarios as possible: mmap/fork-heavy paths, kernel-range flushes, and an overcommitted setup where vCPU preemption is frequent. I'll report the IPI and TLB-flush overhead with and without the feature, broken down per flush API. The coverage is synthetic rather than a single flagship workload, but hopefully it can at least give us a rough, preliminary sense of whether the feature is worth pursuing.
By the original patch from x86, https://lwn.net/Articles/740363/
There is ebissy benchmark, maybe we can use this test case.
Regards
Bibo Mao
Quoted 4 lines
Thanks,
Tao
Bibo Mao
maintainerRe: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
In reply to Tao CuiQuoted 5 lines · Tao Cui
Hi Bibo,
That's really kind of you. I'd really value the chance to work with you.
Quoted 7 lines
Quoted 3 lines · Bibo Mao
Quoted 2 lines
+
__cpumask_clear_cpu(cpu, flushmask);is it possible that vCPU is running however with
KVM_VCPU_PREEMPTEDset?It shouldn't happen with a correct host: the bit is set on
vcpu_put(preempt) and cleared on entry throughkvm_update_stolen_time(), so a
From the code, I think it may happen. The clear operation may fail in kvm_update_stolen_time(), so vCPU will continue to run in guest mode with flag KVM_VCPU_PREEMPTED set .
Quoted 26 lines
running vCPU always has preempted == 0. The guest-side race is closed by
try_cmpxchg(). If the host clears the byte between the guest'sREAD_ONCEand the cmpxchg, the cmpxchg fails and the CPU stays in the IPI mask, so no flush is missed. The only thing that can break the invariant is the error path from patch 1, which the conservative handling there closes. Point taken though: the design's correctness rests entirely on the host honoring this, so I agree the patch-1 fix is necessary rather than optional.Quoted 9 lines
Quoted 3 lines
+ }
+
+on_each_cpu_mask(flushmask, func, info, 1);LoongArch VM does not support post interrupt now, IPI with vCPU will trap to hypervisor.
There is another PV TLB flush method using hypercall like PV IPI, which will make a request with wait flag such as
KVM_REQ_TLB_FLUSHand kick vCPU.With this PV TLB flush method, there is not IPI interrupt in VM, and wait is handled in hypervisor, the negative effect is that it flushes all TLBs.
This is the point I most want to get right, and you're correct. Since LoongArch has no posted interrupts, an IPI to a *running* vCPU still traps, so the steal-time path only genuinely helps for preempted/idle vCPUs; for running ones it's no better (a little heavier) than a plain IPI. The hypercall approach (analogous to our PV IPI) would avoid in-guest IPIs entirely, at the cost of a full TLB flush and hypervisor-side wait handling.
Quoted 4 lines
Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.
well, that is the best. It is my pleasure to discuss this with you.
Regards
Bibo Mao
Quoted 12 lines
Quoted 2 lines
FYI, another PV TLB flush method in
arch/x86/hyperv/mmu.c...Thanks for the pointer. The Hyper-V path is a pure hypercall: the guest hands the hypervisor the address space, an address-range list, and the target VP set, so the flush stays fine-grained instead of nuking the whole TLB. That's a good template if the numbers point us toward a hypercall-based design.
Thanks,
Tao
Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
Hi Bibo,
Quoted 7 lines · Bibo Mao
Quoted 5 lines
Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.
well, that is the best. It is my pleasure to discuss this with you.
A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.
The two workloads:
- ebizzy (all threads busy, mm-flush heavy)
- tlb_bench in sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)
ebizzy (records/s, higher is better), 32 vCPUs:
no-PV ~103,737 hypercall ~105,779 steal-time ~105,872
-> all within noise (±2%); no measurable difference.
tlb_bench sleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
no-PV ~166,536
steal-time ~149,553
hypercall ~88,686
ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the tlb_bench figure above.
So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.
Best,
Tao
Bibo Mao
maintainerRe: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
Quoted 22 lines · Tao Cui
Hi Bibo,
Quoted 7 lines · Bibo Mao
Quoted 5 lines
Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.
well, that is the best. It is my pleasure to discuss this with you.
A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.
The two workloads:
- ebizzy (all threads busy, mm-flush heavy)
-tlb_benchin sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)
Quoted 5 lines
ebizzy (records/s, higher is better), 32 vCPUs:
no-PV ~103,737 hypercall ~105,779 steal-time ~105,872-> all within noise (±2%); no measurable difference.
Hi Tao,
what is ebizzy command? ebizzy -m or ebizzy -M.
could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?
Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
./ebizzy -m ./ebizzy -M
host 8633 158898 VM(32 vCPUs) 6610 133153 VM/host 76% 83%
Regards
Bibo Mao
Quoted 21 lines
tlb_benchsleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:no-PV ~166,536 steal-time ~149,553 hypercall ~88,686ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the
tlb_benchfigure above.So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.
Best,
Tao
Bibo Mao
maintainerRe: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
Quoted 34 lines · Bibo Mao
Quoted 22 lines · Tao Cui
Hi Bibo,
Quoted 7 lines · Bibo Mao
Quoted 5 lines
Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.
well, that is the best. It is my pleasure to discuss this with you.
A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.
The two workloads:
- ebizzy (all threads busy, mm-flush heavy)
-tlb_benchin sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)Quoted 5 lines
ebizzy (records/s, higher is better), 32 vCPUs:
no-PV ~103,737
hypercall ~105,779
steal-time ~105,872
-> all within noise (±2%); no measurable difference.Hi Tao,
what is ebizzy command? ebizzy -m or ebizzy -M.
Quoted 8 lines
could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?
Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
./ebizzy -m ./ebizzy -M
host 8633 158898
VM(32 vCPUs) 6610 133153
VM/host 76% 83%
just ./ebizzy -M is enough, it seems that CPU number is one key factor.
Quoted 24 lines
Regards
Bibo MaoQuoted 21 lines
tlb_benchsleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
no-PV ~166,536
steal-time ~149,553
hypercall ~88,686ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the
tlb_benchfigure above.So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.
Best,
Tao
Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui
Hi Bibo,
Quoted 7 lines · Bibo Mao
Quoted 5 lines
Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.
well, that is the best. It is my pleasure to discuss this with you.
A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.
The two workloads:
- ebizzy (all threads busy, mm-flush heavy)
-tlb_benchin sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)Quoted 5 lines
ebizzy (records/s, higher is better), 32 vCPUs:
no-PV ~103,737
hypercall ~105,779
steal-time ~105,872
-> all within noise (±2%); no measurable difference.Hi Tao,
what is ebizzy command? ebizzy -m or ebizzy -M.
could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?
Quoted 8 lines
Quoted 6 lines
Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
./ebizzy -m ./ebizzy -M
host 8633 158898
VM(32 vCPUs) 6610 133153
VM/host 76% 83%just ./ebizzy -M is enough, it seems that CPU number is one key factor.
Sorry for the delay — it turned out my ebizzy command was wrong. I had been running ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.
I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).
ebizzy -m ebizzy -M host ~20,000 ~55,000 VM (8 vCPU,1:1) ~17,000 ~53,000 VM/host ~86% ~97%
The -m ratio (86%) is close to your 76%.
I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have CONFIG_PARAVIRT enabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.
I'll share the verified numbers once I have them.
Thanks,
Tao
Quoted 25 lines
Quoted 24 lines
Regards
Bibo MaoQuoted 21 lines
tlb_benchsleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
no-PV ~166,536
steal-time ~149,553
hypercall ~88,686ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the
tlb_benchfigure above.So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.
Best,
Tao
Bibo Mao
maintainerRe: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
Quoted 59 lines · Tao Cui
Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui
Hi Bibo,
Quoted 7 lines · Bibo Mao
Quoted 5 lines
Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.
well, that is the best. It is my pleasure to discuss this with you.
A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.
The two workloads:
- ebizzy (all threads busy, mm-flush heavy)
-tlb_benchin sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)Quoted 5 lines
ebizzy (records/s, higher is better), 32 vCPUs:
no-PV ~103,737
hypercall ~105,779
steal-time ~105,872
-> all within noise (±2%); no measurable difference.Hi Tao,
what is ebizzy command? ebizzy -m or ebizzy -M.
could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?
Quoted 8 lines
Quoted 6 lines
Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
./ebizzy -m ./ebizzy -M
host 8633 158898
VM(32 vCPUs) 6610 133153
VM/host 76% 83%just ./ebizzy -M is enough, it seems that CPU number is one key factor.
Sorry for the delay — it turned out my ebizzy command was wrong. I had been running
ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).
Quoted 6 lines
ebizzy -m ebizzy -M
host ~20,000 ~55,000 VM (8 vCPU,1:1) ~17,000 ~53,000 VM/host ~86% ~97%The -m ratio (86%) is close to your 76%.
On my 3C5000 Dual-way machine, VM has the same CPU/memory topology with physical machine, the kernel is mainline without any patch.
./ebizzy -M Host (32 pCPUs) 158898 One VM(32 vCPUs) 133153 83% of host Two VMs(32 vCPUs each) 9083 + 9630 = 18713 11% of host
It seems that with ebizzy benchmark, there is big difference if vCPU is preempted. Even if vCPU is not preempted, the performance is only 83% of host on my 3C5000 Dual-way machine.
Regards
Bibo Mao
Quoted 41 lines
I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have
CONFIG_PARAVIRTenabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.I'll share the verified numbers once I have them.
Thanks,
TaoQuoted 25 lines
Quoted 24 lines
Regards
Bibo MaoQuoted 21 lines
tlb_benchsleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
no-PV ~166,536
steal-time ~149,553
hypercall ~88,686ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the
tlb_benchfigure above.So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.
Best,
Tao
Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
Quoted 67 lines · Bibo Mao
Quoted 64 lines · Tao Cui
Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui
Hi Bibo,
Quoted 7 lines · Bibo Mao
Quoted 5 lines
Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.
well, that is the best. It is my pleasure to discuss this with you.
A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.
The two workloads:
- ebizzy (all threads busy, mm-flush heavy)
-tlb_benchin sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)Quoted 5 lines
ebizzy (records/s, higher is better), 32 vCPUs:
no-PV ~103,737
hypercall ~105,779
steal-time ~105,872
-> all within noise (±2%); no measurable difference.Hi Tao,
what is ebizzy command? ebizzy -m or ebizzy -M.
could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?
Quoted 8 lines
Quoted 6 lines
Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
./ebizzy -m ./ebizzy -M
host 8633 158898
VM(32 vCPUs) 6610 133153
VM/host 76% 83%just ./ebizzy -M is enough, it seems that CPU number is one key factor.
Sorry for the delay — it turned out my ebizzy command was wrong. I had been running
ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).
ebizzy -m ebizzy -M
host ~20,000 ~55,000
VM (8 vCPU,1:1) ~17,000 ~53,000
VM/host ~86% ~97%
Quoted 8 lines
Quoted 1 line
The -m ratio (86%) is close to your 76%.
On my 3C5000 Dual-way machine, VM has the same CPU/memory topology with physical machine, the kernel is mainline without any patch. ./ebizzy -M
Host (32 pCPUs) 158898
One VM(32 vCPUs) 133153 83% of host
Two VMs(32 vCPUs each) 9083 + 9630 = 18713 11% of hostIt seems that with ebizzy benchmark, there is big difference if vCPU is preempted. Even if vCPU is not preempted, the performance is only 83% of host on my 3C5000 Dual-way machine.
After fixing the ebizzy command, I
have multi-VM overcommit results for both approaches.
Setup: 8-core LoongArch (single-socket, single NUMA), KVM, linux-next-20260623, 8 vCPU per VM. All VMs run ebizzy -M simultaneously, 3 runs each. The PV-off baseline uses a guest kernel with CONFIG_PARAVIRT=y but without the PV TLB flush patches, so PV IPI and steal-time are active in all three columns.
ebizzy -M, total records/s across all VMs:
PV-off steal-time hypercall 1:1 (1VM) 53,600 53,800 53,900 2:1 (2VM) 2,600 42,600 45,300 3:1 (3VM) 2,800 44,700 46,000
At 1:1 there is no difference — no vCPU gets preempted. Under overcommit, without PV TLB flush the throughput drops to ~3-5% of the single-VM case, because every remote TLB flush sends IPIs to preempted vCPUs. With either PV TLB flush variant, preempted vCPUs are skipped, and total throughput stays at ~85-90% of single-VM (bounded by physical cores).
Hypercall is consistently 3-6% above steal-time in the overcommit cases. A possible reason is that the hypercall hands the entire target set to the host in one call, while steal-time still IPIs the running vCPUs and only defers the preempted ones.
On our 8-core machine the collapse is more severe than on your 3C5000 (~5% vs 11% of host at 2 VMs), likely due to the smaller core count and single-NUMA topology.
Thanks,
Tao
Quoted 45 lines
Regards
Bibo MaoQuoted 41 lines
I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have
CONFIG_PARAVIRTenabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.I'll share the verified numbers once I have them.
Thanks,
TaoQuoted 25 lines
Quoted 24 lines
Regards
Bibo MaoQuoted 21 lines
tlb_benchsleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
no-PV ~166,536
steal-time ~149,553
hypercall ~88,686ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the
tlb_benchfigure above.So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.
Best,
Tao
Bibo Mao
maintainerRe: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
Quoted 88 lines · Tao Cui
Quoted 67 lines · Bibo Mao
Quoted 64 lines · Tao Cui
Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui
Hi Bibo,
Quoted 7 lines · Bibo Mao
Quoted 5 lines
Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.
well, that is the best. It is my pleasure to discuss this with you.
A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.
The two workloads:
- ebizzy (all threads busy, mm-flush heavy)
-tlb_benchin sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)Quoted 5 lines
ebizzy (records/s, higher is better), 32 vCPUs:
no-PV ~103,737
hypercall ~105,779
steal-time ~105,872
-> all within noise (±2%); no measurable difference.Hi Tao,
what is ebizzy command? ebizzy -m or ebizzy -M.
could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?
Quoted 8 lines
Quoted 6 lines
Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
./ebizzy -m ./ebizzy -M
host 8633 158898
VM(32 vCPUs) 6610 133153
VM/host 76% 83%just ./ebizzy -M is enough, it seems that CPU number is one key factor.
Sorry for the delay — it turned out my ebizzy command was wrong. I had been running
ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).
ebizzy -m ebizzy -M
host ~20,000 ~55,000
VM (8 vCPU,1:1) ~17,000 ~53,000
VM/host ~86% ~97%Quoted 8 lines
Quoted 1 line
The -m ratio (86%) is close to your 76%.
On my 3C5000 Dual-way machine, VM has the same CPU/memory topology with physical machine, the kernel is mainline without any patch.
./ebizzy -M
Host (32 pCPUs) 158898
One VM(32 vCPUs) 133153 83% of host
Two VMs(32 vCPUs each) 9083 + 9630 = 18713 11% of hostIt seems that with ebizzy benchmark, there is big difference if vCPU is preempted. Even if vCPU is not preempted, the performance is only 83% of host on my 3C5000 Dual-way machine.
After fixing the ebizzy command, I
have multi-VM overcommit results for both approaches.Setup: 8-core LoongArch (single-socket, single NUMA), KVM, linux-next-20260623, 8 vCPU per VM. All VMs run ebizzy -M simultaneously, 3 runs each. The PV-off baseline uses a guest kernel with
CONFIG_PARAVIRT=ybut without the PV TLB flush patches, so PV IPI and steal-time are active in all three columns.
Quoted 8 lines
ebizzy -M, total records/s across all VMs:
PV-off steal-time hypercall 1:1 (1VM) 53,600 53,800 53,900 2:1 (2VM) 2,600 42,600 45,300 3:1 (3VM) 2,800 44,700 46,000At 1:1 there is no difference — no vCPU gets preempted. Under
From the previous result, the score on host is 55,000, with 1:1 mode the virtualization efficiency is 97%, it is hard to improve actually. On my 3C5000 Dual-way machine, the virtualization efficiency is 83%, I think it can improve even with 1:1 mode.
Also the paper at https://dl.acm.org/doi/pdf/10.1145/2892242.2892245 proves it with 1:1 mode.
Quoted 6 lines
overcommit, without PV TLB flush the throughput drops to ~3-5% of the single-VM case, because every remote TLB flush sends IPIs to preempted vCPUs. With either PV TLB flush variant, preempted vCPUs are skipped, and total throughput stays at ~85-90% of single-VM (bounded by physical cores).
Quoted 8 lines
Hypercall is consistently 3-6% above steal-time in the overcommit cases. A possible reason is that the hypercall hands the entire target set to the host in one call, while steal-time still IPIs the running vCPUs and only defers the preempted ones.
On our 8-core machine the collapse is more severe than on your 3C5000 (~5% vs 11% of host at 2 VMs), likely due to the smaller core count and single-NUMA topology.
yeap, with more CPUs the benefit is more high. I think that this kind of patch had better be tested on server platform if there is such hardware by hand after all KVM is used in server platform in most time. I will test the patch in later.
Regards
Bibo Mao
Quoted 49 lines
Thanks,
TaoQuoted 45 lines
Regards
Bibo MaoQuoted 41 lines
I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have
CONFIG_PARAVIRTenabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.I'll share the verified numbers once I have them.
Thanks,
TaoQuoted 25 lines
Quoted 24 lines
Regards
Bibo MaoQuoted 21 lines
tlb_benchsleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
no-PV ~166,536
steal-time ~149,553
hypercall ~88,686ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the
tlb_benchfigure above.So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.
Best,
Tao
Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
Quoted 98 lines · Bibo Mao
Quoted 95 lines · Tao Cui
Quoted 67 lines · Bibo Mao
Quoted 64 lines · Tao Cui
Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui
Hi Bibo,
Quoted 7 lines · Bibo Mao
Quoted 5 lines
Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.
well, that is the best. It is my pleasure to discuss this with you.
A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.
The two workloads:
- ebizzy (all threads busy, mm-flush heavy)
-tlb_benchin sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)Quoted 5 lines
ebizzy (records/s, higher is better), 32 vCPUs:
no-PV ~103,737
hypercall ~105,779
steal-time ~105,872
-> all within noise (±2%); no measurable difference.Hi Tao,
what is ebizzy command? ebizzy -m or ebizzy -M.
could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?
Quoted 8 lines
Quoted 6 lines
Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
./ebizzy -m ./ebizzy -M
host 8633 158898
VM(32 vCPUs) 6610 133153
VM/host 76% 83%just ./ebizzy -M is enough, it seems that CPU number is one key factor.
Sorry for the delay — it turned out my ebizzy command was wrong. I had been running
ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).
ebizzy -m ebizzy -M
host ~20,000 ~55,000
VM (8 vCPU,1:1) ~17,000 ~53,000
VM/host ~86% ~97%Quoted 8 lines
Quoted 1 line
The -m ratio (86%) is close to your 76%.
On my 3C5000 Dual-way machine, VM has the same CPU/memory topology with physical machine, the kernel is mainline without any patch. ./ebizzy -M
Host (32 pCPUs) 158898
One VM(32 vCPUs) 133153 83% of host
Two VMs(32 vCPUs each) 9083 + 9630 = 18713 11% of hostIt seems that with ebizzy benchmark, there is big difference if vCPU is preempted. Even if vCPU is not preempted, the performance is only 83% of host on my 3C5000 Dual-way machine.
After fixing the ebizzy command, I
have multi-VM overcommit results for both approaches.Setup: 8-core LoongArch (single-socket, single NUMA), KVM, linux-next-20260623, 8 vCPU per VM. All VMs run ebizzy -M simultaneously, 3 runs each. The PV-off baseline uses a guest kernel with
CONFIG_PARAVIRT=ybut without the PV TLB flush patches, so PV IPI and steal-time are active in all three columns.ebizzy -M, total records/s across all VMs:
PV-off steal-time hypercall
1:1 (1VM) 53,600 53,800 53,900
2:1 (2VM) 2,600 42,600 45,300
3:1 (3VM) 2,800 44,700 46,000
Quoted 4 lines
Quoted 1 line
At 1:1 there is no difference — no vCPU gets preempted. Under
From the previous result, the score on host is 55,000, with 1:1 mode the virtualization efficiency is 97%, it is hard to improve actually. On my 3C5000 Dual-way machine, the virtualization efficiency is 83%, I think it can improve even with 1:1 mode.
Also the paper at https://dl.acm.org/doi/pdf/10.1145/2892242.2892245 proves it with 1:1 mode.
Thanks for the reference. That makes sense — the 1:1 overhead is larger on multi-socket systems, so there may be room to improve even without overcommit.
Quoted 11 lines
Quoted 11 lines
overcommit, without PV TLB flush the throughput drops to ~3-5% of the single-VM case, because every remote TLB flush sends IPIs to preempted vCPUs. With either PV TLB flush variant, preempted vCPUs are skipped, and total throughput stays at ~85-90% of single-VM (bounded by physical cores).
Hypercall is consistently 3-6% above steal-time in the overcommit cases. A possible reason is that the hypercall hands the entire target set to the host in one call, while steal-time still IPIs the running vCPUs and only defers the preempted ones.
Quoted 5 lines
Quoted 3 lines
On our 8-core machine the collapse is more severe than on your 3C5000 (~5% vs 11% of host at 2 VMs), likely due to the smaller core count and single-NUMA topology.
yeap, with more CPUs the benefit is more high. I think that this kind of patch had better be tested on server platform if there is such hardware by hand after all KVM is used in server platform in most time. I will test the patch in later.
I'll try to get access to a server-class LoongArch machine for further testing. If you're able to run the patches on the 3C5000, that would be great too.
Quoted 52 lines
Regards
Bibo MaoQuoted 49 lines
Thanks,
TaoQuoted 45 lines
Regards
Bibo MaoQuoted 41 lines
I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have
CONFIG_PARAVIRTenabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.I'll share the verified numbers once I have them.
Thanks,
TaoQuoted 25 lines
Quoted 24 lines
Regards
Bibo MaoQuoted 21 lines
tlb_benchsleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
no-PV ~166,536
steal-time ~149,553
hypercall ~88,686ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the
tlb_benchfigure above.So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.
Best,
Tao
Bibo Mao
maintainerRe: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush
Quoted 124 lines · Tao Cui
Quoted 98 lines · Bibo Mao
Quoted 95 lines · Tao Cui
Quoted 67 lines · Bibo Mao
Quoted 64 lines · Tao Cui
Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui
Hi Bibo,
Quoted 7 lines · Bibo Mao
Quoted 5 lines
Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.
well, that is the best. It is my pleasure to discuss this with you.
A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.
The two workloads:
- ebizzy (all threads busy, mm-flush heavy)
-tlb_benchin sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)Quoted 5 lines
ebizzy (records/s, higher is better), 32 vCPUs:
no-PV ~103,737
hypercall ~105,779
steal-time ~105,872
-> all within noise (±2%); no measurable difference.Hi Tao,
what is ebizzy command? ebizzy -m or ebizzy -M.
could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?
Quoted 8 lines
Quoted 6 lines
Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
./ebizzy -m ./ebizzy -M
host 8633 158898
VM(32 vCPUs) 6610 133153
VM/host 76% 83%just ./ebizzy -M is enough, it seems that CPU number is one key factor.
Sorry for the delay — it turned out my ebizzy command was wrong. I had been running
ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).
ebizzy -m ebizzy -M
host ~20,000 ~55,000
VM (8 vCPU,1:1) ~17,000 ~53,000
VM/host ~86% ~97%Quoted 8 lines
Quoted 1 line
The -m ratio (86%) is close to your 76%.
On my 3C5000 Dual-way machine, VM has the same CPU/memory topology with physical machine, the kernel is mainline without any patch.
./ebizzy -M
Host (32 pCPUs) 158898
One VM(32 vCPUs) 133153 83% of host
Two VMs(32 vCPUs each) 9083 + 9630 = 18713 11% of hostIt seems that with ebizzy benchmark, there is big difference if vCPU is preempted. Even if vCPU is not preempted, the performance is only 83% of host on my 3C5000 Dual-way machine.
After fixing the ebizzy command, I
have multi-VM overcommit results for both approaches.Setup: 8-core LoongArch (single-socket, single NUMA), KVM, linux-next-20260623, 8 vCPU per VM. All VMs run ebizzy -M simultaneously, 3 runs each. The PV-off baseline uses a guest kernel with
CONFIG_PARAVIRT=ybut without the PV TLB flush patches, so PV IPI and steal-time are active in all three columns.ebizzy -M, total records/s across all VMs:
PV-off steal-time hypercall 1:1 (1VM) 53,600 53,800 53,900 2:1 (2VM) 2,600 42,600 45,300 3:1 (3VM) 2,800 44,700 46,000
Quoted 4 lines
Quoted 1 line
At 1:1 there is no difference — no vCPU gets preempted. Under
From the previous result, the score on host is 55,000, with 1:1 mode the virtualization efficiency is 97%, it is hard to improve actually. On my 3C5000 Dual-way machine, the virtualization efficiency is 83%, I think it can improve even with 1:1 mode.
Also the paper at https://dl.acm.org/doi/pdf/10.1145/2892242.2892245 proves it with 1:1 mode.
Thanks for the reference. That makes sense — the 1:1 overhead is larger on multi-socket systems, so there may be room to improve even without overcommit.
Quoted 14 lines
Quoted 14 lines
overcommit, without PV TLB flush the throughput drops to ~3-5% of the single-VM case, because every remote TLB flush sends IPIs to preempted vCPUs. With either PV TLB flush variant, preempted vCPUs are skipped, and total throughput stays at ~85-90% of single-VM (bounded by physical cores).
Hypercall is consistently 3-6% above steal-time in the overcommit cases. A possible reason is that the hypercall hands the entire target set to the host in one call, while steal-time still IPIs the running vCPUs and only defers the preempted ones.
On our 8-core machine the collapse is more severe than on your 3C5000 (~5% vs 11% of host at 2 VMs), likely due to the smaller core count and single-NUMA topology.
Quoted 6 lines
Quoted 2 lines
yeap, with more CPUs the benefit is more high. I think that this kind of patch had better be tested on server platform if there is such hardware by hand after all KVM is used in server platform in most time. I will test the patch in later.
I'll try to get access to a server-class LoongArch machine for further testing. If you're able to run the patches on the 3C5000, that would be great too.
Sure, I will verify it on Server board. The flow may be slow since there
are still many pending issues such as:
1. which TLB flush APIs should be hooked
2. On KVM hypervisor side, should fine-grained TLB flush be supported?
all TLBs or TLBs from GVA->GPA, or asid based TLBs flush
3. Or at first use a basic TLB flush method, and then optimize in later.
All these require that there are the same test bed and test cases, then we discuss it based the real performance data.
Regards
Bibo Mao
Quoted 53 lines
Quoted 53 lines
Regards
Bibo MaoQuoted 49 lines
Thanks,
TaoQuoted 45 lines
Regards
Bibo MaoQuoted 41 lines
I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have
CONFIG_PARAVIRTenabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.I'll share the verified numbers once I have them.
Thanks,
TaoQuoted 25 lines
Quoted 24 lines
Regards
Bibo MaoQuoted 21 lines
tlb_benchsleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
no-PV ~166,536
steal-time ~149,553
hypercall ~88,686ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the
tlb_benchfigure above.So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.
Best,
Tao
Bibo Mao
maintainerQuoted 10 lines · Tao Cui
From: Tao Cui <[email protected]>
This series implements paravirtualized TLB flush for LoongArch KVM guests.
In multi-vCPU KVM guests, remote TLB flushes broadcast IPIs to all target vCPUs, including those preempted by the host. Sending IPIs to preempted vCPUs causes unnecessary VM exits and grows with vCPU count, becoming severe in overcommitted deployments.
Quoted 7 lines
Reuse the existing steal-time shared memory page by adding a new
KVM_VCPU_FLUSH_TLBflag to the preempted byte. On the guest side, skip IPIs to preempted vCPUs and set the flag via cmpxchg instead. On the host side, when re-entering a vCPU inkvm_update_stolen_time(), check and clear the flag; if set, drop the VPID to trigger a full TLB flush on the next VM entry. No new shared memory page, hypercall, orKconfigoption is needed.
After a second thought, the PV TLB flush can only work on hardware PTW, there is problem on software PTW.
With TLB flush flow, the current CPU sends IPI interrupter to other CPUs to flush TLB. With software PTW, other CPUs may execute TLB refill or TLB exception with interrupt disabled. IPI interrupt will not trigger until TLB refill/exception finishes, however with PV TLB method, other vCPUs will trap to host and flush TLBs, and continue to execute TLB refill/exception, stale TLB may be added.
Regards
Bibo Mao
Quoted 96 lines
The feature is advertised through the existing KVM PV feature
negotiation: the kernel exposesKVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSHas a VM-level capability, and userspace (QEMU) setsKVM_FEATURE_PV_TLB_FLUSHin the guest'sCPUCFG_KVM_FEATUREmask after probing it. A corresponding QEMU patch ("target/loongarch: Enable PV TLB flush advertisement to the guest") is posted alongside this series.- Host side: only trace a PV TLB flush request when one is observed.
- (Carried over) Host uses amand_db.w to atomically read and clear the preempted byte; selftest gained input validation and failure cleanup.Testing: the PV TLB flush path itself is unchanged from v3, so the benchmark numbers below still hold. Note that, unlike v3, the feature must now be enabled by userspace: run a QEMU built with the companion patch (or with -cpu kvm-pv-tlb-flush=on) so the guest actually observes
KVM_FEATURE_PV_TLB_FLUSH. Boot a 32-vCPU guest and run the selftest inside it with sleep-idle (PV helps) and busy-spin (PV cannot optimize) modes respectively:qemu-system-loongarch64 \ -m 4G -smp 32 --cpu la464 --machine virt \ -bios .../QEMU_EFI.fd \ -kernel .../vmlinuz-...-pvtlb-v4+ \ -initrd /tmp/ramdisk_test.gz \ -serial mon:stdio \ -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \ -device virtio-net-pci,netdev=net0 \ -append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" -nographic# PV TLB flush enabled (idle threads sleep, vCPUs get preempted) guest# ./pv_tlb_flush_test 1 31 50000 0
# Baseline (idle threads busy-spin, all vCPUs stay active) guest# ./pv_tlb_flush_test 1 31 50000 1
With PV TLB flush (sleep idle): ~152,285 ns/flush
Without PV TLB flush (busy-spin): ~481,045 ns/flushImprovement: ~68% latency reduction (~3.2x throughput increase)
Tao Cui (3):
LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
LoongArch: KVM: Implement guest-side PV TLB flush
KVM: selftests: loongarch: Add PV TLB flush performance testarch/loongarch/include/asm/kvm_host.h | 1 + arch/loongarch/include/asm/kvm_para.h | 9 + arch/loongarch/include/asm/paravirt.h | 21 +++ arch/loongarch/include/uapi/asm/kvm.h | 1 + arch/loongarch/include/uapi/asm/kvm_para.h | 1 + arch/loongarch/kernel/paravirt.c | 60 ++++++ arch/loongarch/kernel/smp.c | 30 +++- arch/loongarch/kvm/trace.h | 15 ++ arch/loongarch/kvm/vcpu.c | 34 +++- arch/loongarch/kvm/vm.c | 3 + .../selftests/kvm/loongarch/pv_tlb_flush_test.c | 194 +++++++++++++++++++++ 11 files changed, 362 insertions(+), 7 deletions(-)---
Changes in v4:
- Drop the "Preserve auto-enabled PV features on userspace override"
patch: forcing auto-enabled features back on is migration-unsafe, and for PV TLB flush (which cannot degrade gracefully) a missed flush would corrupt the guest. Enablement now follows the usual KVM model -- the kernel advertisesKVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSHand userspace (QEMU) explicitly setsKVM_FEATURE_PV_TLB_FLUSHafter probing it; a companion QEMU patch is posted alongside, and the feature can be kept off for an older destination
(-cpu kvm-pv-tlb-flush=off).Changes in v3:
- Host side: replace amswap_db.w with amand_db.w to atomically read and clear only the preempted byte, preserving the pad bytes for future UAPI use. Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data.
- Host side: move thepv_auto_featuresOR operation before the compatibility check inkvm_loongarch_cpucfg_set_attr()so that userspace does not need updating for pure kernel-internal PV feature additions.
- Selftest: add input validation, error checking onpthread_create, and cleanup handling on failure.Changes in v2:
- Host side: replace non-atomicunsafe_get_user+unsafe_put_userwith atomic amswap_db.w inline assembly. This fixes two issues:
1)unsafe_put_userfailure could skip the TLB flush entirely
2) non-atomic read+write race with guest-sidetry_cmpxchgcould causeFLUSH_TLBrequests to be lost
- Guest side: consolidate two separateREAD_ONCEcalls into a singleREAD_ONCEto eliminate a TOCTOU race where the host could clear preempted between the two reads. Also switch from byte-sizedtry_cmpxchgto 32-bittry_cmpxchgon the aligned word containing preempted.
Quoted 20 lines · Bibo Mao
Quoted 10 lines · Tao Cui
From: Tao Cui <[email protected]>
This series implements paravirtualized TLB flush for LoongArch KVM guests.
In multi-vCPU KVM guests, remote TLB flushes broadcast IPIs to all target vCPUs, including those preempted by the host. Sending IPIs to preempted vCPUs causes unnecessary VM exits and grows with vCPU count, becoming severe in overcommitted deployments.
Quoted 7 lines
Reuse the existing steal-time shared memory page by adding a new
KVM_VCPU_FLUSH_TLBflag to the preempted byte. On the guest side, skip IPIs to preempted vCPUs and set the flag via cmpxchg instead. On the host side, when re-entering a vCPU inkvm_update_stolen_time(), check and clear the flag; if set, drop the VPID to trigger a full TLB flush on the next VM entry. No new shared memory page, hypercall, orKconfigoption is needed.
Quoted 4 lines
After a second thought, the PV TLB flush can only work on hardware PTW, there is problem on software PTW.
With TLB flush flow, the current CPU sends IPI interrupter to other CPUs to flush TLB. With software PTW, other CPUs may execute TLB refill or TLB exception with interrupt disabled. IPI interrupt will not trigger until TLB refill/exception finishes, however with PV TLB method, other vCPUs will trap to host and flush TLBs, and continue to execute TLB refill/exception, stale TLB may be added.
Sorry for that.
This is indeed something I had overlooked -- thank you for catching it. It is a correctness issue, not just a performance one, so it has to gate the feature.
I understand the race exactly as you describe it. With software PTW, a TLB miss runs the guest's refill handler with interrupts disabled, and that handler may already have loaded the old PTE into a register. A native remote flush sends an IPI, but the IPI is held until IE is re-enabled, so the flush always lands *after* the refill completes -- any stale entry the refill wrote gets purged. The PV path breaks this: the host flushes the target's TLB (drops the VPID) at an arbitrary instant, then the target resumes its disabled refill handler and writes the stale entry back with tlbfill/tlbwr, and nothing flushes it afterwards. The same applies to the steal-time variant if a vCPU is preempted mid-refill:
it is flushed on re-entry and then resumes the refill. Only a hardware PTW is safe, because the refill is atomic and always reads the current table.
So the feature must only be offered when the guest actually uses a hardware PTW. The real condition is on the *guest* -- a -cpu la464 guest still does software refill even on a PTW host -- so the host gate alone is not enough; a non-PTW guest must not observe the feature either.
For the host side, cpu_has_ptw is the natural precondition, and there's already a precedent in kvm_vm_init_features() where KVM_LOONGARCH_VM_FEAT_PTW is gated on cpu_has_ptw. I'd wrap the advertisement the same way:
if (cpu_has_ptw) {
kvm->arch.pv_features |= BIT(KVM_FEATURE_PV_TLB_FLUSH);
kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH);
}For the guest side: the guest discovers the feature by reading the CPUCFG_KVM_FEATURE leaf, so if feasible I'd mask KVM_FEATURE_PV_TLB_FLUSH off right there -- in kvm_emu_cpucfg() (the CPUCFG_KVM_FEATURE case in exit.c) and the matching one_reg getter kvm_loongarch_cpucfg_get_attr() in vcpu.c -- whenever the guest's own CPUCFG2.PTW (vcpu->arch.cpucfg[2], bit 24) is clear. The equivalent could also be done in QEMU alongside the companion patch, since QEMU already controls the guest model's PTW bit. Both look implementable; I'll need to actually test them to see how they behave.
On the hardware side, I checked the box I ran the benchmarks on. It reports as a 3A6000 / LA664, but reading CPUCFG2 directly gives 0x7e7cccc7 with the PTW bit (24) clear, so it has no hardware PTW either -- likely an early part. Your 3C5000 is LA464, which is software PTW too. That means the ebizzy and tlb_bench numbers I shared earlier were collected with PV TLB flush active on a -cpu la464 (software-refill) guest, i.e. exactly the unsafe configuration you pointed out. They show the mechanism works and its direction (hypercall ahead of steal-time, and PV preventing the overcommit collapse), but they were measured in a correctness-broken config, so I won't use them to justify enabling the feature on software-PTW silicon.
For now I don't have a PTW-capable machine to test on. My reading is that a software-PTW-safe variant isn't really possible -- the native IPI is safe precisely because the flush is deferred past the IE=0 refill window, and a host-side flush can't reproduce that ordering without effectively becoming an IPI again. I'll spend some time confirming that, but unless I've missed something, gating to hardware-PTW guests is the answer.
Thanks,
Tao
Quoted 99 lines
Regards
Bibo MaoQuoted 96 lines
The feature is advertised through the existing KVM PV feature
negotiation: the kernel exposesKVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSHas a VM-level capability, and userspace (QEMU) setsKVM_FEATURE_PV_TLB_FLUSHin the guest'sCPUCFG_KVM_FEATUREmask after probing it. A corresponding QEMU patch ("target/loongarch: Enable PV TLB flush advertisement to the guest") is posted alongside this series.- Host side: only trace a PV TLB flush request when one is observed.
- (Carried over) Host uses amand_db.w to atomically read and clear the preempted byte; selftest gained input validation and failure cleanup.Testing: the PV TLB flush path itself is unchanged from v3, so the benchmark numbers below still hold. Note that, unlike v3, the feature must now be enabled by userspace: run a QEMU built with the companion patch (or with -cpu kvm-pv-tlb-flush=on) so the guest actually observes KVM_FEATURE_PV_TLB_FLUSH. Boot a 32-vCPU guest and run the selftest inside it with sleep-idle (PV helps) and busy-spin (PV cannot optimize) modes respectively:
qemu-system-loongarch64 \
-m 4G -smp 32 --cpu la464 --machine virt \
-bios .../QEMU_EFI.fd \
-kernel .../vmlinuz-...-pvtlb-v4+ \
-initrd/tmp/ramdisk_test.gz\
-serial mon:stdio \
-netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
-device virtio-net-pci,netdev=net0 \
-append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" -nographic# PV TLB flush enabled (idle threads sleep, vCPUs get preempted) guest# ./pv_tlb_flush_test 1 31 50000 0
# Baseline (idle threads busy-spin, all vCPUs stay active) guest# ./pv_tlb_flush_test 1 31 50000 1
With PV TLB flush (sleep idle): ~152,285 ns/flush Without PV TLB flush (busy-spin): ~481,045 ns/flush
Improvement: ~68% latency reduction (~3.2x throughput increase)
Tao Cui (3):
LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
LoongArch: KVM: Implement guest-side PV TLB flush
KVM: selftests: loongarch: Add PV TLB flush performance testarch/loongarch/include/asm/kvm_host.h | 1 + arch/loongarch/include/asm/kvm_para.h | 9 + arch/loongarch/include/asm/paravirt.h | 21 +++ arch/loongarch/include/uapi/asm/kvm.h | 1 +arch/loongarch/include/uapi/asm/kvm_para.h | 1 + arch/loongarch/kernel/paravirt.c | 60 ++++++ arch/loongarch/kernel/smp.c | 30 +++- arch/loongarch/kvm/trace.h | 15 ++ arch/loongarch/kvm/vcpu.c | 34 +++- arch/loongarch/kvm/vm.c | 3 + .../selftests/kvm/loongarch/pv_tlb_flush_test.c | 194 +++++++++++++++++++++
11 files changed, 362 insertions(+), 7 deletions(-)---
Changes in v4:
- Drop the "Preserve auto-enabled PV features on userspace override"
patch: forcing auto-enabled features back on is migration-unsafe, and for PV TLB flush (which cannot degrade gracefully) a missed flush would corrupt the guest. Enablement now follows the usual KVM model -- the kernel advertisesKVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSHand userspace (QEMU) explicitly setsKVM_FEATURE_PV_TLB_FLUSHafter probing it; a companion QEMU patch is posted alongside, and the feature can be kept off for an older destination
(-cpu kvm-pv-tlb-flush=off).Changes in v3:
- Host side: replace amswap_db.w with amand_db.w to atomically read and clear only the preempted byte, preserving the pad bytes for future UAPI use. Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data.
- Host side: move thepv_auto_featuresOR operation before the compatibility check inkvm_loongarch_cpucfg_set_attr()so that userspace does not need updating for pure kernel-internal PV feature additions.
- Selftest: add input validation, error checking onpthread_create, and cleanup handling on failure.Changes in v2:
- Host side: replace non-atomicunsafe_get_user+unsafe_put_userwith atomic amswap_db.w inline assembly. This fixes two issues:
1)unsafe_put_userfailure could skip the TLB flush entirely
2) non-atomic read+write race with guest-sidetry_cmpxchgcould causeFLUSH_TLBrequests to be lost
- Guest side: consolidate two separateREAD_ONCEcalls into a singleREAD_ONCEto eliminate a TOCTOU race where the host could clear preempted between the two reads. Also switch from byte-sizedtry_cmpxchgto 32-bittry_cmpxchgon the aligned word containing preempted.
Bibo Mao
maintainerQuoted 85 lines · Tao Cui
Quoted 20 lines · Bibo Mao
Quoted 10 lines · Tao Cui
From: Tao Cui <[email protected]>
This series implements paravirtualized TLB flush for LoongArch KVM guests.
In multi-vCPU KVM guests, remote TLB flushes broadcast IPIs to all target vCPUs, including those preempted by the host. Sending IPIs to preempted vCPUs causes unnecessary VM exits and grows with vCPU count, becoming severe in overcommitted deployments.
Quoted 7 lines
Reuse the existing steal-time shared memory page by adding a new
KVM_VCPU_FLUSH_TLBflag to the preempted byte. On the guest side, skip IPIs to preempted vCPUs and set the flag via cmpxchg instead. On the host side, when re-entering a vCPU inkvm_update_stolen_time(), check and clear the flag; if set, drop the VPID to trigger a full TLB flush on the next VM entry. No new shared memory page, hypercall, orKconfigoption is needed.Quoted 4 lines
After a second thought, the PV TLB flush can only work on hardware PTW, there is problem on software PTW.
With TLB flush flow, the current CPU sends IPI interrupter to other CPUs to flush TLB. With software PTW, other CPUs may execute TLB refill or TLB exception with interrupt disabled. IPI interrupt will not trigger until TLB refill/exception finishes, however with PV TLB method, other vCPUs will trap to host and flush TLBs, and continue to execute TLB refill/exception, stale TLB may be added.
Sorry for that.
This is indeed something I had overlooked -- thank you for catching it. It is a correctness issue, not just a performance one, so it has to gate the feature.
I understand the race exactly as you describe it. With software PTW, a TLB miss runs the guest's refill handler with interrupts disabled, and that handler may already have loaded the old PTE into a register. A native remote flush sends an IPI, but the IPI is held until IE is re-enabled, so the flush always lands *after* the refill completes -- any stale entry the refill wrote gets purged. The PV path breaks this: the host flushes the target's TLB (drops the VPID) at an arbitrary instant, then the target resumes its disabled refill handler and writes the stale entry back with tlbfill/tlbwr, and nothing flushes it afterwards. The same applies to the steal-time variant if a vCPU is preempted mid-refill:
it is flushed on re-entry and then resumes the refill. Only a hardware PTW is safe, because the refill is atomic and always reads the current table.So the feature must only be offered when the guest actually uses a hardware PTW. The real condition is on the *guest* -- a -cpu la464 guest still does software refill even on a PTW host -- so the host gate alone is not enough; a non-PTW guest must not observe the feature either.
For the host side,
cpu_has_ptwis the natural precondition, and there's already a precedent inkvm_vm_init_features()whereKVM_LOONGARCH_VM_FEAT_PTWis gated oncpu_has_ptw. I'd wrap the advertisement the same way:if (cpu_has_ptw) { kvm->arch.pv_features |= BIT(KVM_FEATURE_PV_TLB_FLUSH); kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH); }For the guest side: the guest discovers the feature by reading the
CPUCFG_KVM_FEATUREleaf, so if feasible I'd maskKVM_FEATURE_PV_TLB_FLUSHoff right there -- inkvm_emu_cpucfg()(theCPUCFG_KVM_FEATUREcase inexit.c) and the matchingone_reggetterkvm_loongarch_cpucfg_get_attr()invcpu.c-- whenever the guest's own CPUCFG2.PTW (vcpu->arch.cpucfg[2], bit 24) is clear. The equivalent could also be done in QEMU alongside the companion patch, since QEMU already controls the guest model's PTW bit. Both look implementable; I'll need to actually test them to see how they behave.On the hardware side, I checked the box I ran the benchmarks on. It reports as a 3A6000 / LA664, but reading CPUCFG2 directly gives
0x7e7cccc7with the PTW bit (24) clear, so it has no hardware PTW either -- likely an early part. Your 3C5000 is LA464, which is software PTW too. That means the ebizzy andtlb_benchnumbers I shared earlier were collected with PV TLB flush active on a -cpu la464 (software-refill) guest, i.e. exactly the unsafe configuration you pointed out. They show the mechanism works and its direction (hypercall ahead of steal-time, and PV preventing the overcommit collapse), but they were measured in a correctness-broken config, so I won't use them to justify enabling the feature on software-PTW silicon.
Quoted 6 lines
For now I don't have a PTW-capable machine to test on. My reading is that a software-PTW-safe variant isn't really possible -- the native IPI is safe precisely because the flush is deferred past the IE=0 refill window, and a host-side flush can't reproduce that ordering without effectively becoming an IPI again. I'll spend some time confirming that, but unless I've missed something, gating to hardware-PTW guests is the
PV TLB flush is a good technology, it can be verified again when there is hardware with HW PTW supported.
Regards
Bibo Mao
Quoted 104 lines
answer.
Thanks,
TaoQuoted 99 lines
Regards
Bibo MaoQuoted 96 lines
The feature is advertised through the existing KVM PV feature
negotiation: the kernel exposesKVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSHas a VM-level capability, and userspace (QEMU) setsKVM_FEATURE_PV_TLB_FLUSHin the guest'sCPUCFG_KVM_FEATUREmask after probing it. A corresponding QEMU patch ("target/loongarch: Enable PV TLB flush advertisement to the guest") is posted alongside this series.- Host side: only trace a PV TLB flush request when one is observed.
- (Carried over) Host uses amand_db.w to atomically read and clear the preempted byte; selftest gained input validation and failure cleanup.Testing: the PV TLB flush path itself is unchanged from v3, so the benchmark numbers below still hold. Note that, unlike v3, the feature must now be enabled by userspace: run a QEMU built with the companion patch (or with -cpu kvm-pv-tlb-flush=on) so the guest actually observes KVM_FEATURE_PV_TLB_FLUSH. Boot a 32-vCPU guest and run the selftest inside it with sleep-idle (PV helps) and busy-spin (PV cannot optimize) modes respectively:
qemu-system-loongarch64 \
-m 4G -smp 32 --cpu la464 --machine virt \
-bios .../QEMU_EFI.fd \
-kernel .../vmlinuz-...-pvtlb-v4+ \
-initrd/tmp/ramdisk_test.gz\
-serial mon:stdio \
-netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
-device virtio-net-pci,netdev=net0 \
-append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" -nographic# PV TLB flush enabled (idle threads sleep, vCPUs get preempted) guest# ./pv_tlb_flush_test 1 31 50000 0
# Baseline (idle threads busy-spin, all vCPUs stay active) guest# ./pv_tlb_flush_test 1 31 50000 1
With PV TLB flush (sleep idle): ~152,285 ns/flush Without PV TLB flush (busy-spin): ~481,045 ns/flush
Improvement: ~68% latency reduction (~3.2x throughput increase)
Tao Cui (3):
LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
LoongArch: KVM: Implement guest-side PV TLB flush
KVM: selftests: loongarch: Add PV TLB flush performance testarch/loongarch/include/asm/kvm_host.h | 1 + arch/loongarch/include/asm/kvm_para.h | 9 + arch/loongarch/include/asm/paravirt.h | 21 +++ arch/loongarch/include/uapi/asm/kvm.h | 1 + arch/loongarch/include/uapi/asm/kvm_para.h | 1 + arch/loongarch/kernel/paravirt.c | 60 ++++++ arch/loongarch/kernel/smp.c | 30 +++- arch/loongarch/kvm/trace.h | 15 ++ arch/loongarch/kvm/vcpu.c | 34 +++- arch/loongarch/kvm/vm.c | 3 + .../selftests/kvm/loongarch/pv_tlb_flush_test.c | 194 +++++++++++++++++++++ 11 files changed, 362 insertions(+), 7 deletions(-)---
Changes in v4:
- Drop the "Preserve auto-enabled PV features on userspace override"
patch: forcing auto-enabled features back on is migration-unsafe, and for PV TLB flush (which cannot degrade gracefully) a missed flush would corrupt the guest. Enablement now follows the usual KVM model -- the kernel advertisesKVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSHand userspace (QEMU) explicitly setsKVM_FEATURE_PV_TLB_FLUSHafter probing it; a companion QEMU patch is posted alongside, and the feature can be kept off for an older destination
(-cpu kvm-pv-tlb-flush=off).Changes in v3:
- Host side: replace amswap_db.w with amand_db.w to atomically read and clear only the preempted byte, preserving the pad bytes for future UAPI use. Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data.
- Host side: move thepv_auto_featuresOR operation before the compatibility check inkvm_loongarch_cpucfg_set_attr()so that userspace does not need updating for pure kernel-internal PV feature additions.
- Selftest: add input validation, error checking onpthread_create, and cleanup handling on failure.Changes in v2:
- Host side: replace non-atomicunsafe_get_user+unsafe_put_userwith atomic amswap_db.w inline assembly. This fixes two issues:
1)unsafe_put_userfailure could skip the TLB flush entirely
2) non-atomic read+write race with guest-sidetry_cmpxchgcould causeFLUSH_TLBrequests to be lost
- Guest side: consolidate two separateREAD_ONCEcalls into a singleREAD_ONCEto eliminate a TOCTOU race where the host could clear preempted between the two reads. Also switch from byte-sizedtry_cmpxchgto 32-bittry_cmpxchgon the aligned word containing preempted.
Quoted 88 lines · Bibo Mao
Quoted 85 lines · Tao Cui
Quoted 20 lines · Bibo Mao
Quoted 10 lines · Tao Cui
From: Tao Cui <[email protected]>
This series implements paravirtualized TLB flush for LoongArch KVM guests.
In multi-vCPU KVM guests, remote TLB flushes broadcast IPIs to all target vCPUs, including those preempted by the host. Sending IPIs to preempted vCPUs causes unnecessary VM exits and grows with vCPU count, becoming severe in overcommitted deployments.
Quoted 7 lines
Reuse the existing steal-time shared memory page by adding a new
KVM_VCPU_FLUSH_TLBflag to the preempted byte. On the guest side, skip IPIs to preempted vCPUs and set the flag via cmpxchg instead. On the host side, when re-entering a vCPU inkvm_update_stolen_time(), check and clear the flag; if set, drop the VPID to trigger a full TLB flush on the next VM entry. No new shared memory page, hypercall, orKconfigoption is needed.Quoted 4 lines
After a second thought, the PV TLB flush can only work on hardware PTW, there is problem on software PTW.
With TLB flush flow, the current CPU sends IPI interrupter to other CPUs to flush TLB. With software PTW, other CPUs may execute TLB refill or TLB exception with interrupt disabled. IPI interrupt will not trigger until TLB refill/exception finishes, however with PV TLB method, other vCPUs will trap to host and flush TLBs, and continue to execute TLB refill/exception, stale TLB may be added.
Sorry for that.
This is indeed something I had overlooked -- thank you for catching it. It is a correctness issue, not just a performance one, so it has to gate the feature.
I understand the race exactly as you describe it. With software PTW, a TLB miss runs the guest's refill handler with interrupts disabled, and that handler may already have loaded the old PTE into a register. A native remote flush sends an IPI, but the IPI is held until IE is re-enabled, so the flush always lands *after* the refill completes -- any stale entry the refill wrote gets purged. The PV path breaks this: the host flushes the target's TLB (drops the VPID) at an arbitrary instant, then the target resumes its disabled refill handler and writes the stale entry back with tlbfill/tlbwr, and nothing flushes it afterwards. The same applies to the steal-time variant if a vCPU is preempted mid-refill:
it is flushed on re-entry and then resumes the refill. Only a hardware PTW is safe, because the refill is atomic and always reads the current table.So the feature must only be offered when the guest actually uses a hardware PTW. The real condition is on the *guest* -- a -cpu la464 guest still does software refill even on a PTW host -- so the host gate alone is not enough; a non-PTW guest must not observe the feature either.
For the host side,
cpu_has_ptwis the natural precondition, and there's already a precedent inkvm_vm_init_features()whereKVM_LOONGARCH_VM_FEAT_PTWis gated oncpu_has_ptw. I'd wrap the advertisement the same way:if (
cpu_has_ptw) {
kvm->arch.pv_features |=BIT(KVM_FEATURE_PV_TLB_FLUSH);kvm->arch.kvm_features|=BIT(KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH); }For the guest side: the guest discovers the feature by reading the
CPUCFG_KVM_FEATUREleaf, so if feasible I'd maskKVM_FEATURE_PV_TLB_FLUSHoff right there -- inkvm_emu_cpucfg()(theCPUCFG_KVM_FEATUREcase inexit.c) and the matchingone_reggetterkvm_loongarch_cpucfg_get_attr()invcpu.c-- whenever the guest's own CPUCFG2.PTW (vcpu->arch.cpucfg[2], bit 24) is clear. The equivalent could also be done in QEMU alongside the companion patch, since QEMU already controls the guest model's PTW bit. Both look implementable; I'll need to actually test them to see how they behave.On the hardware side, I checked the box I ran the benchmarks on. It reports as a 3A6000 / LA664, but reading CPUCFG2 directly gives
0x7e7cccc7with the PTW bit (24) clear, so it has no hardware PTW either -- likely an early part. Your 3C5000 is LA464, which is software PTW too. That means the ebizzy andtlb_benchnumbers I shared earlier were collected with PV TLB flush active on a -cpu la464 (software-refill) guest, i.e. exactly the unsafe configuration you pointed out. They show the mechanism works and its direction (hypercall ahead of steal-time, and PV preventing the overcommit collapse), but they were measured in a correctness-broken config, so I won't use them to justify enabling the feature on software-PTW silicon.
Quoted 8 lines
Quoted 6 lines
For now I don't have a PTW-capable machine to test on. My reading is that a software-PTW-safe variant isn't really possible -- the native IPI is safe precisely because the flush is deferred past the IE=0 refill window, and a host-side flush can't reproduce that ordering without effectively becoming an IPI again. I'll spend some time confirming that, but unless I've missed something, gating to hardware-PTW guests is the
PV TLB flush is a good technology, it can be verified again when there is hardware with HW PTW supported.
Thanks a lot for all the help on this. You raised a real correctness issue I'd missed, and I genuinely enjoyed working through it with you -- learned a good bit too. Yeah, makes sense to revisit PV TLB flush once there's HW-PTW hardware to test on.
Looking forward to next time.
Cheers,
Tao
Quoted 107 lines
Regards
Bibo MaoQuoted 104 lines
answer.
Thanks,
TaoQuoted 99 lines
Regards
Bibo MaoQuoted 96 lines
The feature is advertised through the existing KVM PV feature
negotiation: the kernel exposesKVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSHas a VM-level capability, and userspace (QEMU) setsKVM_FEATURE_PV_TLB_FLUSHin the guest'sCPUCFG_KVM_FEATUREmask after probing it. A corresponding QEMU patch ("target/loongarch: Enable PV TLB flush advertisement to the guest") is posted alongside this series.- Host side: only trace a PV TLB flush request when one is observed.
- (Carried over) Host uses amand_db.w to atomically read and clear the preempted byte; selftest gained input validation and failure cleanup.Testing: the PV TLB flush path itself is unchanged from v3, so the benchmark numbers below still hold. Note that, unlike v3, the feature must now be enabled by userspace: run a QEMU built with the companion patch (or with -cpu kvm-pv-tlb-flush=on) so the guest actually observes KVM_FEATURE_PV_TLB_FLUSH. Boot a 32-vCPU guest and run the selftest inside it with sleep-idle (PV helps) and busy-spin (PV cannot optimize) modes respectively:
qemu-system-loongarch64 \
-m 4G -smp 32 --cpu la464 --machine virt \
-bios .../QEMU_EFI.fd \
-kernel .../vmlinuz-...-pvtlb-v4+ \
-initrd/tmp/ramdisk_test.gz\
-serial mon:stdio \
-netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
-device virtio-net-pci,netdev=net0 \
-append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" -nographic# PV TLB flush enabled (idle threads sleep, vCPUs get preempted) guest# ./pv_tlb_flush_test 1 31 50000 0
# Baseline (idle threads busy-spin, all vCPUs stay active) guest# ./pv_tlb_flush_test 1 31 50000 1
With PV TLB flush (sleep idle): ~152,285 ns/flush Without PV TLB flush (busy-spin): ~481,045 ns/flush
Improvement: ~68% latency reduction (~3.2x throughput increase)
Tao Cui (3):
LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
LoongArch: KVM: Implement guest-side PV TLB flush
KVM: selftests: loongarch: Add PV TLB flush performance testarch/loongarch/include/asm/kvm_host.h | 1 + arch/loongarch/include/asm/kvm_para.h | 9 + arch/loongarch/include/asm/paravirt.h | 21 +++ arch/loongarch/include/uapi/asm/kvm.h | 1 +arch/loongarch/include/uapi/asm/kvm_para.h | 1 + arch/loongarch/kernel/paravirt.c | 60 ++++++ arch/loongarch/kernel/smp.c | 30 +++- arch/loongarch/kvm/trace.h | 15 ++ arch/loongarch/kvm/vcpu.c | 34 +++- arch/loongarch/kvm/vm.c | 3 + .../selftests/kvm/loongarch/pv_tlb_flush_test.c | 194 +++++++++++++++++++++
11 files changed, 362 insertions(+), 7 deletions(-)---
Changes in v4:
- Drop the "Preserve auto-enabled PV features on userspace override"
patch: forcing auto-enabled features back on is migration-unsafe, and for PV TLB flush (which cannot degrade gracefully) a missed flush would corrupt the guest. Enablement now follows the usual KVM model -- the kernel advertisesKVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSHand userspace (QEMU) explicitly setsKVM_FEATURE_PV_TLB_FLUSHafter probing it; a companion QEMU patch is posted alongside, and the feature can be kept off for an older destination
(-cpu kvm-pv-tlb-flush=off).Changes in v3:
- Host side: replace amswap_db.w with amand_db.w to atomically read and clear only the preempted byte, preserving the pad bytes for future UAPI use. Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data.
- Host side: move thepv_auto_featuresOR operation before the compatibility check inkvm_loongarch_cpucfg_set_attr()so that userspace does not need updating for pure kernel-internal PV feature additions.
- Selftest: add input validation, error checking onpthread_create, and cleanup handling on failure.Changes in v2:
- Host side: replace non-atomicunsafe_get_user+unsafe_put_userwith atomic amswap_db.w inline assembly. This fixes two issues:
1)unsafe_put_userfailure could skip the TLB flush entirely
2) non-atomic read+write race with guest-sidetry_cmpxchgcould causeFLUSH_TLBrequests to be lost
- Guest side: consolidate two separateREAD_ONCEcalls into a singleREAD_ONCEto eliminate a TOCTOU race where the host could clear preempted between the two reads. Also switch from byte-sizedtry_cmpxchgto 32-bittry_cmpxchgon the aligned word containing preempted.