Skip to contentSkip to conversation

LoongArch: KVM: Add PV TLB flush support

PATCH In review Tao Cui 3 participants 28 messages Jun 15 – Jul 17

Subsystems (5)
lore
Share & tools
lore

Tao Cui

author
#0 lore raw

v4 cover LoongArch: KVM: Add PV TLB flush support

From: Tao Cui <[email protected]>

This series implements paravirtualized TLB flush for LoongArch KVM guests.

In multi-vCPU KVM guests, remote TLB flushes broadcast IPIs to all target vCPUs, including those preempted by the host. Sending IPIs to preempted vCPUs causes unnecessary VM exits and grows with vCPU count, becoming severe in overcommitted deployments.

Reuse the existing steal-time shared memory page by adding a new KVM_VCPU_FLUSH_TLB flag to the preempted byte. On the guest side, skip IPIs to preempted vCPUs and set the flag via cmpxchg instead. On the host side, when re-entering a vCPU in kvm_update_stolen_time(), check and clear the flag; if set, drop the VPID to trigger a full TLB flush on the next VM entry. No new shared memory page, hypercall, or Kconfig option is needed.

The feature is advertised through the existing KVM PV feature
negotiation: the kernel exposes KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH as a VM-level capability, and userspace (QEMU) sets KVM_FEATURE_PV_TLB_FLUSH in the guest's CPUCFG_KVM_FEATURE mask after probing it. A corresponding QEMU patch ("target/loongarch: Enable PV TLB flush advertisement to the guest") is posted alongside this series.

- Host side: only trace a PV TLB flush request when one is observed.
- (Carried over) Host uses amand_db.w to atomically read and clear the preempted byte; selftest gained input validation and failure cleanup.

Testing: the PV TLB flush path itself is unchanged from v3, so the benchmark numbers below still hold. Note that, unlike v3, the feature must now be enabled by userspace: run a QEMU built with the companion patch (or with -cpu kvm-pv-tlb-flush=on) so the guest actually observes KVM_FEATURE_PV_TLB_FLUSH. Boot a 32-vCPU guest and run the selftest inside it with sleep-idle (PV helps) and busy-spin (PV cannot optimize) modes respectively:

  qemu-system-loongarch64 \
    -m 4G -smp 32 --cpu la464 --machine virt \
    -bios .../QEMU_EFI.fd \
    -kernel .../vmlinuz-...-pvtlb-v4+ \
    -initrd /tmp/ramdisk_test.gz \
    -serial mon:stdio \
    -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
    -device virtio-net-pci,netdev=net0 \
    -append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" -nographic

# PV TLB flush enabled (idle threads sleep, vCPUs get preempted) guest# ./pv_tlb_flush_test 1 31 50000 0

# Baseline (idle threads busy-spin, all vCPUs stay active) guest# ./pv_tlb_flush_test 1 31 50000 1

With PV TLB flush (sleep idle): ~152,285 ns/flush
Without PV TLB flush (busy-spin): ~481,045 ns/flush

Improvement: ~68% latency reduction (~3.2x throughput increase)

Tao Cui (3):
LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
LoongArch: KVM: Implement guest-side PV TLB flush
KVM: selftests: loongarch: Add PV TLB flush performance test

 arch/loongarch/include/asm/kvm_host.h              |   1 +
 arch/loongarch/include/asm/kvm_para.h              |   9 +
 arch/loongarch/include/asm/paravirt.h              |  21 +++
 arch/loongarch/include/uapi/asm/kvm.h              |   1 +
 arch/loongarch/include/uapi/asm/kvm_para.h         |   1 +
 arch/loongarch/kernel/paravirt.c                   |  60 ++++++
 arch/loongarch/kernel/smp.c                        |  30 +++-
 arch/loongarch/kvm/trace.h                         |  15 ++
 arch/loongarch/kvm/vcpu.c                          |  34 +++-
 arch/loongarch/kvm/vm.c                            |   3 +
 .../selftests/kvm/loongarch/pv_tlb_flush_test.c    | 194 +++++++++++++++++++++
 11 files changed, 362 insertions(+), 7 deletions(-)

---
Changes in v4:
- Drop the "Preserve auto-enabled PV features on userspace override"
patch: forcing auto-enabled features back on is migration-unsafe, and for PV TLB flush (which cannot degrade gracefully) a missed flush would corrupt the guest. Enablement now follows the usual KVM model -- the kernel advertises KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH and userspace (QEMU) explicitly sets KVM_FEATURE_PV_TLB_FLUSH after probing it; a companion QEMU patch is posted alongside, and the feature can be kept off for an older destination
(-cpu kvm-pv-tlb-flush=off).

Changes in v3:
- Host side: replace amswap_db.w with amand_db.w to atomically read and clear only the preempted byte, preserving the pad bytes for future UAPI use. Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data.
- Host side: move the pv_auto_features OR operation before the compatibility check in kvm_loongarch_cpucfg_set_attr() so that userspace does not need updating for pure kernel-internal PV feature additions.
- Selftest: add input validation, error checking on pthread_create, and cleanup handling on failure.

Changes in v2:
- Host side: replace non-atomic unsafe_get_user + unsafe_put_user with atomic amswap_db.w inline assembly. This fixes two issues:
1) unsafe_put_user failure could skip the TLB flush entirely
2) non-atomic read+write race with guest-side try_cmpxchg could cause FLUSH_TLB requests to be lost
- Guest side: consolidate two separate READ_ONCE calls into a single READ_ONCE to eliminate a TOCTOU race where the host could clear preempted between the two reads. Also switch from byte-sized try_cmpxchg to 32-bit try_cmpxchg on the aligned word containing preempted.

Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v4 1/3 LoongArch: KVM: Add PV TLB flush support via steal-time shared memory

From: Tao Cui <[email protected]>

Implement paravirtualized TLB flush for LoongArch KVM guests using the existing steal-time shared memory page.

The mechanism uses the preempted byte in struct kvm_steal_time with an additional KVM_VCPU_FLUSH_TLB flag bit:

- When a guest vCPU needs remote TLB flush but the target vCPU is preempted (not running), it atomically sets KVM_VCPU_FLUSH_TLB in the target's steal-time preempted byte instead of sending an IPI.
- When the host re-enters the target vCPU (kvm_update_stolen_time()), it atomically reads and clears only the preempted byte via amand_db.w with mask ~0xff on the 32-bit aligned word. amand_db.w provides full barrier semantics and avoids the race with guest-side try_cmpxchg(). LoongArch is little-endian, so the preempted byte occupies bits [7:0]; this preserves pad[0..2] for future UAPI extension. If KVM_VCPU_FLUSH_TLB was set, the host drops the vCPU's VPID, which triggers a full TLB flush on the next VM entry via kvm_check_vpid().
- For non-preempted vCPUs, the guest falls back to normal IPI-based flush, avoiding unnecessary VM exits.

Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data when the cache line was last modified by a different core.

This significantly reduces TLB flush overhead in multi-vCPU workloads where target vCPUs are often idle/preempted.

---
 arch/loongarch/include/asm/kvm_host.h      |  1 +
 arch/loongarch/include/asm/kvm_para.h      |  1 +
 arch/loongarch/include/uapi/asm/kvm.h      |  1 +
 arch/loongarch/include/uapi/asm/kvm_para.h |  1 +
 arch/loongarch/kvm/trace.h                 | 15 ++++++++++
 arch/loongarch/kvm/vcpu.c                  | 34 +++++++++++++++++++++-
 arch/loongarch/kvm/vm.c                    |  3 ++
 7 files changed, 55 insertions(+), 1 deletion(-)
arch/loongarch/include/asm/kvm_host.h +1 -0
index 776bc487a705..1750632699e0 100644
@@ -168,6 +168,7 @@ enum emulation_result {
 #define LOONGARCH_PV_FEAT_MASK		(BIT(KVM_FEATURE_IPI) |		\
 					 BIT(KVM_FEATURE_PREEMPT) |	\
 					 BIT(KVM_FEATURE_STEAL_TIME) |	\
+					 BIT(KVM_FEATURE_PV_TLB_FLUSH) |\
 					 BIT(KVM_FEATURE_USER_HCALL) |	\
 					 BIT(KVM_FEATURE_VIRT_EXTIOI))
 
arch/loongarch/include/asm/kvm_para.h +1 -0
index fb17ba0fa101..28e3fa3b4c0e 100644
@@ -41,6 +41,7 @@ struct kvm_steal_time {
 	__u8  pad[47];
 };
 #define KVM_VCPU_PREEMPTED		(1 << 0)
+#define KVM_VCPU_FLUSH_TLB		(1 << 1)
 
 /*
  * Hypercall interface for KVM hypervisor
arch/loongarch/include/uapi/asm/kvm.h +1 -0
index cd0b5c11ca9c..e4cd4bbf8914 100644
@@ -106,6 +106,7 @@ struct kvm_fpu {
 #define  KVM_LOONGARCH_VM_FEAT_PTW		8
 #define  KVM_LOONGARCH_VM_FEAT_MSGINT		9
 #define  KVM_LOONGARCH_VM_FEAT_PV_PREEMPT	10
+#define  KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH	11
 
 /* Device Control API on vcpu fd */
 #define KVM_LOONGARCH_VCPU_CPUCFG	0
arch/loongarch/include/uapi/asm/kvm_para.h +1 -0
index d28cbcadd276..8872839251cc 100644
@@ -16,6 +16,7 @@
 #define  KVM_FEATURE_IPI		1
 #define  KVM_FEATURE_STEAL_TIME		2
 #define  KVM_FEATURE_PREEMPT		3
+#define  KVM_FEATURE_PV_TLB_FLUSH	4
 /* BIT 24 - 31 are features configurable by user space vmm */
 #define  KVM_FEATURE_VIRT_EXTIOI	24
 #define  KVM_FEATURE_USER_HCALL		25
arch/loongarch/kvm/trace.h +15 -0
index 3467ee22b704..8556954fa196 100644
@@ -210,6 +210,21 @@ TRACE_EVENT(kvm_vpid_change,
 	    TP_printk("VPID: 0x%08lx", __entry->vpid)
 );
 
+TRACE_EVENT(kvm_pv_tlb_flush,
+	TP_PROTO(struct kvm_vcpu *vcpu, bool need_flush),
+	TP_ARGS(vcpu, need_flush),
+	TP_STRUCT__entry(
+		__field(unsigned int, vcpu_id)
+		__field(bool, need_flush)
+	),
+	TP_fast_assign(
+		__entry->vcpu_id = vcpu->vcpu_id;
+		__entry->need_flush = need_flush;
+	),
+	TP_printk("vcpu %u need_flush %u", __entry->vcpu_id,
+		  __entry->need_flush)
+);
+
 #endif /* _TRACE_KVM_H */
 
 #undef TRACE_INCLUDE_PATH
arch/loongarch/kvm/vcpu.c +33 -1
index e28084c49e68..5230e95a7816 100644
@@ -173,7 +173,39 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 	}
 
 	st = (struct kvm_steal_time __user *)ghc->hva;
-	if (kvm_guest_has_pv_feature(vcpu, KVM_FEATURE_PREEMPT)) {
+	if (kvm_guest_has_pv_feature(vcpu, KVM_FEATURE_PV_TLB_FLUSH)) {
+		u32 old = 0;
+		int err = 0;
+
+		/*
+		 * Prime the cache line with a normal load before the coherent
+		 * atomic below; it was observed (when the line was last dirtied
+		 * by another core) to be needed for amand_db.w to see a current
+		 * value.  amand_db.w overwrites `old` with the real pre-AND value,
+		 * so this load contributes only its cache side-effect.
+		 */
+		unsafe_get_user(old, (u32 __user *)&st->preempted, out);
+
+		/* Atomically read and clear the preempted byte via amand_db.w. */
+		asm volatile(
+		"1: amand_db.w %1, %3, %2	\n"
+		"2:				\n"
+		_ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1)
+		: "+r" (err), "+&r" (old),
+		  "+ZB" (*(u32 *)&st->preempted)
+		: "r" ((u32)~0xffu)
+		: "memory");
+
+		if (err)
+			goto out;
+
+		vcpu->arch.st.preempted = 0;
+
+		if ((u8)old & KVM_VCPU_FLUSH_TLB) {
+			vcpu->arch.vpid = 0;	/* Drop vpid to flush TLB */
+			trace_kvm_pv_tlb_flush(vcpu, true);
+		}
+	} else if (kvm_guest_has_pv_feature(vcpu, KVM_FEATURE_PREEMPT)) {
 		unsafe_put_user(0, &st->preempted, out);
 		vcpu->arch.st.preempted = 0;
 	}
arch/loongarch/kvm/vm.c +3 -0
index 1317c718f896..cfba45a7343c 100644
@@ -54,8 +54,10 @@ static void kvm_vm_init_features(struct kvm *kvm)
 	if (kvm_pvtime_supported()) {
 		kvm->arch.pv_features |= BIT(KVM_FEATURE_PREEMPT);
 		kvm->arch.pv_features |= BIT(KVM_FEATURE_STEAL_TIME);
+		kvm->arch.pv_features |= BIT(KVM_FEATURE_PV_TLB_FLUSH);
 		kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_PREEMPT);
 		kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_STEALTIME);
+		kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH);
 	}
 }
 
@@ -158,6 +160,7 @@ static int kvm_vm_feature_has_attr(struct kvm *kvm, struct kvm_device_attr *attr
 	case KVM_LOONGARCH_VM_FEAT_PV_IPI:
 	case KVM_LOONGARCH_VM_FEAT_PV_PREEMPT:
 	case KVM_LOONGARCH_VM_FEAT_PV_STEALTIME:
+	case KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH:
 		if (kvm_vm_support(&kvm->arch, attr->attr))
 			return 0;
 		return -ENXIO;
Signature · 1 line
2.43.0
author
#5 lore raw

PATCH v4 2/3 LoongArch: KVM: Implement guest-side PV TLB flush

In reply to Tao Cui

From: Tao Cui <[email protected]>

Add the guest-side implementation of PV TLB flush for LoongArch KVM guests, complementing the host-side support added in the previous commit.

When running as a KVM guest, remote TLB flushes are optimized by avoiding IPIs to preempted vCPUs:

- kvm_flush_tlb_mask() checks each target vCPU's steal-time preempted flag. If a vCPU is preempted, it atomically sets KVM_VCPU_FLUSH_TLB in the shared preempted byte via cmpxchg and removes that CPU from the IPI mask.
- Only non-preempted vCPUs receive IPIs via on_each_cpu_mask().
- When the host schedules a deferred-flush vCPU back in, it invalidates the VPID and flushes the TLB automatically.

All six SMP TLB flush functions (flush_tlb_all, flush_tlb_mm, flush_tlb_range, flush_tlb_kernel_range, flush_tlb_page, flush_tlb_one) are updated to use the PV path when the static key pv_tlb_flush_key is enabled.

The feature is gated by KVM_FEATURE_PV_TLB_FLUSH and requires KVM_FEATURE_STEAL_TIME (for the shared memory page). Per-CPU cpumask buffers are allocated via arch_initcall.

---
 arch/loongarch/include/asm/kvm_para.h |  8 ++++
 arch/loongarch/include/asm/paravirt.h | 21 ++++++++++
 arch/loongarch/kernel/paravirt.c      | 60 +++++++++++++++++++++++++++
 arch/loongarch/kernel/smp.c           | 30 +++++++++++---
 4 files changed, 113 insertions(+), 6 deletions(-)
arch/loongarch/include/asm/kvm_para.h +8 -0
index 28e3fa3b4c0e..7956aeff2436 100644
@@ -187,4 +187,12 @@ static inline bool kvm_check_and_clear_guest_paused(void)
 	return false;
 }
 
+static inline bool kvm_pv_tlb_flush_supported(void)
+{
+	unsigned int feat = kvm_arch_para_features();
+
+	return (feat & (1U << KVM_FEATURE_PV_TLB_FLUSH)) &&
+	       (feat & (1U << KVM_FEATURE_STEAL_TIME));
+}
+
 #endif /* _ASM_LOONGARCH_KVM_PARA_H */
arch/loongarch/include/asm/paravirt.h +21 -0
index acae1c5e5f88..6ce62b555a5d 100644
@@ -5,15 +5,26 @@
 #ifdef CONFIG_PARAVIRT
 
 #include <linux/jump_label.h>
+#include <linux/cpumask.h>
+#include <linux/smp.h>
 
 DECLARE_STATIC_KEY_FALSE(virt_preempt_key);
 DECLARE_STATIC_KEY_FALSE(virt_spin_lock_key);
+DECLARE_STATIC_KEY_FALSE(pv_tlb_flush_key);
 DECLARE_PER_CPU(struct kvm_steal_time, steal_time);
 
 int __init pv_ipi_init(void);
 int __init pv_time_init(void);
 int __init pv_spinlock_init(void);
 
+void kvm_flush_tlb_mask(const struct cpumask *cpumask,
+			smp_call_func_t func, void *info);
+
+static inline bool pv_tlb_flush_enabled(void)
+{
+	return static_branch_unlikely(&pv_tlb_flush_key);
+}
+
 #else
 
 static inline int pv_ipi_init(void)
@@ -31,5 +42,15 @@ static inline int pv_spinlock_init(void)
 	return 0;
 }
 
+static inline bool pv_tlb_flush_enabled(void)
+{
+	return false;
+}
+
+static inline void kvm_flush_tlb_mask(const struct cpumask *cpumask,
+				       smp_call_func_t func, void *info)
+{
+}
+
 #endif // CONFIG_PARAVIRT
 #endif
arch/loongarch/kernel/paravirt.c +60 -0

73 lines not on this page · raw message

arch/loongarch/kernel/smp.c +24 -6
index 50922610758b..bb3451b057ed 100644
@@ -727,7 +727,10 @@ static void flush_tlb_all_ipi(void *info)
 
 void flush_tlb_all(void)
 {
-	on_each_cpu(flush_tlb_all_ipi, NULL, 1);
+	if (pv_tlb_flush_enabled())
+		kvm_flush_tlb_mask(cpu_online_mask, flush_tlb_all_ipi, NULL);
+	else
+		on_each_cpu(flush_tlb_all_ipi, NULL, 1);
 }
 
 static void flush_tlb_mm_ipi(void *mm)
@@ -743,7 +746,10 @@ void flush_tlb_mm(struct mm_struct *mm)
 	preempt_disable();
 
 	if ((atomic_read(&mm->mm_users) != 1) || (current->mm != mm)) {
-		on_each_cpu_mask(mm_cpumask(mm), flush_tlb_mm_ipi, mm, 1);
+		if (pv_tlb_flush_enabled())
+			kvm_flush_tlb_mask(mm_cpumask(mm), flush_tlb_mm_ipi, mm);
+		else
+			on_each_cpu_mask(mm_cpumask(mm), flush_tlb_mm_ipi, mm, 1);
 	} else {
 		unsigned int cpu;
 
@@ -782,7 +788,10 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned l
 			.addr2 = end,
 		};
 
-		on_each_cpu_mask(mm_cpumask(mm), flush_tlb_range_ipi, &fd, 1);
+		if (pv_tlb_flush_enabled())
+			kvm_flush_tlb_mask(mm_cpumask(mm), flush_tlb_range_ipi, &fd);
+		else
+			on_each_cpu_mask(mm_cpumask(mm), flush_tlb_range_ipi, &fd, 1);
 	} else {
 		unsigned int cpu;
 
@@ -809,7 +818,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 		.addr2 = end,
 	};
 
-	on_each_cpu(flush_tlb_kernel_range_ipi, &fd, 1);
+	if (pv_tlb_flush_enabled())
+		kvm_flush_tlb_mask(cpu_online_mask, flush_tlb_kernel_range_ipi, &fd);
+	else
+		on_each_cpu(flush_tlb_kernel_range_ipi, &fd, 1);
 }
 
 static void flush_tlb_page_ipi(void *info)
@@ -828,7 +840,10 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long page)
 			.addr1 = page,
 		};
 
-		on_each_cpu_mask(mm_cpumask(vma->vm_mm), flush_tlb_page_ipi, &fd, 1);
+		if (pv_tlb_flush_enabled())
+			kvm_flush_tlb_mask(mm_cpumask(vma->vm_mm), flush_tlb_page_ipi, &fd);
+		else
+			on_each_cpu_mask(mm_cpumask(vma->vm_mm), flush_tlb_page_ipi, &fd, 1);
 	} else {
 		unsigned int cpu;
 
@@ -851,6 +866,9 @@ static void flush_tlb_one_ipi(void *info)
 
 void flush_tlb_one(unsigned long vaddr)
 {
-	on_each_cpu(flush_tlb_one_ipi, (void *)vaddr, 1);
+	if (pv_tlb_flush_enabled())
+		kvm_flush_tlb_mask(cpu_online_mask, flush_tlb_one_ipi, (void *)vaddr);
+	else
+		on_each_cpu(flush_tlb_one_ipi, (void *)vaddr, 1);
 }
 EXPORT_SYMBOL(flush_tlb_one);
Signature · 1 line
2.43.0
author
#19 lore raw

PATCH v4 3/3 KVM: selftests: loongarch: Add PV TLB flush performance test

In reply to Tao Cui

From: Tao Cui <[email protected]>

Add a multi-threaded benchmark to measure PV TLB flush performance inside LoongArch KVM guests.

The test spawns flusher threads that repeatedly mmap/munmap to trigger TLB shootdown IPIs, alongside idle threads that either sleep or busy-spin. With PV TLB flush enabled, IPIs to preempted vCPUs are replaced by deferred flags in the steal-time shared page.

Usage (inside guest):
./pv_tlb_flush_test <flushers> <idle> <iterations> <busy_idle> busy_idle=0: idle threads sleep (PV can skip IPIs to preempted vCPUs) busy_idle=1: idle threads spin (all vCPUs active, PV cannot optimize)

---
 .../kvm/loongarch/pv_tlb_flush_test.c         | 194 ++++++++++++++++++
 1 file changed, 194 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +194 -0
new file mode 100644
index 000000000000..63efaf9ef1cd
@@ -0,0 +1,194 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * LoongArch PV TLB Flush Performance Test
+ *
+ * Measure the overhead of remote TLB flushes in a KVM guest by spawning
+ * flusher threads that repeatedly mmap/munmap (triggering TLB shootdown
+ * IPIs) alongside idle threads that either sleep or busy-spin.
+ *
+ * With PV TLB flush enabled, IPIs to preempted vCPUs are replaced by
+ * deferred flags in the steal-time shared page, reducing flush latency.
+ *
+ * Usage:
+ *   Compile on LoongArch guest:
+ *     gcc -O2 -static -pthread -o pv_tlb_flush_test pv_tlb_flush_test.c
+ *   Run (inside KVM guest):
+ *     ./pv_tlb_flush_test <flushers> <idle> <iterations> <busy_idle>
+ *   Examples:
+ *     ./pv_tlb_flush_test 1 31 50000 0   # 1 flusher, 31 sleep, PV helps
+ *     ./pv_tlb_flush_test 1 31 50000 1   # 1 flusher, 31 busy-spin, no PV
+ *
+ *   busy_idle=0: idle threads sleep, vCPUs get preempted, PV TLB flush
+ *                can skip IPIs to them
+ *   busy_idle=1: idle threads spin, all vCPUs stay active, PV TLB flush
+ *                cannot optimize (baseline for comparison)
+ */
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/mman.h>
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <sched.h>
+#include <pthread.h>
+
+#define MEM_SIZE (2*1024*1024)
+#define DEFAULT_ITERS 50000
+#define MAX_THREADS 64
+
+static int nr_iters = DEFAULT_ITERS;
+static volatile int start_barrier;
+static volatile int stop_flag;
+static int busy_idle = 0;
+
+struct thread_args {
+	int cpu;
+	unsigned long *result;
+	int *completed;
+};
+
+static inline unsigned long clock_ns(void) {
+    struct timespec ts;
+    clock_gettime(CLOCK_MONOTONIC, &ts);
+    return (unsigned long)ts.tv_sec * 1000000000UL + ts.tv_nsec;
+}
+
+static void pin_cpu(int cpu) {
+    cpu_set_t set;
+    if (cpu < 0)
+        return;
+    CPU_ZERO(&set);
+    CPU_SET(cpu, &set);
+    sched_setaffinity(0, sizeof(set), &set);
+}
+
+static void *idle_thread(void *arg) {
+    struct thread_args *ta = arg;
+    pin_cpu(ta->cpu);
+    while (!__atomic_load_n(&start_barrier, __ATOMIC_ACQUIRE));
+    if (busy_idle) {
+        volatile long sink = 0;
+        while (!__atomic_load_n(&stop_flag, __ATOMIC_ACQUIRE))
+            sink++;
+    } else {
+        while (!__atomic_load_n(&stop_flag, __ATOMIC_ACQUIRE))
+            usleep(1000);
+    }
+    return NULL;
+}
+
+static void *flush_thread(void *arg) {
+    struct thread_args *ta = arg;
+    unsigned long start, end;
+    int i;
+    size_t mem_size = MEM_SIZE;
+    pin_cpu(ta->cpu);
+    while (!__atomic_load_n(&start_barrier, __ATOMIC_ACQUIRE));
+    start = clock_ns();
+    for (i = 0; i < nr_iters && !__atomic_load_n(&stop_flag, __ATOMIC_ACQUIRE); i++) {
+        void *p = mmap(NULL, mem_size, PROT_READ|PROT_WRITE,
+                       MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
+        if (p == MAP_FAILED) break;
+        for (size_t off = 0; off < mem_size; off += 65536)
+            ((volatile char*)p)[off] = 0;
+        munmap(p, mem_size);
+    }
+    end = clock_ns();
+    *ta->result = end - start;
+    *ta->completed = i;
+    return NULL;
+}
+
+int main(int argc, char **argv) {
+    int nr_flush = 1, nr_idle = 3, i, run;
+    int ncpus = sysconf(_SC_NPROCESSORS_ONLN);
+    if (argc > 1) nr_flush = atoi(argv[1]);
+    if (argc > 2) nr_idle = atoi(argv[2]);
+    if (argc > 3) nr_iters = atoi(argv[3]);
+    if (argc > 4) busy_idle = atoi(argv[4]);
+
+    if (nr_flush < 1 || nr_idle < 0 || nr_flush + nr_idle > MAX_THREADS) {
+        fprintf(stderr, "Usage: %s [flushers(1-%d)] [idle(0-%d)] [iters] [busy_idle]\n",
+                argv[0], MAX_THREADS, MAX_THREADS);
+        return 1;
+    }
+    if (nr_iters <= 0) {
+        fprintf(stderr, "Error: iterations must be positive\n");
+        return 1;
+    }
+
+    printf("=== TLB Flush Benchmark ===\n");
+    printf("CPUs: %d  Flushers: %d  Idle: %d  Iters: %d  Mode: %s\n",
+           ncpus, nr_flush, nr_idle, nr_iters,
+           busy_idle ? "busy-spin" : "sleep");
+
+    for (run = 0; run < 3; run++) {
+        int total = nr_flush + nr_idle;
+        int do_pin = (total <= ncpus);
+        int created = 0;
+        pthread_t threads[MAX_THREADS];
+        unsigned long results[MAX_THREADS];
+        int completed[MAX_THREADS];
+        struct thread_args args[MAX_THREADS];
+        start_barrier = 0; stop_flag = 0;
+
+        for (i = 0; i < nr_idle; i++) {
+            args[i].cpu = do_pin ? nr_flush + i : -1;
+            args[i].result = NULL;
+            args[i].completed = NULL;
+            if (pthread_create(&threads[i], NULL, idle_thread, &args[i])) {
+                perror("pthread_create idle");
+                goto cleanup;
+            }
+            created++;
+        }
+        for (i = 0; i < nr_flush; i++) {
+            int idx = nr_idle + i;
+            results[idx] = 0;
+            completed[idx] = 0;
+            args[idx].cpu = do_pin ? i : -1;
+            args[idx].result = &results[idx];
+            args[idx].completed = &completed[idx];
+            if (pthread_create(&threads[idx], NULL, flush_thread, &args[idx])) {
+                perror("pthread_create flush");
+                goto cleanup;
+            }
+            created++;
+        }
+
+        usleep(10000);
+        __atomic_store_n(&start_barrier, 1, __ATOMIC_RELEASE);
+        for (i = 0; i < nr_flush; i++)
+            pthread_join(threads[nr_idle + i], NULL);
+        __atomic_store_n(&stop_flag, 1, __ATOMIC_RELEASE);
+        for (i = 0; i < nr_idle; i++)
+            pthread_join(threads[i], NULL);
+
+        unsigned long total_ns = 0;
+        unsigned long total_done = 0;
+        for (i = 0; i < nr_flush; i++) {
+            int idx = nr_idle + i;
+            unsigned long done = completed[idx];
+            if (done == 0) {
+                printf("  Run %d flusher %d: no iterations completed\n", run, i);
+                continue;
+            }
+            printf("  Run %d flusher %d: %lu ns/flush (%lu iters)\n",
+                   run, i, results[idx] / done, done);
+            total_ns += results[idx];
+            total_done += done;
+        }
+        if (total_done > 0)
+            printf("  Run %d Avg: %lu ns/flush\n", run, total_ns / total_done);
+        continue;
+
+cleanup:
+        __atomic_store_n(&start_barrier, 1, __ATOMIC_RELEASE);
+        __atomic_store_n(&stop_flag, 1, __ATOMIC_RELEASE);
+        for (i = 0; i < created; i++)
+            pthread_join(threads[i], NULL);
+        return 1;
+    }
+    return 0;
+}
Signature · 1 line
2.43.0
#20 lore raw

Re: [PATCH v4 3/3] KVM: selftests: loongarch: Add PV TLB flush performance test

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] Missing Makefile integration for the new test file `pv_tlb_flush_test.c`. - [Medium] Signed…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Missing Makefile integration for the new test file pv_tlb_flush_test.c.
- [Medium] Signed integer overflow in thread bounds check leads to stack buffer overflow. --

commit 56616f9df639ca6715014c26c5864780ced6ff41
Author: Tao Cui <[email protected]>

KVM: selftests: loongarch: Add PV TLB flush performance test

Add a multi-threaded benchmark to measure PV TLB flush performance inside LoongArch KVM guests.

Quoted 12 lines
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +6 -0
new file mode 100644
index 0000000000000..63efaf9ef1cd0
@@ -0,0 +1,194 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * LoongArch PV TLB Flush Performance Test
+ *
+ * Measure the overhead of remote TLB flushes in a KVM guest by spawning
+ * flusher threads that repeatedly mmap/munmap (triggering TLB shootdown

tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c line 16

Quoted 10 lines
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +10 -0
+ * IPIs) alongside idle threads that either sleep or busy-spin.
+ *
+ * With PV TLB flush enabled, IPIs to preempted vCPUs are replaced by
+ * deferred flags in the steal-time shared page, reducing flush latency.
+ *
+ * Usage:
+ *   Compile on LoongArch guest:
+ *     gcc -O2 -static -pthread -o pv_tlb_flush_test pv_tlb_flush_test.c
+ *   Run (inside KVM guest):
+ *     ./pv_tlb_flush_test <flushers> <idle> <iterations> <busy_idle>

[Severity: Medium]
Since this test is provided as a standalone script and not added to the selftests Makefile (e.g., TEST_PROGS or TEST_GEN_PROGS), will it be missed during make install and automated test execution?

Also, is it intentional that this test bypasses the standard KVM selftests framework (kvm_util) which typically provisions VMs and runs guest payloads programmatically?

[ ... ]

tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c line 111

Quoted 9 lines
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +9 -0
+int main(int argc, char **argv) {
+    int nr_flush = 1, nr_idle = 3, i, run;
+    int ncpus = sysconf(_SC_NPROCESSORS_ONLN);
+    if (argc > 1) nr_flush = atoi(argv[1]);
+    if (argc > 2) nr_idle = atoi(argv[2]);
+    if (argc > 3) nr_iters = atoi(argv[3]);
+    if (argc > 4) busy_idle = atoi(argv[4]);
+
+    if (nr_flush < 1 || nr_idle < 0 || nr_flush + nr_idle > MAX_THREADS) {

[Severity: Medium]
Could this bounds check be bypassed if a user provides values that sum to a value exceeding the 32-bit signed integer limit (e.g., nr_flush=1 and nr_idle=2147483647)? The signed addition would overflow to a negative number.

If bypassed, would the program then loop nr_idle times and write out of bounds to the stack-allocated arrays like threads and args in the loop below?

[ ... ]

Quoted 11 lines
tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +11 -0
+        pthread_t threads[MAX_THREADS];
+        unsigned long results[MAX_THREADS];
+        int completed[MAX_THREADS];
+        struct thread_args args[MAX_THREADS];
+        start_barrier = 0; stop_flag = 0;
+
+        for (i = 0; i < nr_idle; i++) {
+            args[i].cpu = do_pin ? nr_flush + i : -1;
+            args[i].result = NULL;
+            args[i].completed = NULL;
+            if (pthread_create(&threads[i], NULL, idle_thread, &args[i])) {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#2 lore raw

Re: [PATCH v4 1/3] LoongArch: KVM: Add PV TLB flush support via steal-time shared memory

In reply to Tao Cui
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] KVM_FEATURE_PV_TLB_FLUSH is unconditionally enabled by default, breaking capability negotiation…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] KVM_FEATURE_PV_TLB_FLUSH is unconditionally enabled by default, breaking capability negotiation and live migration. --

commit f9022e883153b129074818c0b5b2ff53a144e2f5
Author: Tao Cui <[email protected]>

LoongArch: KVM: Add PV TLB flush support via steal-time shared memory

This commit implements paravirtualized TLB flushes for LoongArch guests using the preempted byte in the steal-time shared memory page. It introduces the KVM_VCPU_FLUSH_TLB flag to avoid sending IPIs when the target vCPU is currently preempted.

[ ... ]

arch/loongarch/kvm/vm.c static void kvm_vm_init_features(struct kvm *kvm) · line 60

Quoted 12 lines
arch/loongarch/kvm/vm.c +2 -0
index 1317c718f896a..cfba45a7343c5 100644
@@ -54,8 +54,10 @@ static void kvm_vm_init_features(struct kvm *kvm)
 	if (kvm_pvtime_supported()) {
 		kvm->arch.pv_features |= BIT(KVM_FEATURE_PREEMPT);
 		kvm->arch.pv_features |= BIT(KVM_FEATURE_STEAL_TIME);
+		kvm->arch.pv_features |= BIT(KVM_FEATURE_PV_TLB_FLUSH);
 		kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_PREEMPT);
 		kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_STEALTIME);
+		kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH);

[Severity: High]
Should this new guest-visible feature be disabled by default?

If a VM starts on a kernel with this feature enabled by default, the guest will detect and use PV TLB flushes. If that VM is later live-migrated to an older kernel that lacks PV TLB flush support, the destination host will silently ignore the KVM_VCPU_FLUSH_TLB flag in the steal-time structure.

Could this lead to the guest assuming its TLB was flushed when it wasn't, resulting in stale translations or memory corruption? Would it be safer to require an explicit userspace capability enablement before exposing this to the guest?

Quoted 2 lines
arch/loongarch/kvm/vm.c +0 -0
 	}
 }

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#21 lore raw

Re: [PATCH v4 3/3] KVM: selftests: loongarch: Add PV TLB flush performance test

In reply to Tao Cui

Hi Tao,

Thanks for you patch.

Now LoongArch KVM is just workable and not used with wide scale, I am a little worried the stability of this PV feature.

Do you have real work load to show the benefits of the PV TLB flush, and which TLB APIs should be hooked.

Regards
Bibo Mao

Quoted 222 lines · Tao Cui

From: Tao Cui <[email protected]>

Add a multi-threaded benchmark to measure PV TLB flush performance inside LoongArch KVM guests.

The test spawns flusher threads that repeatedly mmap/munmap to trigger TLB shootdown IPIs, alongside idle threads that either sleep or busy-spin. With PV TLB flush enabled, IPIs to preempted vCPUs are replaced by deferred flags in the steal-time shared page.

Usage (inside guest):
./pv_tlb_flush_test <flushers> <idle> <iterations> <busy_idle> busy_idle=0: idle threads sleep (PV can skip IPIs to preempted vCPUs) busy_idle=1: idle threads spin (all vCPUs active, PV cannot optimize)

---
  .../kvm/loongarch/pv_tlb_flush_test.c         | 194 ++++++++++++++++++
  1 file changed, 194 insertions(+)

create mode 100644 tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c

tools/testing/selftests/kvm/loongarch/pv_tlb_flush_test.c new file +194 -0
new file mode 100644
index 000000000000..63efaf9ef1cd
@@ -0,0 +1,194 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * LoongArch PV TLB Flush Performance Test
+ *
+ * Measure the overhead of remote TLB flushes in a KVM guest by spawning
+ * flusher threads that repeatedly mmap/munmap (triggering TLB shootdown
+ * IPIs) alongside idle threads that either sleep or busy-spin.
+ *
+ * With PV TLB flush enabled, IPIs to preempted vCPUs are replaced by
+ * deferred flags in the steal-time shared page, reducing flush latency.
+ *
+ * Usage:
+ *   Compile on LoongArch guest:
+ *     gcc -O2 -static -pthread -o pv_tlb_flush_test pv_tlb_flush_test.c
+ *   Run (inside KVM guest):
+ *     ./pv_tlb_flush_test <flushers> <idle> <iterations> <busy_idle>
+ *   Examples:
+ *     ./pv_tlb_flush_test 1 31 50000 0   # 1 flusher, 31 sleep, PV helps
+ *     ./pv_tlb_flush_test 1 31 50000 1   # 1 flusher, 31 busy-spin, no PV
+ *
+ *   busy_idle=0: idle threads sleep, vCPUs get preempted, PV TLB flush
+ *                can skip IPIs to them
+ *   busy_idle=1: idle threads spin, all vCPUs stay active, PV TLB flush
+ *                cannot optimize (baseline for comparison)
+ */
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/mman.h>
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <sched.h>
+#include <pthread.h>
+
+#define MEM_SIZE (2*1024*1024)
+#define DEFAULT_ITERS 50000
+#define MAX_THREADS 64
+
+static int nr_iters = DEFAULT_ITERS;
+static volatile int start_barrier;
+static volatile int stop_flag;
+static int busy_idle = 0;
+
+struct thread_args {
+	int cpu;
+	unsigned long *result;
+	int *completed;
+};
+
+static inline unsigned long clock_ns(void) {
+    struct timespec ts;
+    clock_gettime(CLOCK_MONOTONIC, &ts);
+    return (unsigned long)ts.tv_sec * 1000000000UL + ts.tv_nsec;
+}
+
+static void pin_cpu(int cpu) {
+    cpu_set_t set;
+    if (cpu < 0)
+        return;
+    CPU_ZERO(&set);
+    CPU_SET(cpu, &set);
+    sched_setaffinity(0, sizeof(set), &set);
+}
+
+static void *idle_thread(void *arg) {
+    struct thread_args *ta = arg;
+    pin_cpu(ta->cpu);
+    while (!__atomic_load_n(&start_barrier, __ATOMIC_ACQUIRE));
+    if (busy_idle) {
+        volatile long sink = 0;
+        while (!__atomic_load_n(&stop_flag, __ATOMIC_ACQUIRE))
+            sink++;
+    } else {
+        while (!__atomic_load_n(&stop_flag, __ATOMIC_ACQUIRE))
+            usleep(1000);
+    }
+    return NULL;
+}
+
+static void *flush_thread(void *arg) {
+    struct thread_args *ta = arg;
+    unsigned long start, end;
+    int i;
+    size_t mem_size = MEM_SIZE;
+    pin_cpu(ta->cpu);
+    while (!__atomic_load_n(&start_barrier, __ATOMIC_ACQUIRE));
+    start = clock_ns();
+    for (i = 0; i < nr_iters && !__atomic_load_n(&stop_flag, __ATOMIC_ACQUIRE); i++) {
+        void *p = mmap(NULL, mem_size, PROT_READ|PROT_WRITE,
+                       MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
+        if (p == MAP_FAILED) break;
+        for (size_t off = 0; off < mem_size; off += 65536)
+            ((volatile char*)p)[off] = 0;
+        munmap(p, mem_size);
+    }
+    end = clock_ns();
+    *ta->result = end - start;
+    *ta->completed = i;
+    return NULL;
+}
+
+int main(int argc, char **argv) {
+    int nr_flush = 1, nr_idle = 3, i, run;
+    int ncpus = sysconf(_SC_NPROCESSORS_ONLN);
+    if (argc > 1) nr_flush = atoi(argv[1]);
+    if (argc > 2) nr_idle = atoi(argv[2]);
+    if (argc > 3) nr_iters = atoi(argv[3]);
+    if (argc > 4) busy_idle = atoi(argv[4]);
+
+    if (nr_flush < 1 || nr_idle < 0 || nr_flush + nr_idle > MAX_THREADS) {
+        fprintf(stderr, "Usage: %s [flushers(1-%d)] [idle(0-%d)] [iters] [busy_idle]\n",
+                argv[0], MAX_THREADS, MAX_THREADS);
+        return 1;
+    }
+    if (nr_iters <= 0) {
+        fprintf(stderr, "Error: iterations must be positive\n");
+        return 1;
+    }
+
+    printf("=== TLB Flush Benchmark ===\n");
+    printf("CPUs: %d  Flushers: %d  Idle: %d  Iters: %d  Mode: %s\n",
+           ncpus, nr_flush, nr_idle, nr_iters,
+           busy_idle ? "busy-spin" : "sleep");
+
+    for (run = 0; run < 3; run++) {
+        int total = nr_flush + nr_idle;
+        int do_pin = (total <= ncpus);
+        int created = 0;
+        pthread_t threads[MAX_THREADS];
+        unsigned long results[MAX_THREADS];
+        int completed[MAX_THREADS];
+        struct thread_args args[MAX_THREADS];
+        start_barrier = 0; stop_flag = 0;
+
+        for (i = 0; i < nr_idle; i++) {
+            args[i].cpu = do_pin ? nr_flush + i : -1;
+            args[i].result = NULL;
+            args[i].completed = NULL;
+            if (pthread_create(&threads[i], NULL, idle_thread, &args[i])) {
+                perror("pthread_create idle");
+                goto cleanup;
+            }
+            created++;
+        }
+        for (i = 0; i < nr_flush; i++) {
+            int idx = nr_idle + i;
+            results[idx] = 0;
+            completed[idx] = 0;
+            args[idx].cpu = do_pin ? i : -1;
+            args[idx].result = &results[idx];
+            args[idx].completed = &completed[idx];
+            if (pthread_create(&threads[idx], NULL, flush_thread, &args[idx])) {
+                perror("pthread_create flush");
+                goto cleanup;
+            }
+            created++;
+        }
+
+        usleep(10000);
+        __atomic_store_n(&start_barrier, 1, __ATOMIC_RELEASE);
+        for (i = 0; i < nr_flush; i++)
+            pthread_join(threads[nr_idle + i], NULL);
+        __atomic_store_n(&stop_flag, 1, __ATOMIC_RELEASE);
+        for (i = 0; i < nr_idle; i++)
+            pthread_join(threads[i], NULL);
+
+        unsigned long total_ns = 0;
+        unsigned long total_done = 0;
+        for (i = 0; i < nr_flush; i++) {
+            int idx = nr_idle + i;
+            unsigned long done = completed[idx];
+            if (done == 0) {
+                printf("  Run %d flusher %d: no iterations completed\n", run, i);
+                continue;
+            }
+            printf("  Run %d flusher %d: %lu ns/flush (%lu iters)\n",
+                   run, i, results[idx] / done, done);
+            total_ns += results[idx];
+            total_done += done;
+        }
+        if (total_done > 0)
+            printf("  Run %d Avg: %lu ns/flush\n", run, total_ns / total_done);
+        continue;
+
+cleanup:
+        __atomic_store_n(&start_barrier, 1, __ATOMIC_RELEASE);
+        __atomic_store_n(&stop_flag, 1, __ATOMIC_RELEASE);
+        for (i = 0; i < created; i++)
+            pthread_join(threads[i], NULL);
+        return 1;
+    }
+    return 0;
+}
maintainer
#3 lore raw

Re: [PATCH v4 1/3] LoongArch: KVM: Add PV TLB flush support via steal-time shared memory

In reply to Tao Cui
Quoted 126 lines · Tao Cui

From: Tao Cui <[email protected]>

Implement paravirtualized TLB flush for LoongArch KVM guests using the existing steal-time shared memory page.

The mechanism uses the preempted byte in struct kvm_steal_time with an additional KVM_VCPU_FLUSH_TLB flag bit:

- When a guest vCPU needs remote TLB flush but the target vCPU is preempted (not running), it atomically sets KVM_VCPU_FLUSH_TLB in the target's steal-time preempted byte instead of sending an IPI.
- When the host re-enters the target vCPU (kvm_update_stolen_time()), it atomically reads and clears only the preempted byte via amand_db.w with mask ~0xff on the 32-bit aligned word. amand_db.w provides full barrier semantics and avoids the race with guest-side try_cmpxchg(). LoongArch is little-endian, so the preempted byte occupies bits [7:0]; this preserves pad[0..2] for future UAPI extension. If KVM_VCPU_FLUSH_TLB was set, the host drops the vCPU's VPID, which triggers a full TLB flush on the next VM entry via kvm_check_vpid().
- For non-preempted vCPUs, the guest falls back to normal IPI-based flush, avoiding unnecessary VM exits.

Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data when the cache line was last modified by a different core.

This significantly reduces TLB flush overhead in multi-vCPU workloads where target vCPUs are often idle/preempted.

---
  arch/loongarch/include/asm/kvm_host.h      |  1 +
  arch/loongarch/include/asm/kvm_para.h      |  1 +
  arch/loongarch/include/uapi/asm/kvm.h      |  1 +
  arch/loongarch/include/uapi/asm/kvm_para.h |  1 +
  arch/loongarch/kvm/trace.h                 | 15 ++++++++++
  arch/loongarch/kvm/vcpu.c                  | 34 +++++++++++++++++++++-
  arch/loongarch/kvm/vm.c                    |  3 ++
  7 files changed, 55 insertions(+), 1 deletion(-)
arch/loongarch/include/asm/kvm_host.h +1 -0
index 776bc487a705..1750632699e0 100644
@@ -168,6 +168,7 @@ enum emulation_result {
  #define LOONGARCH_PV_FEAT_MASK		(BIT(KVM_FEATURE_IPI) |		\
  					 BIT(KVM_FEATURE_PREEMPT) |	\
  					 BIT(KVM_FEATURE_STEAL_TIME) |	\
+					 BIT(KVM_FEATURE_PV_TLB_FLUSH) |\
  					 BIT(KVM_FEATURE_USER_HCALL) |	\
  					 BIT(KVM_FEATURE_VIRT_EXTIOI))
 
arch/loongarch/include/asm/kvm_para.h +1 -0
index fb17ba0fa101..28e3fa3b4c0e 100644
@@ -41,6 +41,7 @@ struct kvm_steal_time {
  	__u8  pad[47];
  };
  #define KVM_VCPU_PREEMPTED		(1 << 0)
+#define KVM_VCPU_FLUSH_TLB		(1 << 1)
 
  /*
   * Hypercall interface for KVM hypervisor
arch/loongarch/include/uapi/asm/kvm.h +1 -0
index cd0b5c11ca9c..e4cd4bbf8914 100644
@@ -106,6 +106,7 @@ struct kvm_fpu {
  #define  KVM_LOONGARCH_VM_FEAT_PTW		8
  #define  KVM_LOONGARCH_VM_FEAT_MSGINT		9
  #define  KVM_LOONGARCH_VM_FEAT_PV_PREEMPT	10
+#define  KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH	11
 
  /* Device Control API on vcpu fd */
  #define KVM_LOONGARCH_VCPU_CPUCFG	0
arch/loongarch/include/uapi/asm/kvm_para.h +1 -0
index d28cbcadd276..8872839251cc 100644
@@ -16,6 +16,7 @@
  #define  KVM_FEATURE_IPI		1
  #define  KVM_FEATURE_STEAL_TIME		2
  #define  KVM_FEATURE_PREEMPT		3
+#define  KVM_FEATURE_PV_TLB_FLUSH	4
  /* BIT 24 - 31 are features configurable by user space vmm */
  #define  KVM_FEATURE_VIRT_EXTIOI	24
  #define  KVM_FEATURE_USER_HCALL		25
arch/loongarch/kvm/trace.h +15 -0
index 3467ee22b704..8556954fa196 100644
@@ -210,6 +210,21 @@ TRACE_EVENT(kvm_vpid_change,
  	    TP_printk("VPID: 0x%08lx", __entry->vpid)
  );
 
+TRACE_EVENT(kvm_pv_tlb_flush,
+	TP_PROTO(struct kvm_vcpu *vcpu, bool need_flush),
+	TP_ARGS(vcpu, need_flush),
+	TP_STRUCT__entry(
+		__field(unsigned int, vcpu_id)
+		__field(bool, need_flush)
+	),
+	TP_fast_assign(
+		__entry->vcpu_id = vcpu->vcpu_id;
+		__entry->need_flush = need_flush;
+	),
+	TP_printk("vcpu %u need_flush %u", __entry->vcpu_id,
+		  __entry->need_flush)
+);
+
  #endif /* _TRACE_KVM_H */
 
  #undef TRACE_INCLUDE_PATH
arch/loongarch/kvm/vcpu.c +2 -1
index e28084c49e68..5230e95a7816 100644
@@ -173,7 +173,39 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
  	}
 
  	st = (struct kvm_steal_time __user *)ghc->hva;
-	if (kvm_guest_has_pv_feature(vcpu, KVM_FEATURE_PREEMPT)) {
+	if (kvm_guest_has_pv_feature(vcpu, KVM_FEATURE_PV_TLB_FLUSH)) {
+		u32 old = 0;

arch/loongarch/kvm/vcpu.c static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · line 187

Quoted 10 lines
arch/loongarch/kvm/vcpu.c +10 -0
+		int err = 0;
+
+		/*
+		 * Prime the cache line with a normal load before the coherent
+		 * atomic below; it was observed (when the line was last dirtied
+		 * by another core) to be needed for amand_db.w to see a current
+		 * value.  amand_db.w overwrites `old` with the real pre-AND value,
+		 * so this load contributes only its cache side-effect.
+		 */
+		unsafe_get_user(old, (u32 __user *)&st->preempted, out);

If there is a failure, it does nothing and go to out directly, however guest TLB remains stale without flush operation.

arch/loongarch/kvm/vcpu.c static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · line 200

Quoted 13 lines
arch/loongarch/kvm/vcpu.c +13 -0
+
+		/* Atomically read and clear the preempted byte via amand_db.w. */
+		asm volatile(
+		"1: amand_db.w %1, %3, %2	\n"
+		"2:				\n"
+		_ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1)
+		: "+r" (err), "+&r" (old),
+		  "+ZB" (*(u32 *)&st->preempted)
+		: "r" ((u32)~0xffu)
+		: "memory");
+
+		if (err)
+			goto out;

Ditto, if there is error, guest TLB is not flushed and vCPU continues to run with preempted bit set. The next TLB flush will be skipped with this vCPU since its preempted bit is not clear.

Regards
Bibo Mao

Quoted 35 lines
arch/loongarch/kvm/vcpu.c +8 -0
+
+		vcpu->arch.st.preempted = 0;
+
+		if ((u8)old & KVM_VCPU_FLUSH_TLB) {
+			vcpu->arch.vpid = 0;	/* Drop vpid to flush TLB */
+			trace_kvm_pv_tlb_flush(vcpu, true);
+		}
+	} else if (kvm_guest_has_pv_feature(vcpu, KVM_FEATURE_PREEMPT)) {
  		unsafe_put_user(0, &st->preempted, out);
  		vcpu->arch.st.preempted = 0;
  	}
arch/loongarch/kvm/vm.c +3 -0
index 1317c718f896..cfba45a7343c 100644
@@ -54,8 +54,10 @@ static void kvm_vm_init_features(struct kvm *kvm)
  	if (kvm_pvtime_supported()) {
  		kvm->arch.pv_features |= BIT(KVM_FEATURE_PREEMPT);
  		kvm->arch.pv_features |= BIT(KVM_FEATURE_STEAL_TIME);
+		kvm->arch.pv_features |= BIT(KVM_FEATURE_PV_TLB_FLUSH);
  		kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_PREEMPT);
  		kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_STEALTIME);
+		kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH);
  	}
  }
 
@@ -158,6 +160,7 @@ static int kvm_vm_feature_has_attr(struct kvm *kvm, struct kvm_device_attr *attr
  	case KVM_LOONGARCH_VM_FEAT_PV_IPI:
  	case KVM_LOONGARCH_VM_FEAT_PV_PREEMPT:
  	case KVM_LOONGARCH_VM_FEAT_PV_STEALTIME:
+	case KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH:
  		if (kvm_vm_support(&kvm->arch, attr->attr))
  			return 0;
  		return -ENXIO;
maintainer
#6 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

In reply to Tao Cui
Quoted 133 lines · Tao Cui

From: Tao Cui <[email protected]>

Add the guest-side implementation of PV TLB flush for LoongArch KVM guests, complementing the host-side support added in the previous commit.

When running as a KVM guest, remote TLB flushes are optimized by avoiding IPIs to preempted vCPUs:

- kvm_flush_tlb_mask() checks each target vCPU's steal-time preempted flag. If a vCPU is preempted, it atomically sets KVM_VCPU_FLUSH_TLB in the shared preempted byte via cmpxchg and removes that CPU from the IPI mask.
- Only non-preempted vCPUs receive IPIs via on_each_cpu_mask().
- When the host schedules a deferred-flush vCPU back in, it invalidates the VPID and flushes the TLB automatically.

All six SMP TLB flush functions (flush_tlb_all, flush_tlb_mm, flush_tlb_range, flush_tlb_kernel_range, flush_tlb_page, flush_tlb_one) are updated to use the PV path when the static key pv_tlb_flush_key is enabled.

The feature is gated by KVM_FEATURE_PV_TLB_FLUSH and requires KVM_FEATURE_STEAL_TIME (for the shared memory page). Per-CPU cpumask buffers are allocated via arch_initcall.

---
  arch/loongarch/include/asm/kvm_para.h |  8 ++++
  arch/loongarch/include/asm/paravirt.h | 21 ++++++++++
  arch/loongarch/kernel/paravirt.c      | 60 +++++++++++++++++++++++++++
  arch/loongarch/kernel/smp.c           | 30 +++++++++++---
  4 files changed, 113 insertions(+), 6 deletions(-)
arch/loongarch/include/asm/kvm_para.h +8 -0
index 28e3fa3b4c0e..7956aeff2436 100644
@@ -187,4 +187,12 @@ static inline bool kvm_check_and_clear_guest_paused(void)
  	return false;
  }
 
+static inline bool kvm_pv_tlb_flush_supported(void)
+{
+	unsigned int feat = kvm_arch_para_features();
+
+	return (feat & (1U << KVM_FEATURE_PV_TLB_FLUSH)) &&
+	       (feat & (1U << KVM_FEATURE_STEAL_TIME));
+}
+
  #endif /* _ASM_LOONGARCH_KVM_PARA_H */
arch/loongarch/include/asm/paravirt.h +21 -0
index acae1c5e5f88..6ce62b555a5d 100644
@@ -5,15 +5,26 @@
  #ifdef CONFIG_PARAVIRT
 
  #include <linux/jump_label.h>
+#include <linux/cpumask.h>
+#include <linux/smp.h>
 
  DECLARE_STATIC_KEY_FALSE(virt_preempt_key);
  DECLARE_STATIC_KEY_FALSE(virt_spin_lock_key);
+DECLARE_STATIC_KEY_FALSE(pv_tlb_flush_key);
  DECLARE_PER_CPU(struct kvm_steal_time, steal_time);
 
  int __init pv_ipi_init(void);
  int __init pv_time_init(void);
  int __init pv_spinlock_init(void);
 
+void kvm_flush_tlb_mask(const struct cpumask *cpumask,
+			smp_call_func_t func, void *info);
+
+static inline bool pv_tlb_flush_enabled(void)
+{
+	return static_branch_unlikely(&pv_tlb_flush_key);
+}
+
  #else
 
  static inline int pv_ipi_init(void)
@@ -31,5 +42,15 @@ static inline int pv_spinlock_init(void)
  	return 0;
  }
 
+static inline bool pv_tlb_flush_enabled(void)
+{
+	return false;
+}
+
+static inline void kvm_flush_tlb_mask(const struct cpumask *cpumask,
+				       smp_call_func_t func, void *info)
+{
+}
+
  #endif // CONFIG_PARAVIRT
  #endif
arch/loongarch/kernel/paravirt.c +20 -0
index 10821cce554c..eab84c9b94f6 100644
@@ -12,7 +12,9 @@
  static int has_steal_clock;
  DEFINE_STATIC_KEY_FALSE(virt_preempt_key);
  DEFINE_STATIC_KEY_FALSE(virt_spin_lock_key);
+DEFINE_STATIC_KEY_FALSE(pv_tlb_flush_key);
  DEFINE_PER_CPU(struct kvm_steal_time, steal_time) __aligned(64);
+static DEFINE_PER_CPU(cpumask_var_t, __pv_cpu_mask);
 
  static bool steal_acc = true;
 
@@ -208,6 +210,64 @@ int __init pv_ipi_init(void)
  	return 0;
  }
 
+#ifdef CONFIG_SMP
+void kvm_flush_tlb_mask(const struct cpumask *cpumask,
+			       smp_call_func_t func, void *info)
+{
+	int cpu;
+	struct kvm_steal_time *src;
+	struct cpumask *flushmask;
+
+	preempt_disable();
+	flushmask = this_cpu_cpumask_var_ptr(__pv_cpu_mask);
+	cpumask_copy(flushmask, cpumask);
+
+	/*
+	 * We have to call flush only on online vCPUs. And
+	 * queue flush_on_enter for pre-empted vCPUs
+	 */
+	for_each_cpu(cpu, flushmask) {
+		u32 *ptr, old, new;

arch/loongarch/kernel/paravirt.c int __init pv_ipi_init(void) · line 240

Quoted 10 lines
arch/loongarch/kernel/paravirt.c +10 -0
+
+		src = &per_cpu(steal_time, cpu);
+		ptr = (u32 *)&src->preempted;
+		old = READ_ONCE(*ptr);
+		if (!((u8)old & KVM_VCPU_PREEMPTED))
+			continue;
+
+		new = old | KVM_VCPU_FLUSH_TLB;
+		if (try_cmpxchg(ptr, &old, new))
+			__cpumask_clear_cpu(cpu, flushmask);

is it possible that vCPU is running however with KVM_VCPU_PREEMPTED set?

arch/loongarch/kernel/paravirt.c int __init pv_ipi_init(void) · line 243

Quoted 3 lines
arch/loongarch/kernel/paravirt.c +3 -0
+	}
+
+	on_each_cpu_mask(flushmask, func, info, 1);

LoongArch VM does not support post interrupt now, IPI with vCPU will trap to hypervisor.

There is another PV TLB flush method using hypercall like PV IPI, which will make a request with wait flag such as KVM_REQ_TLB_FLUSH and kick vCPU.

With this PV TLB flush method, there is not IPI interrupt in VM, and wait is handled in hypervisor, the negative effect is that it flushes all TLBs.

Regards
Bibo Mao

Quoted 106 lines
arch/loongarch/kernel/paravirt.c +27 -0
+	preempt_enable();
+}
+
+static int __init pv_tlb_flush_init(void)
+{
+	int cpu;
+
+	if (!kvm_pv_tlb_flush_supported())
+		return 0;
+
+	for_each_possible_cpu(cpu) {
+		if (!zalloc_cpumask_var_node(per_cpu_ptr(&__pv_cpu_mask, cpu),
+					    GFP_KERNEL, cpu_to_node(cpu))) {
+			while (--cpu >= 0)
+				free_cpumask_var(*per_cpu_ptr(&__pv_cpu_mask, cpu));
+			return -ENOMEM;
+		}
+	}
+
+	static_branch_enable(&pv_tlb_flush_key);
+	pr_info("KVM setup pv remote TLB flush\n");
+
+	return 0;
+}
+arch_initcall(pv_tlb_flush_init);
+#endif
+
  static int pv_enable_steal_time(void)
  {
  	int cpu = smp_processor_id();
arch/loongarch/kernel/smp.c +24 -6
index 50922610758b..bb3451b057ed 100644
@@ -727,7 +727,10 @@ static void flush_tlb_all_ipi(void *info)
 
  void flush_tlb_all(void)
  {
-	on_each_cpu(flush_tlb_all_ipi, NULL, 1);
+	if (pv_tlb_flush_enabled())
+		kvm_flush_tlb_mask(cpu_online_mask, flush_tlb_all_ipi, NULL);
+	else
+		on_each_cpu(flush_tlb_all_ipi, NULL, 1);
  }
 
  static void flush_tlb_mm_ipi(void *mm)
@@ -743,7 +746,10 @@ void flush_tlb_mm(struct mm_struct *mm)
  	preempt_disable();
 
  	if ((atomic_read(&mm->mm_users) != 1) || (current->mm != mm)) {
-		on_each_cpu_mask(mm_cpumask(mm), flush_tlb_mm_ipi, mm, 1);
+		if (pv_tlb_flush_enabled())
+			kvm_flush_tlb_mask(mm_cpumask(mm), flush_tlb_mm_ipi, mm);
+		else
+			on_each_cpu_mask(mm_cpumask(mm), flush_tlb_mm_ipi, mm, 1);
  	} else {
  		unsigned int cpu;
 
@@ -782,7 +788,10 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned l
  			.addr2 = end,
  		};
 
-		on_each_cpu_mask(mm_cpumask(mm), flush_tlb_range_ipi, &fd, 1);
+		if (pv_tlb_flush_enabled())
+			kvm_flush_tlb_mask(mm_cpumask(mm), flush_tlb_range_ipi, &fd);
+		else
+			on_each_cpu_mask(mm_cpumask(mm), flush_tlb_range_ipi, &fd, 1);
  	} else {
  		unsigned int cpu;
 
@@ -809,7 +818,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
  		.addr2 = end,
  	};
 
-	on_each_cpu(flush_tlb_kernel_range_ipi, &fd, 1);
+	if (pv_tlb_flush_enabled())
+		kvm_flush_tlb_mask(cpu_online_mask, flush_tlb_kernel_range_ipi, &fd);
+	else
+		on_each_cpu(flush_tlb_kernel_range_ipi, &fd, 1);
  }
 
  static void flush_tlb_page_ipi(void *info)
@@ -828,7 +840,10 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long page)
  			.addr1 = page,
  		};
 
-		on_each_cpu_mask(mm_cpumask(vma->vm_mm), flush_tlb_page_ipi, &fd, 1);
+		if (pv_tlb_flush_enabled())
+			kvm_flush_tlb_mask(mm_cpumask(vma->vm_mm), flush_tlb_page_ipi, &fd);
+		else
+			on_each_cpu_mask(mm_cpumask(vma->vm_mm), flush_tlb_page_ipi, &fd, 1);
  	} else {
  		unsigned int cpu;
 
@@ -851,6 +866,9 @@ static void flush_tlb_one_ipi(void *info)
 
  void flush_tlb_one(unsigned long vaddr)
  {
-	on_each_cpu(flush_tlb_one_ipi, (void *)vaddr, 1);
+	if (pv_tlb_flush_enabled())
+		kvm_flush_tlb_mask(cpu_online_mask, flush_tlb_one_ipi, (void *)vaddr);
+	else
+		on_each_cpu(flush_tlb_one_ipi, (void *)vaddr, 1);
  }
  EXPORT_SYMBOL(flush_tlb_one);
maintainer
#18 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

In reply to Tao Cui
Quoted 138 lines · Tao Cui

From: Tao Cui <[email protected]>

Add the guest-side implementation of PV TLB flush for LoongArch KVM guests, complementing the host-side support added in the previous commit.

When running as a KVM guest, remote TLB flushes are optimized by avoiding IPIs to preempted vCPUs:

- kvm_flush_tlb_mask() checks each target vCPU's steal-time preempted flag. If a vCPU is preempted, it atomically sets KVM_VCPU_FLUSH_TLB in the shared preempted byte via cmpxchg and removes that CPU from the IPI mask.
- Only non-preempted vCPUs receive IPIs via on_each_cpu_mask().
- When the host schedules a deferred-flush vCPU back in, it invalidates the VPID and flushes the TLB automatically.

All six SMP TLB flush functions (flush_tlb_all, flush_tlb_mm, flush_tlb_range, flush_tlb_kernel_range, flush_tlb_page, flush_tlb_one) are updated to use the PV path when the static key pv_tlb_flush_key is enabled.

The feature is gated by KVM_FEATURE_PV_TLB_FLUSH and requires KVM_FEATURE_STEAL_TIME (for the shared memory page). Per-CPU cpumask buffers are allocated via arch_initcall.

---
  arch/loongarch/include/asm/kvm_para.h |  8 ++++
  arch/loongarch/include/asm/paravirt.h | 21 ++++++++++
  arch/loongarch/kernel/paravirt.c      | 60 +++++++++++++++++++++++++++
  arch/loongarch/kernel/smp.c           | 30 +++++++++++---
  4 files changed, 113 insertions(+), 6 deletions(-)
arch/loongarch/include/asm/kvm_para.h +8 -0
index 28e3fa3b4c0e..7956aeff2436 100644
@@ -187,4 +187,12 @@ static inline bool kvm_check_and_clear_guest_paused(void)
  	return false;
  }
 
+static inline bool kvm_pv_tlb_flush_supported(void)
+{
+	unsigned int feat = kvm_arch_para_features();
+
+	return (feat & (1U << KVM_FEATURE_PV_TLB_FLUSH)) &&
+	       (feat & (1U << KVM_FEATURE_STEAL_TIME));
+}
+
  #endif /* _ASM_LOONGARCH_KVM_PARA_H */
arch/loongarch/include/asm/paravirt.h +21 -0
index acae1c5e5f88..6ce62b555a5d 100644
@@ -5,15 +5,26 @@
  #ifdef CONFIG_PARAVIRT
 
  #include <linux/jump_label.h>
+#include <linux/cpumask.h>
+#include <linux/smp.h>
 
  DECLARE_STATIC_KEY_FALSE(virt_preempt_key);
  DECLARE_STATIC_KEY_FALSE(virt_spin_lock_key);
+DECLARE_STATIC_KEY_FALSE(pv_tlb_flush_key);
  DECLARE_PER_CPU(struct kvm_steal_time, steal_time);
 
  int __init pv_ipi_init(void);
  int __init pv_time_init(void);
  int __init pv_spinlock_init(void);
 
+void kvm_flush_tlb_mask(const struct cpumask *cpumask,
+			smp_call_func_t func, void *info);
+
+static inline bool pv_tlb_flush_enabled(void)
+{
+	return static_branch_unlikely(&pv_tlb_flush_key);
+}
+
  #else
 
  static inline int pv_ipi_init(void)
@@ -31,5 +42,15 @@ static inline int pv_spinlock_init(void)
  	return 0;
  }
 
+static inline bool pv_tlb_flush_enabled(void)
+{
+	return false;
+}
+
+static inline void kvm_flush_tlb_mask(const struct cpumask *cpumask,
+				       smp_call_func_t func, void *info)
+{
+}
+
  #endif // CONFIG_PARAVIRT
  #endif
arch/loongarch/kernel/paravirt.c +25 -0
index 10821cce554c..eab84c9b94f6 100644
@@ -12,7 +12,9 @@
  static int has_steal_clock;
  DEFINE_STATIC_KEY_FALSE(virt_preempt_key);
  DEFINE_STATIC_KEY_FALSE(virt_spin_lock_key);
+DEFINE_STATIC_KEY_FALSE(pv_tlb_flush_key);
  DEFINE_PER_CPU(struct kvm_steal_time, steal_time) __aligned(64);
+static DEFINE_PER_CPU(cpumask_var_t, __pv_cpu_mask);
 
  static bool steal_acc = true;
 
@@ -208,6 +210,64 @@ int __init pv_ipi_init(void)
  	return 0;
  }
 
+#ifdef CONFIG_SMP
+void kvm_flush_tlb_mask(const struct cpumask *cpumask,
+			       smp_call_func_t func, void *info)
+{
+	int cpu;
+	struct kvm_steal_time *src;
+	struct cpumask *flushmask;
+
+	preempt_disable();
+	flushmask = this_cpu_cpumask_var_ptr(__pv_cpu_mask);
+	cpumask_copy(flushmask, cpumask);
+
+	/*
+	 * We have to call flush only on online vCPUs. And
+	 * queue flush_on_enter for pre-empted vCPUs
+	 */
+	for_each_cpu(cpu, flushmask) {
+		u32 *ptr, old, new;
+
+		src = &per_cpu(steal_time, cpu);
+		ptr = (u32 *)&src->preempted;
+		old = READ_ONCE(*ptr);
+		if (!((u8)old & KVM_VCPU_PREEMPTED))

arch/loongarch/kernel/paravirt.c int __init pv_ipi_init(void) · line 245

Quoted 10 lines
arch/loongarch/kernel/paravirt.c +10 -0
+			continue;
+
+		new = old | KVM_VCPU_FLUSH_TLB;
+		if (try_cmpxchg(ptr, &old, new))
+			__cpumask_clear_cpu(cpu, flushmask);
+	}
+
+	on_each_cpu_mask(flushmask, func, info, 1);
+	preempt_enable();
+}

FYI, another PV TLB flush method in arch/x86/hyperv/mmu.c on hyperV which is complicated and fine-gained.

Regards
Bibo Mao

Quoted 104 lines
arch/loongarch/kernel/paravirt.c +25 -0
+
+static int __init pv_tlb_flush_init(void)
+{
+	int cpu;
+
+	if (!kvm_pv_tlb_flush_supported())
+		return 0;
+
+	for_each_possible_cpu(cpu) {
+		if (!zalloc_cpumask_var_node(per_cpu_ptr(&__pv_cpu_mask, cpu),
+					    GFP_KERNEL, cpu_to_node(cpu))) {
+			while (--cpu >= 0)
+				free_cpumask_var(*per_cpu_ptr(&__pv_cpu_mask, cpu));
+			return -ENOMEM;
+		}
+	}
+
+	static_branch_enable(&pv_tlb_flush_key);
+	pr_info("KVM setup pv remote TLB flush\n");
+
+	return 0;
+}
+arch_initcall(pv_tlb_flush_init);
+#endif
+
  static int pv_enable_steal_time(void)
  {
  	int cpu = smp_processor_id();
arch/loongarch/kernel/smp.c +24 -6
index 50922610758b..bb3451b057ed 100644
@@ -727,7 +727,10 @@ static void flush_tlb_all_ipi(void *info)
 
  void flush_tlb_all(void)
  {
-	on_each_cpu(flush_tlb_all_ipi, NULL, 1);
+	if (pv_tlb_flush_enabled())
+		kvm_flush_tlb_mask(cpu_online_mask, flush_tlb_all_ipi, NULL);
+	else
+		on_each_cpu(flush_tlb_all_ipi, NULL, 1);
  }
 
  static void flush_tlb_mm_ipi(void *mm)
@@ -743,7 +746,10 @@ void flush_tlb_mm(struct mm_struct *mm)
  	preempt_disable();
 
  	if ((atomic_read(&mm->mm_users) != 1) || (current->mm != mm)) {
-		on_each_cpu_mask(mm_cpumask(mm), flush_tlb_mm_ipi, mm, 1);
+		if (pv_tlb_flush_enabled())
+			kvm_flush_tlb_mask(mm_cpumask(mm), flush_tlb_mm_ipi, mm);
+		else
+			on_each_cpu_mask(mm_cpumask(mm), flush_tlb_mm_ipi, mm, 1);
  	} else {
  		unsigned int cpu;
 
@@ -782,7 +788,10 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned l
  			.addr2 = end,
  		};
 
-		on_each_cpu_mask(mm_cpumask(mm), flush_tlb_range_ipi, &fd, 1);
+		if (pv_tlb_flush_enabled())
+			kvm_flush_tlb_mask(mm_cpumask(mm), flush_tlb_range_ipi, &fd);
+		else
+			on_each_cpu_mask(mm_cpumask(mm), flush_tlb_range_ipi, &fd, 1);
  	} else {
  		unsigned int cpu;
 
@@ -809,7 +818,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
  		.addr2 = end,
  	};
 
-	on_each_cpu(flush_tlb_kernel_range_ipi, &fd, 1);
+	if (pv_tlb_flush_enabled())
+		kvm_flush_tlb_mask(cpu_online_mask, flush_tlb_kernel_range_ipi, &fd);
+	else
+		on_each_cpu(flush_tlb_kernel_range_ipi, &fd, 1);
  }
 
  static void flush_tlb_page_ipi(void *info)
@@ -828,7 +840,10 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long page)
  			.addr1 = page,
  		};
 
-		on_each_cpu_mask(mm_cpumask(vma->vm_mm), flush_tlb_page_ipi, &fd, 1);
+		if (pv_tlb_flush_enabled())
+			kvm_flush_tlb_mask(mm_cpumask(vma->vm_mm), flush_tlb_page_ipi, &fd);
+		else
+			on_each_cpu_mask(mm_cpumask(vma->vm_mm), flush_tlb_page_ipi, &fd, 1);
  	} else {
  		unsigned int cpu;
 
@@ -851,6 +866,9 @@ static void flush_tlb_one_ipi(void *info)
 
  void flush_tlb_one(unsigned long vaddr)
  {
-	on_each_cpu(flush_tlb_one_ipi, (void *)vaddr, 1);
+	if (pv_tlb_flush_enabled())
+		kvm_flush_tlb_mask(cpu_online_mask, flush_tlb_one_ipi, (void *)vaddr);
+	else
+		on_each_cpu(flush_tlb_one_ipi, (void *)vaddr, 1);
  }
  EXPORT_SYMBOL(flush_tlb_one);
author
#4 lore raw

Re: [PATCH v4 1/3] LoongArch: KVM: Add PV TLB flush support via steal-time shared memory

In reply to Bibo Mao

Hi Bibo,

Thank you for the review.

You're right, and honestly I didn't think the
error path through carefully enough here.

The current handling leaves the preempted byte uncleared and skips the flush, which is exactly the cascade you described: a vCPU re-entered on the error path keeps running with the bit set, so later flushes keep deferring it and its TLB stays stale until some later entry happens to succeed. A stale TLB is a correctness issue, so this path needs conservative handling even though the fault itself is rare.

I'll drop the VPID on error so the TLB is flushed conservatively and the logic self-heals:

    if (err) {
        vcpu->arch.vpid = 0;    /* be conservative: flush anyway */
        goto out;
    }

The byte isn't cleared on the fault, but the TLB is clean this round; the next successful amand_db.w reads the still-set FLUSH_TLB bit and flushes again (redundantly but harmlessly), then clears the byte. The same covers the unsafe_get_user() priming-load failure.

I'll fix this in the next version.

Thanks,
Tao

Quoted 37 lines · Bibo Mao
Quoted 8 lines
arch/loongarch/kvm/vcpu.c +2 -1
@@ -173,7 +173,39 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
       }
         st = (struct kvm_steal_time __user *)ghc->hva;
-    if (kvm_guest_has_pv_feature(vcpu, KVM_FEATURE_PREEMPT)) {
+    if (kvm_guest_has_pv_feature(vcpu, KVM_FEATURE_PV_TLB_FLUSH)) {
+        u32 old = 0;

arch/loongarch/kvm/vcpu.c static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · line 186

Quoted 10 lines
arch/loongarch/kvm/vcpu.c +10 -0
+        int err = 0;
+
+        /*
+         * Prime the cache line with a normal load before the coherent
+         * atomic below; it was observed (when the line was last dirtied
+         * by another core) to be needed for amand_db.w to see a current
+         * value.  amand_db.w overwrites `old` with the real pre-AND value,
+         * so this load contributes only its cache side-effect.
+         */
+        unsafe_get_user(old, (u32 __user *)&st->preempted, out);

If there is a failure, it does nothing and go to out directly, however guest TLB remains stale without flush operation.

arch/loongarch/kvm/vcpu.c static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · line 200

Quoted 13 lines
arch/loongarch/kvm/vcpu.c +13 -0
+
+        /* Atomically read and clear the preempted byte via amand_db.w. */
+        asm volatile(
+        "1: amand_db.w %1, %3, %2    \n"
+        "2:                \n"
+        _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1)
+        : "+r" (err), "+&r" (old),
+          "+ZB" (*(u32 *)&st->preempted)
+        : "r" ((u32)~0xffu)
+        : "memory");
+
+        if (err)
+            goto out;

Ditto, if there is error, guest TLB is not flushed and vCPU continues to run with preempted bit set. The next TLB flush will be skipped with this vCPU since its preempted bit is not clear.

Regards
Bibo Mao

author
#7 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

In reply to Bibo Mao

Hi Bibo,

That's really kind of you. I'd really value the chance to work with you.

Quoted 3 lines · Bibo Mao
Quoted 2 lines

+            __cpumask_clear_cpu(cpu, flushmask);

is it possible that vCPU is running however with KVM_VCPU_PREEMPTED set?

It shouldn't happen with a correct host: the bit is set on vcpu_put (preempt) and cleared on entry through kvm_update_stolen_time(), so a running vCPU always has preempted == 0. The guest-side race is closed by try_cmpxchg(). If the host clears the byte between the guest's READ_ONCE and the cmpxchg, the cmpxchg fails and the CPU stays in the IPI mask, so no flush is missed. The only thing that can break the invariant is the error path from patch 1, which the conservative handling there closes. Point taken though: the design's correctness rests entirely on the host honoring this, so I agree the patch-1 fix is necessary rather than optional.

Quoted 9 lines
Quoted 3 lines

+    }
+
+    on_each_cpu_mask(flushmask, func, info, 1);

LoongArch VM does not support post interrupt now, IPI with vCPU will trap to hypervisor.

There is another PV TLB flush method using hypercall like PV IPI, which will make a request with wait flag such as KVM_REQ_TLB_FLUSH and kick vCPU.

With this PV TLB flush method, there is not IPI interrupt in VM, and wait is handled in hypervisor, the negative effect is that it flushes all TLBs.

This is the point I most want to get right, and you're correct. Since LoongArch has no posted interrupts, an IPI to a *running* vCPU still traps, so the steal-time path only genuinely helps for preempted/idle vCPUs; for running ones it's no better (a little heavier) than a plain IPI. The hypercall approach (analogous to our PV IPI) would avoid in-guest IPIs entirely, at the cost of a full TLB flush and hypervisor-side wait handling.

Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.

Quoted 1 line

FYI, another PV TLB flush method in arch/x86/hyperv/mmu.c ...

Thanks for the pointer. The Hyper-V path is a pure hypercall: the guest hands the hypervisor the address space, an address-range list, and the target VP set, so the flush stays fine-grained instead of nuking the whole TLB. That's a good template if the numbers point us toward a hypercall-based design.

Thanks,
Tao

author
#22 lore raw

Re: [PATCH v4 3/3] KVM: selftests: loongarch: Add PV TLB flush performance test

In reply to Bibo Mao

From: Tao Cui <[email protected]>

Thanks, and I share the stability concern. I don't want to land something that adds risk for a feature that isn't clearly earning its place yet.

Quoted 2 lines

Do you have real workload to show the benefits of the PV TLB flush, and which TLB APIs should be hooked.

Completely fair, and to be upfront I don't have a large-scale real-world workload to point to yet. So my plan is to build a set of local test scripts that simulate as many usage scenarios as possible: mmap/fork-heavy paths, kernel-range flushes, and an overcommitted setup where vCPU preemption is frequent. I'll report the IPI and TLB-flush overhead with and without the feature, broken down per flush API. The coverage is synthetic rather than a single flagship workload, but hopefully it can at least give us a rough, preliminary sense of whether the feature is worth pursuing.

Thanks,
Tao

maintainer
#23 lore raw

Re: [PATCH v4 3/3] KVM: selftests: loongarch: Add PV TLB flush performance test

Quoted 8 lines · Tao Cui

From: Tao Cui <[email protected]>

Thanks, and I share the stability concern. I don't want to land something that adds risk for a feature that isn't clearly earning its place yet.

Quoted 2 lines

Do you have real workload to show the benefits of the PV TLB flush, and which TLB APIs should be hooked.

Quoted 9 lines

Completely fair, and to be upfront I don't have a large-scale real-world workload to point to yet. So my plan is to build a set of local test scripts that simulate as many usage scenarios as possible: mmap/fork-heavy paths, kernel-range flushes, and an overcommitted setup where vCPU preemption is frequent. I'll report the IPI and TLB-flush overhead with and without the feature, broken down per flush API. The coverage is synthetic rather than a single flagship workload, but hopefully it can at least give us a rough, preliminary sense of whether the feature is worth pursuing.

By the original patch from x86, https://lwn.net/Articles/740363/
There is ebissy benchmark, maybe we can use this test case.

Regards
Bibo Mao

Quoted 4 lines

Thanks,
Tao

maintainer
#8 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

In reply to Tao Cui
Quoted 5 lines · Tao Cui

Hi Bibo,

That's really kind of you. I'd really value the chance to work with you.

Quoted 7 lines
Quoted 3 lines · Bibo Mao
Quoted 2 lines

+            __cpumask_clear_cpu(cpu, flushmask);

is it possible that vCPU is running however with KVM_VCPU_PREEMPTED set?

It shouldn't happen with a correct host: the bit is set on vcpu_put (preempt) and cleared on entry through kvm_update_stolen_time(), so a

From the code, I think it may happen. The clear operation may fail in kvm_update_stolen_time(), so vCPU will continue to run in guest mode with flag KVM_VCPU_PREEMPTED set .

Quoted 26 lines

running vCPU always has preempted == 0. The guest-side race is closed by try_cmpxchg(). If the host clears the byte between the guest's READ_ONCE and the cmpxchg, the cmpxchg fails and the CPU stays in the IPI mask, so no flush is missed. The only thing that can break the invariant is the error path from patch 1, which the conservative handling there closes. Point taken though: the design's correctness rests entirely on the host honoring this, so I agree the patch-1 fix is necessary rather than optional.

Quoted 9 lines
Quoted 3 lines

+    }
+
+    on_each_cpu_mask(flushmask, func, info, 1);

LoongArch VM does not support post interrupt now, IPI with vCPU will trap to hypervisor.

There is another PV TLB flush method using hypercall like PV IPI, which will make a request with wait flag such as KVM_REQ_TLB_FLUSH and kick vCPU.

With this PV TLB flush method, there is not IPI interrupt in VM, and wait is handled in hypervisor, the negative effect is that it flushes all TLBs.

This is the point I most want to get right, and you're correct. Since LoongArch has no posted interrupts, an IPI to a *running* vCPU still traps, so the steal-time path only genuinely helps for preempted/idle vCPUs; for running ones it's no better (a little heavier) than a plain IPI. The hypercall approach (analogous to our PV IPI) would avoid in-guest IPIs entirely, at the cost of a full TLB flush and hypervisor-side wait handling.

Quoted 4 lines

Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.

well, that is the best. It is my pleasure to discuss this with you.

Regards
Bibo Mao

Quoted 12 lines
Quoted 2 lines

FYI, another PV TLB flush method in arch/x86/hyperv/mmu.c ...

Thanks for the pointer. The Hyper-V path is a pure hypercall: the guest hands the hypervisor the address space, an address-range list, and the target VP set, so the flush stays fine-grained instead of nuking the whole TLB. That's a good template if the numbers point us toward a hypercall-based design.

Thanks,
Tao

author
#9 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

Hi Bibo,

Quoted 7 lines · Bibo Mao
Quoted 5 lines

Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.

well, that is the best. It is my pleasure to discuss this with you.

A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.

The two workloads:
- ebizzy (all threads busy, mm-flush heavy)
- tlb_bench in sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)

ebizzy (records/s, higher is better), 32 vCPUs:

  no-PV       ~103,737
  hypercall   ~105,779
  steal-time  ~105,872

-> all within noise (±2%); no measurable difference.

tlb_bench sleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
no-PV ~166,536
steal-time ~149,553
hypercall ~88,686

ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the tlb_bench figure above.

So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.

Best,
Tao

maintainer
#10 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

Quoted 22 lines · Tao Cui

Hi Bibo,

Quoted 7 lines · Bibo Mao
Quoted 5 lines

Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.

well, that is the best. It is my pleasure to discuss this with you.

A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.

The two workloads:
- ebizzy (all threads busy, mm-flush heavy)
- tlb_bench in sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)

Quoted 5 lines

ebizzy (records/s, higher is better), 32 vCPUs:

   no-PV       ~103,737
   hypercall   ~105,779
   steal-time  ~105,872

-> all within noise (±2%); no measurable difference.

Hi Tao,

what is ebizzy command? ebizzy -m or ebizzy -M.

could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?

Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
./ebizzy -m ./ebizzy -M

host             8633                 158898
VM(32 vCPUs)     6610                 133153
VM/host          76%                  83%

Regards
Bibo Mao

Quoted 21 lines

tlb_bench sleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:

   no-PV       ~166,536
   steal-time  ~149,553
   hypercall    ~88,686

ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the tlb_bench figure above.

So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.

Best,
Tao

maintainer
#11 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

Quoted 34 lines · Bibo Mao
Quoted 22 lines · Tao Cui

Hi Bibo,

Quoted 7 lines · Bibo Mao
Quoted 5 lines

Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.

well, that is the best. It is my pleasure to discuss this with you.

A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.

The two workloads:
  - ebizzy (all threads busy, mm-flush heavy)
  - tlb_bench in sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)

Quoted 5 lines

ebizzy (records/s, higher is better), 32 vCPUs:
   no-PV       ~103,737
   hypercall   ~105,779
   steal-time  ~105,872
   -> all within noise (±2%); no measurable difference.

Hi Tao,

what is ebizzy command? ebizzy -m or ebizzy -M.

Quoted 8 lines

could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?

Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
                ./ebizzy -m          ./ebizzy -M
host             8633                 158898
VM(32 vCPUs)     6610                 133153
VM/host          76%                  83%

just ./ebizzy -M is enough, it seems that CPU number is one key factor.

Quoted 24 lines

Regards
Bibo Mao

Quoted 21 lines

tlb_bench sleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
   no-PV       ~166,536
   steal-time  ~149,553
   hypercall    ~88,686

ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the tlb_bench figure above.

So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.

Best,
Tao

author
#12 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui

Hi Bibo,

Quoted 7 lines · Bibo Mao
Quoted 5 lines

Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.

well, that is the best. It is my pleasure to discuss this with you.

A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.

The two workloads:
  - ebizzy (all threads busy, mm-flush heavy)
  - tlb_bench in sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)

Quoted 5 lines

ebizzy (records/s, higher is better), 32 vCPUs:
   no-PV       ~103,737
   hypercall   ~105,779
   steal-time  ~105,872
   -> all within noise (±2%); no measurable difference.

Hi Tao,

what is ebizzy command? ebizzy -m or ebizzy -M.

could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?

Quoted 8 lines
Quoted 6 lines

Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
                 ./ebizzy -m          ./ebizzy -M
host             8633                 158898
VM(32 vCPUs)     6610                 133153
VM/host          76%                  83%

just ./ebizzy -M is enough, it seems that CPU number is one key factor.

Sorry for the delay — it turned out my ebizzy command was wrong. I had been running ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.

I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).

                 ebizzy -m        ebizzy -M
host             ~20,000          ~55,000
VM (8 vCPU,1:1)  ~17,000          ~53,000
VM/host          ~86%             ~97%

The -m ratio (86%) is close to your 76%.

I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have CONFIG_PARAVIRT enabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.

I'll share the verified numbers once I have them.

Thanks,
Tao

Quoted 25 lines
Quoted 24 lines

Regards
Bibo Mao

Quoted 21 lines

tlb_bench sleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
   no-PV       ~166,536
   steal-time  ~149,553
   hypercall    ~88,686

ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the tlb_bench figure above.

So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.

Best,
Tao

maintainer
#13 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

Quoted 59 lines · Tao Cui
Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui

Hi Bibo,

Quoted 7 lines · Bibo Mao
Quoted 5 lines

Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.

well, that is the best. It is my pleasure to discuss this with you.

A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.

The two workloads:
  - ebizzy (all threads busy, mm-flush heavy)
  - tlb_bench in sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)

Quoted 5 lines

ebizzy (records/s, higher is better), 32 vCPUs:
   no-PV       ~103,737
   hypercall   ~105,779
   steal-time  ~105,872
   -> all within noise (±2%); no measurable difference.

Hi Tao,

what is ebizzy command? ebizzy -m or ebizzy -M.

could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?

Quoted 8 lines
Quoted 6 lines

Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
                 ./ebizzy -m          ./ebizzy -M
host             8633                 158898
VM(32 vCPUs)     6610                 133153
VM/host          76%                  83%

just ./ebizzy -M is enough, it seems that CPU number is one key factor.

Sorry for the delay — it turned out my ebizzy command was wrong. I had been running ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.

I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).

Quoted 6 lines

ebizzy -m ebizzy -M

host             ~20,000          ~55,000
VM (8 vCPU,1:1)  ~17,000          ~53,000
VM/host          ~86%             ~97%

The -m ratio (86%) is close to your 76%.

On my 3C5000 Dual-way machine, VM has the same CPU/memory topology with physical machine, the kernel is mainline without any patch.

                          ./ebizzy -M
Host (32 pCPUs)           158898
One VM(32 vCPUs)          133153                 83% of host
Two VMs(32 vCPUs each)    9083 + 9630 = 18713    11% of host

It seems that with ebizzy benchmark, there is big difference if vCPU is preempted. Even if vCPU is not preempted, the performance is only 83% of host on my 3C5000 Dual-way machine.

Regards
Bibo Mao

Quoted 41 lines

I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have CONFIG_PARAVIRT enabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.

I'll share the verified numbers once I have them.

Thanks,
Tao

Quoted 25 lines
Quoted 24 lines

Regards
Bibo Mao

Quoted 21 lines

tlb_bench sleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
   no-PV       ~166,536
   steal-time  ~149,553
   hypercall    ~88,686

ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the tlb_bench figure above.

So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.

Best,
Tao

author
#14 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

Quoted 67 lines · Bibo Mao
Quoted 64 lines · Tao Cui
Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui

Hi Bibo,

Quoted 7 lines · Bibo Mao
Quoted 5 lines

Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.

well, that is the best. It is my pleasure to discuss this with you.

A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.

The two workloads:
   - ebizzy (all threads busy, mm-flush heavy)
   - tlb_bench in sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)

Quoted 5 lines

ebizzy (records/s, higher is better), 32 vCPUs:
    no-PV       ~103,737
    hypercall   ~105,779
    steal-time  ~105,872
    -> all within noise (±2%); no measurable difference.

Hi Tao,

what is ebizzy command? ebizzy -m or ebizzy -M.

could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?

Quoted 8 lines
Quoted 6 lines

Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
                  ./ebizzy -m          ./ebizzy -M
host             8633                 158898
VM(32 vCPUs)     6610                 133153
VM/host          76%                  83%

just ./ebizzy -M is enough, it seems that CPU number is one key factor.

Sorry for the delay — it turned out my ebizzy command was wrong. I had been running ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.

I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).

                  ebizzy -m        ebizzy -M
host             ~20,000          ~55,000
VM (8 vCPU,1:1)  ~17,000          ~53,000
VM/host          ~86%             ~97%

Quoted 8 lines
Quoted 1 line

The -m ratio (86%) is close to your 76%.

On my 3C5000 Dual-way machine, VM has the same CPU/memory topology with physical machine, the kernel is mainline without any patch. ./ebizzy -M
Host (32 pCPUs)           158898
One VM(32 vCPUs)          133153                 83% of host
Two VMs(32 vCPUs each)    9083 + 9630 = 18713    11% of host

It seems that with ebizzy benchmark, there is big difference if vCPU is preempted. Even if vCPU is not preempted, the performance is only 83% of host on my 3C5000 Dual-way machine.

After fixing the ebizzy command, I
have multi-VM overcommit results for both approaches.

Setup: 8-core LoongArch (single-socket, single NUMA), KVM, linux-next-20260623, 8 vCPU per VM. All VMs run ebizzy -M simultaneously, 3 runs each. The PV-off baseline uses a guest kernel with CONFIG_PARAVIRT=y but without the PV TLB flush patches, so PV IPI and steal-time are active in all three columns.

ebizzy -M, total records/s across all VMs:

              PV-off      steal-time   hypercall
  1:1 (1VM)   53,600      53,800       53,900
  2:1 (2VM)    2,600      42,600       45,300
  3:1 (3VM)    2,800      44,700       46,000

At 1:1 there is no difference — no vCPU gets preempted. Under overcommit, without PV TLB flush the throughput drops to ~3-5% of the single-VM case, because every remote TLB flush sends IPIs to preempted vCPUs. With either PV TLB flush variant, preempted vCPUs are skipped, and total throughput stays at ~85-90% of single-VM (bounded by physical cores).

Hypercall is consistently 3-6% above steal-time in the overcommit cases. A possible reason is that the hypercall hands the entire target set to the host in one call, while steal-time still IPIs the running vCPUs and only defers the preempted ones.

On our 8-core machine the collapse is more severe than on your 3C5000 (~5% vs 11% of host at 2 VMs), likely due to the smaller core count and single-NUMA topology.

Thanks,
Tao

Quoted 45 lines

Regards
Bibo Mao

Quoted 41 lines

I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have CONFIG_PARAVIRT enabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.

I'll share the verified numbers once I have them.

Thanks,
Tao

Quoted 25 lines
Quoted 24 lines

Regards
Bibo Mao

Quoted 21 lines

tlb_bench sleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
    no-PV       ~166,536
    steal-time  ~149,553
    hypercall    ~88,686

ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the tlb_bench figure above.

So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.

Best,
Tao

maintainer
#15 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

Quoted 88 lines · Tao Cui
Quoted 67 lines · Bibo Mao
Quoted 64 lines · Tao Cui
Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui

Hi Bibo,

Quoted 7 lines · Bibo Mao
Quoted 5 lines

Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.

well, that is the best. It is my pleasure to discuss this with you.

A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.

The two workloads:
   - ebizzy (all threads busy, mm-flush heavy)
   - tlb_bench in sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)

Quoted 5 lines

ebizzy (records/s, higher is better), 32 vCPUs:
    no-PV       ~103,737
    hypercall   ~105,779
    steal-time  ~105,872
    -> all within noise (±2%); no measurable difference.

Hi Tao,

what is ebizzy command? ebizzy -m or ebizzy -M.

could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?

Quoted 8 lines
Quoted 6 lines

Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
                  ./ebizzy -m          ./ebizzy -M
host             8633                 158898
VM(32 vCPUs)     6610                 133153
VM/host          76%                  83%

just ./ebizzy -M is enough, it seems that CPU number is one key factor.

Sorry for the delay — it turned out my ebizzy command was wrong. I had been running ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.

I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).

                  ebizzy -m        ebizzy -M
host             ~20,000          ~55,000
VM (8 vCPU,1:1)  ~17,000          ~53,000
VM/host          ~86%             ~97%

Quoted 8 lines
Quoted 1 line

The -m ratio (86%) is close to your 76%.

On my 3C5000 Dual-way machine, VM has the same CPU/memory topology with physical machine, the kernel is mainline without any patch.
                          ./ebizzy -M
Host (32 pCPUs)           158898
One VM(32 vCPUs)          133153                 83% of host
Two VMs(32 vCPUs each)    9083 + 9630 = 18713    11% of host

It seems that with ebizzy benchmark, there is big difference if vCPU is preempted. Even if vCPU is not preempted, the performance is only 83% of host on my 3C5000 Dual-way machine.

After fixing the ebizzy command, I
have multi-VM overcommit results for both approaches.

Setup: 8-core LoongArch (single-socket, single NUMA), KVM, linux-next-20260623, 8 vCPU per VM. All VMs run ebizzy -M simultaneously, 3 runs each. The PV-off baseline uses a guest kernel with CONFIG_PARAVIRT=y but without the PV TLB flush patches, so PV IPI and steal-time are active in all three columns.

Quoted 8 lines

ebizzy -M, total records/s across all VMs:

               PV-off      steal-time   hypercall
   1:1 (1VM)   53,600      53,800       53,900
   2:1 (2VM)    2,600      42,600       45,300
   3:1 (3VM)    2,800      44,700       46,000

At 1:1 there is no difference — no vCPU gets preempted. Under

From the previous result, the score on host is 55,000, with 1:1 mode the virtualization efficiency is 97%, it is hard to improve actually. On my 3C5000 Dual-way machine, the virtualization efficiency is 83%, I think it can improve even with 1:1 mode.

Also the paper at https://dl.acm.org/doi/pdf/10.1145/2892242.2892245 proves it with 1:1 mode.

Quoted 6 lines

overcommit, without PV TLB flush the throughput drops to ~3-5% of the single-VM case, because every remote TLB flush sends IPIs to preempted vCPUs. With either PV TLB flush variant, preempted vCPUs are skipped, and total throughput stays at ~85-90% of single-VM (bounded by physical cores).

Quoted 8 lines

Hypercall is consistently 3-6% above steal-time in the overcommit cases. A possible reason is that the hypercall hands the entire target set to the host in one call, while steal-time still IPIs the running vCPUs and only defers the preempted ones.

On our 8-core machine the collapse is more severe than on your 3C5000 (~5% vs 11% of host at 2 VMs), likely due to the smaller core count and single-NUMA topology.

yeap, with more CPUs the benefit is more high. I think that this kind of patch had better be tested on server platform if there is such hardware by hand after all KVM is used in server platform in most time. I will test the patch in later.

Regards
Bibo Mao

Quoted 49 lines

Thanks,
Tao

Quoted 45 lines

Regards
Bibo Mao

Quoted 41 lines

I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have CONFIG_PARAVIRT enabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.

I'll share the verified numbers once I have them.

Thanks,
Tao

Quoted 25 lines
Quoted 24 lines

Regards
Bibo Mao

Quoted 21 lines

tlb_bench sleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
    no-PV       ~166,536
    steal-time  ~149,553
    hypercall    ~88,686

ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the tlb_bench figure above.

So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.

Best,
Tao

author
#16 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

Quoted 98 lines · Bibo Mao
Quoted 95 lines · Tao Cui
Quoted 67 lines · Bibo Mao
Quoted 64 lines · Tao Cui
Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui

Hi Bibo,

Quoted 7 lines · Bibo Mao
Quoted 5 lines

Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.

well, that is the best. It is my pleasure to discuss this with you.

A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.

The two workloads:
    - ebizzy (all threads busy, mm-flush heavy)
    - tlb_bench in sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)

Quoted 5 lines

ebizzy (records/s, higher is better), 32 vCPUs:
     no-PV       ~103,737
     hypercall   ~105,779
     steal-time  ~105,872
     -> all within noise (±2%); no measurable difference.

Hi Tao,

what is ebizzy command? ebizzy -m or ebizzy -M.

could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?

Quoted 8 lines
Quoted 6 lines

Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
                   ./ebizzy -m          ./ebizzy -M
host             8633                 158898
VM(32 vCPUs)     6610                 133153
VM/host          76%                  83%

just ./ebizzy -M is enough, it seems that CPU number is one key factor.

Sorry for the delay — it turned out my ebizzy command was wrong. I had been running ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.

I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).

                   ebizzy -m        ebizzy -M
host             ~20,000          ~55,000
VM (8 vCPU,1:1)  ~17,000          ~53,000
VM/host          ~86%             ~97%

Quoted 8 lines
Quoted 1 line

The -m ratio (86%) is close to your 76%.

On my 3C5000 Dual-way machine, VM has the same CPU/memory topology with physical machine, the kernel is mainline without any patch. ./ebizzy -M
Host (32 pCPUs)           158898
One VM(32 vCPUs)          133153                 83% of host
Two VMs(32 vCPUs each)    9083 + 9630 = 18713    11% of host

It seems that with ebizzy benchmark, there is big difference if vCPU is preempted. Even if vCPU is not preempted, the performance is only 83% of host on my 3C5000 Dual-way machine.

After fixing the ebizzy command, I
have multi-VM overcommit results for both approaches.

Setup: 8-core LoongArch (single-socket, single NUMA), KVM, linux-next-20260623, 8 vCPU per VM. All VMs run ebizzy -M simultaneously, 3 runs each. The PV-off baseline uses a guest kernel with CONFIG_PARAVIRT=y but without the PV TLB flush patches, so PV IPI and steal-time are active in all three columns.

ebizzy -M, total records/s across all VMs:

               PV-off      steal-time   hypercall
   1:1 (1VM)   53,600      53,800       53,900
   2:1 (2VM)    2,600      42,600       45,300
   3:1 (3VM)    2,800      44,700       46,000

Quoted 4 lines
Quoted 1 line

At 1:1 there is no difference — no vCPU gets preempted. Under

From the previous result, the score on host is 55,000, with 1:1 mode the virtualization efficiency is 97%, it is hard to improve actually. On my 3C5000 Dual-way machine, the virtualization efficiency is 83%, I think it can improve even with 1:1 mode.

Also the paper at https://dl.acm.org/doi/pdf/10.1145/2892242.2892245 proves it with 1:1 mode.

Thanks for the reference. That makes sense — the 1:1 overhead is larger on multi-socket systems, so there may be room to improve even without overcommit.

Quoted 11 lines
Quoted 11 lines

overcommit, without PV TLB flush the throughput drops to ~3-5% of the single-VM case, because every remote TLB flush sends IPIs to preempted vCPUs. With either PV TLB flush variant, preempted vCPUs are skipped, and total throughput stays at ~85-90% of single-VM (bounded by physical cores).

Hypercall is consistently 3-6% above steal-time in the overcommit cases. A possible reason is that the hypercall hands the entire target set to the host in one call, while steal-time still IPIs the running vCPUs and only defers the preempted ones.

Quoted 5 lines
Quoted 3 lines

On our 8-core machine the collapse is more severe than on your 3C5000 (~5% vs 11% of host at 2 VMs), likely due to the smaller core count and single-NUMA topology.

yeap, with more CPUs the benefit is more high. I think that this kind of patch had better be tested on server platform if there is such hardware by hand after all KVM is used in server platform in most time. I will test the patch in later.

I'll try to get access to a server-class LoongArch machine for further testing. If you're able to run the patches on the 3C5000, that would be great too.

Quoted 52 lines

Regards
Bibo Mao

Quoted 49 lines

Thanks,
Tao

Quoted 45 lines

Regards
Bibo Mao

Quoted 41 lines

I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have CONFIG_PARAVIRT enabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.

I'll share the verified numbers once I have them.

Thanks,
Tao

Quoted 25 lines
Quoted 24 lines

Regards
Bibo Mao

Quoted 21 lines

tlb_bench sleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
     no-PV       ~166,536
     steal-time  ~149,553
     hypercall    ~88,686

ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the tlb_bench figure above.

So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.

Best,
Tao

maintainer
#17 lore raw

Re: [PATCH v4 2/3] LoongArch: KVM: Implement guest-side PV TLB flush

Quoted 124 lines · Tao Cui
Quoted 98 lines · Bibo Mao
Quoted 95 lines · Tao Cui
Quoted 67 lines · Bibo Mao
Quoted 64 lines · Tao Cui
Quoted 39 lines · Bibo Mao
Quoted 36 lines · Bibo Mao
Quoted 22 lines · Tao Cui

Hi Bibo,

Quoted 7 lines · Bibo Mao
Quoted 5 lines

Rather than argue from intuition, I'd like to try the hypercall approach you suggested and measure the performance improvement against the current path. I'll share the results with you once the testing is done, so we can decide the direction based on the numbers.

well, that is the best. It is my pleasure to discuss this with you.

A quick update on the testing. I put both the hypercall and the steal-time variants through two benchmarks on an 8-core host with a 4:1 overcommitted guest (32 vCPUs), and wanted to share where things stand.

The two workloads:
    - ebizzy (all threads busy, mm-flush heavy)
    - tlb_bench in sleep-idle mode (1 flusher + 31 sleeping idle threads, so the idle vCPUs get preempted)

Quoted 5 lines

ebizzy (records/s, higher is better), 32 vCPUs:
     no-PV       ~103,737
     hypercall   ~105,779
     steal-time  ~105,872
     -> all within noise (±2%); no measurable difference.

Hi Tao,

what is ebizzy command? ebizzy -m or ebizzy -M.

could you try command on host and one VM without over-committed at first, and then two VMs and three VMs?

Quoted 8 lines
Quoted 6 lines

Here is result on my 3C5000 Dual-way machines with 32 cores and two numa nodes:
                   ./ebizzy -m          ./ebizzy -M
host             8633                 158898
VM(32 vCPUs)     6610                 133153
VM/host          76%                  83%

just ./ebizzy -M is enough, it seems that CPU number is one key factor.

Sorry for the delay — it turned out my ebizzy command was wrong. I had been running ebizzy -t <vcpus> -S 10, which is neither -m nor -M, so neither mmap mode was active and the workload wasn't really stressing the TLB-flush path. Thanks for catching it.

I re-ran with -m and -M on host and a single VM (8-core LoongArch KVM host, 8 vCPU guest, 1:1, no overcommit).

                   ebizzy -m        ebizzy -M
host             ~20,000          ~55,000
VM (8 vCPU,1:1)  ~17,000          ~53,000
VM/host          ~86%             ~97%

Quoted 8 lines
Quoted 1 line

The -m ratio (86%) is close to your 76%.

On my 3C5000 Dual-way machine, VM has the same CPU/memory topology with physical machine, the kernel is mainline without any patch.
                           ./ebizzy -M
Host (32 pCPUs)           158898
One VM(32 vCPUs)          133153                 83% of host
Two VMs(32 vCPUs each)    9083 + 9630 = 18713    11% of host

It seems that with ebizzy benchmark, there is big difference if vCPU is preempted. Even if vCPU is not preempted, the performance is only 83% of host on my 3C5000 Dual-way machine.

After fixing the ebizzy command, I
have multi-VM overcommit results for both approaches.

Setup: 8-core LoongArch (single-socket, single NUMA), KVM, linux-next-20260623, 8 vCPU per VM. All VMs run ebizzy -M simultaneously, 3 runs each. The PV-off baseline uses a guest kernel with CONFIG_PARAVIRT=y but without the PV TLB flush patches, so PV IPI and steal-time are active in all three columns.

ebizzy -M, total records/s across all VMs:

               PV-off      steal-time   hypercall 1:1 (1VM)   53,600      53,800       53,900 2:1 (2VM)    2,600      42,600       45,300 3:1 (3VM)    2,800      44,700       46,000

Quoted 4 lines
Quoted 1 line

At 1:1 there is no difference — no vCPU gets preempted. Under

From the previous result, the score on host is 55,000, with 1:1 mode the virtualization efficiency is 97%, it is hard to improve actually. On my 3C5000 Dual-way machine, the virtualization efficiency is 83%, I think it can improve even with 1:1 mode.

Also the paper at https://dl.acm.org/doi/pdf/10.1145/2892242.2892245 proves it with 1:1 mode.

Thanks for the reference. That makes sense — the 1:1 overhead is larger on multi-socket systems, so there may be room to improve even without overcommit.

Quoted 14 lines
Quoted 14 lines

overcommit, without PV TLB flush the throughput drops to ~3-5% of the single-VM case, because every remote TLB flush sends IPIs to preempted vCPUs. With either PV TLB flush variant, preempted vCPUs are skipped, and total throughput stays at ~85-90% of single-VM (bounded by physical cores).

Hypercall is consistently 3-6% above steal-time in the overcommit cases. A possible reason is that the hypercall hands the entire target set to the host in one call, while steal-time still IPIs the running vCPUs and only defers the preempted ones.

On our 8-core machine the collapse is more severe than on your 3C5000 (~5% vs 11% of host at 2 VMs), likely due to the smaller core count and single-NUMA topology.

Quoted 6 lines
Quoted 2 lines

yeap, with more CPUs the benefit is more high. I think that this kind of patch had better be tested on server platform if there is such hardware by hand after all KVM is used in server platform in most time. I will test the patch in later.

I'll try to get access to a server-class LoongArch machine for further testing. If you're able to run the patches on the 3C5000, that would be great too.

Sure, I will verify it on Server board. The flow may be slow since there
are still many pending issues such as:
1. which TLB flush APIs should be hooked
2. On KVM hypervisor side, should fine-grained TLB flush be supported?

     all TLBs or TLBs from GVA->GPA, or asid based TLBs flush

3. Or at first use a basic TLB flush method, and then optimize in later.

All these require that there are the same test bed and test cases, then we discuss it based the real performance data.

Regards
Bibo Mao

Quoted 53 lines
Quoted 53 lines

Regards
Bibo Mao

Quoted 49 lines

Thanks,
Tao

Quoted 45 lines

Regards
Bibo Mao

Quoted 41 lines

I then tried multi-VM overcommit (2 and 3 VMs, all running ebizzy -M simultaneously). The initial result showed a large gap between the PV-TLB-flush kernel and the baseline under overcommit. Both kernels have CONFIG_PARAVIRT enabled (PV IPI and steal-time are active in both), so PV TLB flush should be the main differentiator — but since they are two separate kernel images rather than a clean on/off toggle, there may be some noise from other differences. I'm now re-running with a QEMU CPU property (kvm-pv-tlb-flush on/off) on the same kernel to isolate the effect cleanly.

I'll share the verified numbers once I have them.

Thanks,
Tao

Quoted 25 lines
Quoted 24 lines

Regards
Bibo Mao

Quoted 21 lines

tlb_bench sleep-idle (ns/flush, lower is better), 1 flusher + 31 idle:
     no-PV       ~166,536
     steal-time  ~149,553
     hypercall    ~88,686

ebizzy's workload is mostly threads staying busy with alloc/copy/free, which drives remote TLB flushes against running vCPUs — that may not be the path this feature is meant to optimize, so the flat result there probably says more about the workload mismatch than about the feature itself. I need to take another look at whether the benchmark actually exercises the cases PV TLB flush targets before reading too much into the numbers, including the tlb_bench figure above.

So I'd hold off on any conclusion for now. Next I'll re-examine the test setup / pick a workload that better matches the feature, and keep you posted once I have something more representative.

Best,
Tao

maintainer
#24 lore raw
In reply to Tao Cui
Quoted 10 lines · Tao Cui

From: Tao Cui <[email protected]>

This series implements paravirtualized TLB flush for LoongArch KVM guests.

In multi-vCPU KVM guests, remote TLB flushes broadcast IPIs to all target vCPUs, including those preempted by the host. Sending IPIs to preempted vCPUs causes unnecessary VM exits and grows with vCPU count, becoming severe in overcommitted deployments.

Quoted 7 lines

Reuse the existing steal-time shared memory page by adding a new KVM_VCPU_FLUSH_TLB flag to the preempted byte. On the guest side, skip IPIs to preempted vCPUs and set the flag via cmpxchg instead. On the host side, when re-entering a vCPU in kvm_update_stolen_time(), check and clear the flag; if set, drop the VPID to trigger a full TLB flush on the next VM entry. No new shared memory page, hypercall, or Kconfig option is needed.

After a second thought, the PV TLB flush can only work on hardware PTW, there is problem on software PTW.

With TLB flush flow, the current CPU sends IPI interrupter to other CPUs to flush TLB. With software PTW, other CPUs may execute TLB refill or TLB exception with interrupt disabled. IPI interrupt will not trigger until TLB refill/exception finishes, however with PV TLB method, other vCPUs will trap to host and flush TLBs, and continue to execute TLB refill/exception, stale TLB may be added.

Regards
Bibo Mao

Quoted 96 lines

The feature is advertised through the existing KVM PV feature
negotiation: the kernel exposes KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH as a VM-level capability, and userspace (QEMU) sets KVM_FEATURE_PV_TLB_FLUSH in the guest's CPUCFG_KVM_FEATURE mask after probing it. A corresponding QEMU patch ("target/loongarch: Enable PV TLB flush advertisement to the guest") is posted alongside this series.

- Host side: only trace a PV TLB flush request when one is observed.
- (Carried over) Host uses amand_db.w to atomically read and clear the preempted byte; selftest gained input validation and failure cleanup.

Testing: the PV TLB flush path itself is unchanged from v3, so the benchmark numbers below still hold. Note that, unlike v3, the feature must now be enabled by userspace: run a QEMU built with the companion patch (or with -cpu kvm-pv-tlb-flush=on) so the guest actually observes KVM_FEATURE_PV_TLB_FLUSH. Boot a 32-vCPU guest and run the selftest inside it with sleep-idle (PV helps) and busy-spin (PV cannot optimize) modes respectively:

   qemu-system-loongarch64 \
     -m 4G -smp 32 --cpu la464 --machine virt \
     -bios .../QEMU_EFI.fd \
     -kernel .../vmlinuz-...-pvtlb-v4+ \
     -initrd /tmp/ramdisk_test.gz \
     -serial mon:stdio \
     -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
     -device virtio-net-pci,netdev=net0 \
     -append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" -nographic

# PV TLB flush enabled (idle threads sleep, vCPUs get preempted) guest# ./pv_tlb_flush_test 1 31 50000 0

# Baseline (idle threads busy-spin, all vCPUs stay active) guest# ./pv_tlb_flush_test 1 31 50000 1

With PV TLB flush (sleep idle): ~152,285 ns/flush
Without PV TLB flush (busy-spin): ~481,045 ns/flush

Improvement: ~68% latency reduction (~3.2x throughput increase)

Tao Cui (3):
LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
LoongArch: KVM: Implement guest-side PV TLB flush
KVM: selftests: loongarch: Add PV TLB flush performance test

  arch/loongarch/include/asm/kvm_host.h              |   1 +
  arch/loongarch/include/asm/kvm_para.h              |   9 +
  arch/loongarch/include/asm/paravirt.h              |  21 +++
  arch/loongarch/include/uapi/asm/kvm.h              |   1 +
  arch/loongarch/include/uapi/asm/kvm_para.h         |   1 +
  arch/loongarch/kernel/paravirt.c                   |  60 ++++++
  arch/loongarch/kernel/smp.c                        |  30 +++-
  arch/loongarch/kvm/trace.h                         |  15 ++
  arch/loongarch/kvm/vcpu.c                          |  34 +++-
  arch/loongarch/kvm/vm.c                            |   3 +
  .../selftests/kvm/loongarch/pv_tlb_flush_test.c    | 194 +++++++++++++++++++++
  11 files changed, 362 insertions(+), 7 deletions(-)

---
Changes in v4:
- Drop the "Preserve auto-enabled PV features on userspace override"
patch: forcing auto-enabled features back on is migration-unsafe, and for PV TLB flush (which cannot degrade gracefully) a missed flush would corrupt the guest. Enablement now follows the usual KVM model -- the kernel advertises KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH and userspace (QEMU) explicitly sets KVM_FEATURE_PV_TLB_FLUSH after probing it; a companion QEMU patch is posted alongside, and the feature can be kept off for an older destination
(-cpu kvm-pv-tlb-flush=off).

Changes in v3:
- Host side: replace amswap_db.w with amand_db.w to atomically read and clear only the preempted byte, preserving the pad bytes for future UAPI use. Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data.
- Host side: move the pv_auto_features OR operation before the compatibility check in kvm_loongarch_cpucfg_set_attr() so that userspace does not need updating for pure kernel-internal PV feature additions.
- Selftest: add input validation, error checking on pthread_create, and cleanup handling on failure.

Changes in v2:
- Host side: replace non-atomic unsafe_get_user + unsafe_put_user with atomic amswap_db.w inline assembly. This fixes two issues:
1) unsafe_put_user failure could skip the TLB flush entirely
2) non-atomic read+write race with guest-side try_cmpxchg could cause FLUSH_TLB requests to be lost
- Guest side: consolidate two separate READ_ONCE calls into a single READ_ONCE to eliminate a TOCTOU race where the host could clear preempted between the two reads. Also switch from byte-sized try_cmpxchg to 32-bit try_cmpxchg on the aligned word containing preempted.

author
#25 lore raw
Quoted 20 lines · Bibo Mao
Quoted 10 lines · Tao Cui

From: Tao Cui <[email protected]>

This series implements paravirtualized TLB flush for LoongArch KVM guests.

In multi-vCPU KVM guests, remote TLB flushes broadcast IPIs to all target vCPUs, including those preempted by the host. Sending IPIs to preempted vCPUs causes unnecessary VM exits and grows with vCPU count, becoming severe in overcommitted deployments.

Quoted 7 lines

Reuse the existing steal-time shared memory page by adding a new KVM_VCPU_FLUSH_TLB flag to the preempted byte. On the guest side, skip IPIs to preempted vCPUs and set the flag via cmpxchg instead. On the host side, when re-entering a vCPU in kvm_update_stolen_time(), check and clear the flag; if set, drop the VPID to trigger a full TLB flush on the next VM entry. No new shared memory page, hypercall, or Kconfig option is needed.

Quoted 4 lines

After a second thought, the PV TLB flush can only work on hardware PTW, there is problem on software PTW.

With TLB flush flow, the current CPU sends IPI interrupter to other CPUs to flush TLB. With software PTW, other CPUs may execute TLB refill or TLB exception with interrupt disabled. IPI interrupt will not trigger until TLB refill/exception finishes, however with PV TLB method, other vCPUs will trap to host and flush TLBs, and continue to execute TLB refill/exception, stale TLB may be added.

Sorry for that.

This is indeed something I had overlooked -- thank you for catching it. It is a correctness issue, not just a performance one, so it has to gate the feature.

I understand the race exactly as you describe it. With software PTW, a TLB miss runs the guest's refill handler with interrupts disabled, and that handler may already have loaded the old PTE into a register. A native remote flush sends an IPI, but the IPI is held until IE is re-enabled, so the flush always lands *after* the refill completes -- any stale entry the refill wrote gets purged. The PV path breaks this: the host flushes the target's TLB (drops the VPID) at an arbitrary instant, then the target resumes its disabled refill handler and writes the stale entry back with tlbfill/tlbwr, and nothing flushes it afterwards. The same applies to the steal-time variant if a vCPU is preempted mid-refill:
it is flushed on re-entry and then resumes the refill. Only a hardware PTW is safe, because the refill is atomic and always reads the current table.

So the feature must only be offered when the guest actually uses a hardware PTW. The real condition is on the *guest* -- a -cpu la464 guest still does software refill even on a PTW host -- so the host gate alone is not enough; a non-PTW guest must not observe the feature either.

For the host side, cpu_has_ptw is the natural precondition, and there's already a precedent in kvm_vm_init_features() where KVM_LOONGARCH_VM_FEAT_PTW is gated on cpu_has_ptw. I'd wrap the advertisement the same way:

    if (cpu_has_ptw) {
        kvm->arch.pv_features  |= BIT(KVM_FEATURE_PV_TLB_FLUSH);
        kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH);
    }

For the guest side: the guest discovers the feature by reading the CPUCFG_KVM_FEATURE leaf, so if feasible I'd mask KVM_FEATURE_PV_TLB_FLUSH off right there -- in kvm_emu_cpucfg() (the CPUCFG_KVM_FEATURE case in exit.c) and the matching one_reg getter kvm_loongarch_cpucfg_get_attr() in vcpu.c -- whenever the guest's own CPUCFG2.PTW (vcpu->arch.cpucfg[2], bit 24) is clear. The equivalent could also be done in QEMU alongside the companion patch, since QEMU already controls the guest model's PTW bit. Both look implementable; I'll need to actually test them to see how they behave.

On the hardware side, I checked the box I ran the benchmarks on. It reports as a 3A6000 / LA664, but reading CPUCFG2 directly gives 0x7e7cccc7 with the PTW bit (24) clear, so it has no hardware PTW either -- likely an early part. Your 3C5000 is LA464, which is software PTW too. That means the ebizzy and tlb_bench numbers I shared earlier were collected with PV TLB flush active on a -cpu la464 (software-refill) guest, i.e. exactly the unsafe configuration you pointed out. They show the mechanism works and its direction (hypercall ahead of steal-time, and PV preventing the overcommit collapse), but they were measured in a correctness-broken config, so I won't use them to justify enabling the feature on software-PTW silicon.

For now I don't have a PTW-capable machine to test on. My reading is that a software-PTW-safe variant isn't really possible -- the native IPI is safe precisely because the flush is deferred past the IE=0 refill window, and a host-side flush can't reproduce that ordering without effectively becoming an IPI again. I'll spend some time confirming that, but unless I've missed something, gating to hardware-PTW guests is the answer.

Thanks,
Tao

Quoted 99 lines

Regards
Bibo Mao

Quoted 96 lines

The feature is advertised through the existing KVM PV feature
negotiation: the kernel exposes KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH as a VM-level capability, and userspace (QEMU) sets KVM_FEATURE_PV_TLB_FLUSH in the guest's CPUCFG_KVM_FEATURE mask after probing it. A corresponding QEMU patch ("target/loongarch: Enable PV TLB flush advertisement to the guest") is posted alongside this series.

- Host side: only trace a PV TLB flush request when one is observed.
- (Carried over) Host uses amand_db.w to atomically read and clear the preempted byte; selftest gained input validation and failure cleanup.

Testing: the PV TLB flush path itself is unchanged from v3, so the benchmark numbers below still hold.  Note that, unlike v3, the feature must now be enabled by userspace: run a QEMU built with the companion patch (or with -cpu kvm-pv-tlb-flush=on) so the guest actually observes KVM_FEATURE_PV_TLB_FLUSH.  Boot a 32-vCPU guest and run the selftest inside it with sleep-idle (PV helps) and busy-spin (PV cannot optimize) modes respectively:

   qemu-system-loongarch64 \
     -m 4G -smp 32 --cpu la464 --machine virt \
     -bios .../QEMU_EFI.fd \
     -kernel .../vmlinuz-...-pvtlb-v4+ \
     -initrd /tmp/ramdisk_test.gz \
     -serial mon:stdio \
     -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
     -device virtio-net-pci,netdev=net0 \
     -append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" -nographic

   # PV TLB flush enabled (idle threads sleep, vCPUs get preempted) guest# ./pv_tlb_flush_test 1 31 50000 0

   # Baseline (idle threads busy-spin, all vCPUs stay active) guest# ./pv_tlb_flush_test 1 31 50000 1

   With PV TLB flush (sleep idle):    ~152,285 ns/flush Without PV TLB flush (busy-spin):  ~481,045 ns/flush

   Improvement: ~68% latency reduction (~3.2x throughput increase)

Tao Cui (3):
   LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
   LoongArch: KVM: Implement guest-side PV TLB flush
   KVM: selftests: loongarch: Add PV TLB flush performance test

  arch/loongarch/include/asm/kvm_host.h              |   1 +
  arch/loongarch/include/asm/kvm_para.h              |   9 +
  arch/loongarch/include/asm/paravirt.h              |  21 +++
  arch/loongarch/include/uapi/asm/kvm.h              |   1 +

  arch/loongarch/include/uapi/asm/kvm_para.h         |   1 + arch/loongarch/kernel/paravirt.c                   |  60 ++++++ arch/loongarch/kernel/smp.c                        |  30 +++- arch/loongarch/kvm/trace.h                         |  15 ++ arch/loongarch/kvm/vcpu.c                          |  34 +++- arch/loongarch/kvm/vm.c                            |   3 + .../selftests/kvm/loongarch/pv_tlb_flush_test.c    | 194 +++++++++++++++++++++

  11 files changed, 362 insertions(+), 7 deletions(-)

---
Changes in v4:
- Drop the "Preserve auto-enabled PV features on userspace override"
   patch: forcing auto-enabled features back on is migration-unsafe, and for PV TLB flush (which cannot degrade gracefully) a missed flush would corrupt the guest.  Enablement now follows the usual KVM model -- the kernel advertises KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH and userspace (QEMU) explicitly sets KVM_FEATURE_PV_TLB_FLUSH after probing it; a companion QEMU patch is posted alongside, and the feature can be kept off for an older destination
   (-cpu kvm-pv-tlb-flush=off).

Changes in v3:
- Host side: replace amswap_db.w with amand_db.w to atomically read and clear only the preempted byte, preserving the pad bytes for future UAPI use.  Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data.
- Host side: move the pv_auto_features OR operation before the compatibility check in kvm_loongarch_cpucfg_set_attr() so that userspace does not need updating for pure kernel-internal PV feature additions.
- Selftest: add input validation, error checking on pthread_create, and cleanup handling on failure.

Changes in v2:
- Host side: replace non-atomic unsafe_get_user + unsafe_put_user with atomic amswap_db.w inline assembly. This fixes two issues:
   1) unsafe_put_user failure could skip the TLB flush entirely
   2) non-atomic read+write race with guest-side try_cmpxchg could cause FLUSH_TLB requests to be lost
- Guest side: consolidate two separate READ_ONCE calls into a single READ_ONCE to eliminate a TOCTOU race where the host could clear preempted between the two reads. Also switch from byte-sized try_cmpxchg to 32-bit try_cmpxchg on the aligned word containing preempted.

maintainer
#26 lore raw
Quoted 85 lines · Tao Cui
Quoted 20 lines · Bibo Mao
Quoted 10 lines · Tao Cui

From: Tao Cui <[email protected]>

This series implements paravirtualized TLB flush for LoongArch KVM guests.

In multi-vCPU KVM guests, remote TLB flushes broadcast IPIs to all target vCPUs, including those preempted by the host. Sending IPIs to preempted vCPUs causes unnecessary VM exits and grows with vCPU count, becoming severe in overcommitted deployments.

Quoted 7 lines

Reuse the existing steal-time shared memory page by adding a new KVM_VCPU_FLUSH_TLB flag to the preempted byte. On the guest side, skip IPIs to preempted vCPUs and set the flag via cmpxchg instead. On the host side, when re-entering a vCPU in kvm_update_stolen_time(), check and clear the flag; if set, drop the VPID to trigger a full TLB flush on the next VM entry. No new shared memory page, hypercall, or Kconfig option is needed.

Quoted 4 lines

After a second thought, the PV TLB flush can only work on hardware PTW, there is problem on software PTW.

With TLB flush flow, the current CPU sends IPI interrupter to other CPUs to flush TLB. With software PTW, other CPUs may execute TLB refill or TLB exception with interrupt disabled. IPI interrupt will not trigger until TLB refill/exception finishes, however with PV TLB method, other vCPUs will trap to host and flush TLBs, and continue to execute TLB refill/exception, stale TLB may be added.

Sorry for that.

This is indeed something I had overlooked -- thank you for catching it. It is a correctness issue, not just a performance one, so it has to gate the feature.

I understand the race exactly as you describe it. With software PTW, a TLB miss runs the guest's refill handler with interrupts disabled, and that handler may already have loaded the old PTE into a register. A native remote flush sends an IPI, but the IPI is held until IE is re-enabled, so the flush always lands *after* the refill completes -- any stale entry the refill wrote gets purged. The PV path breaks this: the host flushes the target's TLB (drops the VPID) at an arbitrary instant, then the target resumes its disabled refill handler and writes the stale entry back with tlbfill/tlbwr, and nothing flushes it afterwards. The same applies to the steal-time variant if a vCPU is preempted mid-refill:
it is flushed on re-entry and then resumes the refill. Only a hardware PTW is safe, because the refill is atomic and always reads the current table.

So the feature must only be offered when the guest actually uses a hardware PTW. The real condition is on the *guest* -- a -cpu la464 guest still does software refill even on a PTW host -- so the host gate alone is not enough; a non-PTW guest must not observe the feature either.

For the host side, cpu_has_ptw is the natural precondition, and there's already a precedent in kvm_vm_init_features() where KVM_LOONGARCH_VM_FEAT_PTW is gated on cpu_has_ptw. I'd wrap the advertisement the same way:

     if (cpu_has_ptw) {
         kvm->arch.pv_features  |= BIT(KVM_FEATURE_PV_TLB_FLUSH);
         kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH);
     }

For the guest side: the guest discovers the feature by reading the CPUCFG_KVM_FEATURE leaf, so if feasible I'd mask KVM_FEATURE_PV_TLB_FLUSH off right there -- in kvm_emu_cpucfg() (the CPUCFG_KVM_FEATURE case in exit.c) and the matching one_reg getter kvm_loongarch_cpucfg_get_attr() in vcpu.c -- whenever the guest's own CPUCFG2.PTW (vcpu->arch.cpucfg[2], bit 24) is clear. The equivalent could also be done in QEMU alongside the companion patch, since QEMU already controls the guest model's PTW bit. Both look implementable; I'll need to actually test them to see how they behave.

On the hardware side, I checked the box I ran the benchmarks on. It reports as a 3A6000 / LA664, but reading CPUCFG2 directly gives 0x7e7cccc7 with the PTW bit (24) clear, so it has no hardware PTW either -- likely an early part. Your 3C5000 is LA464, which is software PTW too. That means the ebizzy and tlb_bench numbers I shared earlier were collected with PV TLB flush active on a -cpu la464 (software-refill) guest, i.e. exactly the unsafe configuration you pointed out. They show the mechanism works and its direction (hypercall ahead of steal-time, and PV preventing the overcommit collapse), but they were measured in a correctness-broken config, so I won't use them to justify enabling the feature on software-PTW silicon.

Quoted 6 lines

For now I don't have a PTW-capable machine to test on. My reading is that a software-PTW-safe variant isn't really possible -- the native IPI is safe precisely because the flush is deferred past the IE=0 refill window, and a host-side flush can't reproduce that ordering without effectively becoming an IPI again. I'll spend some time confirming that, but unless I've missed something, gating to hardware-PTW guests is the

PV TLB flush is a good technology, it can be verified again when there is hardware with HW PTW supported.

Regards
Bibo Mao

Quoted 104 lines

answer.

Thanks,
Tao

Quoted 99 lines

Regards
Bibo Mao

Quoted 96 lines

The feature is advertised through the existing KVM PV feature
negotiation: the kernel exposes KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH as a VM-level capability, and userspace (QEMU) sets KVM_FEATURE_PV_TLB_FLUSH in the guest's CPUCFG_KVM_FEATURE mask after probing it. A corresponding QEMU patch ("target/loongarch: Enable PV TLB flush advertisement to the guest") is posted alongside this series.

- Host side: only trace a PV TLB flush request when one is observed.
- (Carried over) Host uses amand_db.w to atomically read and clear the preempted byte; selftest gained input validation and failure cleanup.

Testing: the PV TLB flush path itself is unchanged from v3, so the benchmark numbers below still hold.  Note that, unlike v3, the feature must now be enabled by userspace: run a QEMU built with the companion patch (or with -cpu kvm-pv-tlb-flush=on) so the guest actually observes KVM_FEATURE_PV_TLB_FLUSH.  Boot a 32-vCPU guest and run the selftest inside it with sleep-idle (PV helps) and busy-spin (PV cannot optimize) modes respectively:

   qemu-system-loongarch64 \
     -m 4G -smp 32 --cpu la464 --machine virt \
     -bios .../QEMU_EFI.fd \
     -kernel .../vmlinuz-...-pvtlb-v4+ \
     -initrd /tmp/ramdisk_test.gz \
     -serial mon:stdio \
     -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
     -device virtio-net-pci,netdev=net0 \
     -append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" -nographic

   # PV TLB flush enabled (idle threads sleep, vCPUs get preempted) guest# ./pv_tlb_flush_test 1 31 50000 0

   # Baseline (idle threads busy-spin, all vCPUs stay active) guest# ./pv_tlb_flush_test 1 31 50000 1

   With PV TLB flush (sleep idle):    ~152,285 ns/flush Without PV TLB flush (busy-spin):  ~481,045 ns/flush

   Improvement: ~68% latency reduction (~3.2x throughput increase)

Tao Cui (3):
   LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
   LoongArch: KVM: Implement guest-side PV TLB flush
   KVM: selftests: loongarch: Add PV TLB flush performance test

   arch/loongarch/include/asm/kvm_host.h              |   1 +
   arch/loongarch/include/asm/kvm_para.h              |   9 +
   arch/loongarch/include/asm/paravirt.h              |  21 +++
   arch/loongarch/include/uapi/asm/kvm.h              |   1 +
   arch/loongarch/include/uapi/asm/kvm_para.h         |   1 +
   arch/loongarch/kernel/paravirt.c                   |  60 ++++++
   arch/loongarch/kernel/smp.c                        |  30 +++-
   arch/loongarch/kvm/trace.h                         |  15 ++
   arch/loongarch/kvm/vcpu.c                          |  34 +++-
   arch/loongarch/kvm/vm.c                            |   3 +
   .../selftests/kvm/loongarch/pv_tlb_flush_test.c    | 194 +++++++++++++++++++++
   11 files changed, 362 insertions(+), 7 deletions(-)

---
Changes in v4:
- Drop the "Preserve auto-enabled PV features on userspace override"
   patch: forcing auto-enabled features back on is migration-unsafe, and for PV TLB flush (which cannot degrade gracefully) a missed flush would corrupt the guest.  Enablement now follows the usual KVM model -- the kernel advertises KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH and userspace (QEMU) explicitly sets KVM_FEATURE_PV_TLB_FLUSH after probing it; a companion QEMU patch is posted alongside, and the feature can be kept off for an older destination
   (-cpu kvm-pv-tlb-flush=off).

Changes in v3:
- Host side: replace amswap_db.w with amand_db.w to atomically read and clear only the preempted byte, preserving the pad bytes for future UAPI use.  Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data.
- Host side: move the pv_auto_features OR operation before the compatibility check in kvm_loongarch_cpucfg_set_attr() so that userspace does not need updating for pure kernel-internal PV feature additions.
- Selftest: add input validation, error checking on pthread_create, and cleanup handling on failure.

Changes in v2:
- Host side: replace non-atomic unsafe_get_user + unsafe_put_user with atomic amswap_db.w inline assembly. This fixes two issues:
   1) unsafe_put_user failure could skip the TLB flush entirely
   2) non-atomic read+write race with guest-side try_cmpxchg could cause FLUSH_TLB requests to be lost
- Guest side: consolidate two separate READ_ONCE calls into a single READ_ONCE to eliminate a TOCTOU race where the host could clear preempted between the two reads. Also switch from byte-sized try_cmpxchg to 32-bit try_cmpxchg on the aligned word containing preempted.

author
#27 lore raw
Quoted 88 lines · Bibo Mao
Quoted 85 lines · Tao Cui
Quoted 20 lines · Bibo Mao
Quoted 10 lines · Tao Cui

From: Tao Cui <[email protected]>

This series implements paravirtualized TLB flush for LoongArch KVM guests.

In multi-vCPU KVM guests, remote TLB flushes broadcast IPIs to all target vCPUs, including those preempted by the host. Sending IPIs to preempted vCPUs causes unnecessary VM exits and grows with vCPU count, becoming severe in overcommitted deployments.

Quoted 7 lines

Reuse the existing steal-time shared memory page by adding a new KVM_VCPU_FLUSH_TLB flag to the preempted byte. On the guest side, skip IPIs to preempted vCPUs and set the flag via cmpxchg instead. On the host side, when re-entering a vCPU in kvm_update_stolen_time(), check and clear the flag; if set, drop the VPID to trigger a full TLB flush on the next VM entry. No new shared memory page, hypercall, or Kconfig option is needed.

Quoted 4 lines

After a second thought, the PV TLB flush can only work on hardware PTW, there is problem on software PTW.

With TLB flush flow, the current CPU sends IPI interrupter to other CPUs to flush TLB. With software PTW, other CPUs may execute TLB refill or TLB exception with interrupt disabled. IPI interrupt will not trigger until TLB refill/exception finishes, however with PV TLB method, other vCPUs will trap to host and flush TLBs, and continue to execute TLB refill/exception, stale TLB may be added.

Sorry for that.

This is indeed something I had overlooked -- thank you for catching it. It is a correctness issue, not just a performance one, so it has to gate the feature.

I understand the race exactly as you describe it. With software PTW, a TLB miss runs the guest's refill handler with interrupts disabled, and that handler may already have loaded the old PTE into a register. A native remote flush sends an IPI, but the IPI is held until IE is re-enabled, so the flush always lands *after* the refill completes -- any stale entry the refill wrote gets purged. The PV path breaks this: the host flushes the target's TLB (drops the VPID) at an arbitrary instant, then the target resumes its disabled refill handler and writes the stale entry back with tlbfill/tlbwr, and nothing flushes it afterwards. The same applies to the steal-time variant if a vCPU is preempted mid-refill:
it is flushed on re-entry and then resumes the refill. Only a hardware PTW is safe, because the refill is atomic and always reads the current table.

So the feature must only be offered when the guest actually uses a hardware PTW. The real condition is on the *guest* -- a -cpu la464 guest still does software refill even on a PTW host -- so the host gate alone is not enough; a non-PTW guest must not observe the feature either.

For the host side, cpu_has_ptw is the natural precondition, and there's already a precedent in kvm_vm_init_features() where KVM_LOONGARCH_VM_FEAT_PTW is gated on cpu_has_ptw. I'd wrap the advertisement the same way:

     if (cpu_has_ptw) {
         kvm->arch.pv_features  |= BIT(KVM_FEATURE_PV_TLB_FLUSH); kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH); }

For the guest side: the guest discovers the feature by reading the CPUCFG_KVM_FEATURE leaf, so if feasible I'd mask KVM_FEATURE_PV_TLB_FLUSH off right there -- in kvm_emu_cpucfg() (the CPUCFG_KVM_FEATURE case in exit.c) and the matching one_reg getter kvm_loongarch_cpucfg_get_attr() in vcpu.c -- whenever the guest's own CPUCFG2.PTW (vcpu->arch.cpucfg[2], bit 24) is clear. The equivalent could also be done in QEMU alongside the companion patch, since QEMU already controls the guest model's PTW bit. Both look implementable; I'll need to actually test them to see how they behave.

On the hardware side, I checked the box I ran the benchmarks on. It reports as a 3A6000 / LA664, but reading CPUCFG2 directly gives 0x7e7cccc7 with the PTW bit (24) clear, so it has no hardware PTW either -- likely an early part. Your 3C5000 is LA464, which is software PTW too. That means the ebizzy and tlb_bench numbers I shared earlier were collected with PV TLB flush active on a -cpu la464 (software-refill) guest, i.e. exactly the unsafe configuration you pointed out. They show the mechanism works and its direction (hypercall ahead of steal-time, and PV preventing the overcommit collapse), but they were measured in a correctness-broken config, so I won't use them to justify enabling the feature on software-PTW silicon.

Quoted 8 lines
Quoted 6 lines

For now I don't have a PTW-capable machine to test on. My reading is that a software-PTW-safe variant isn't really possible -- the native IPI is safe precisely because the flush is deferred past the IE=0 refill window, and a host-side flush can't reproduce that ordering without effectively becoming an IPI again. I'll spend some time confirming that, but unless I've missed something, gating to hardware-PTW guests is the

PV TLB flush is a good technology, it can be verified again when there is hardware with HW PTW supported.

Thanks a lot for all the help on this. You raised a real correctness issue I'd missed, and I genuinely enjoyed working through it with you -- learned a good bit too. Yeah, makes sense to revisit PV TLB flush once there's HW-PTW hardware to test on.

Looking forward to next time.

Cheers,
Tao

Quoted 107 lines

Regards
Bibo Mao

Quoted 104 lines

answer.

Thanks,
Tao

Quoted 99 lines

Regards
Bibo Mao

Quoted 96 lines

The feature is advertised through the existing KVM PV feature
negotiation: the kernel exposes KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH as a VM-level capability, and userspace (QEMU) sets KVM_FEATURE_PV_TLB_FLUSH in the guest's CPUCFG_KVM_FEATURE mask after probing it. A corresponding QEMU patch ("target/loongarch: Enable PV TLB flush advertisement to the guest") is posted alongside this series.

- Host side: only trace a PV TLB flush request when one is observed.
- (Carried over) Host uses amand_db.w to atomically read and clear the preempted byte; selftest gained input validation and failure cleanup.

Testing: the PV TLB flush path itself is unchanged from v3, so the benchmark numbers below still hold.  Note that, unlike v3, the feature must now be enabled by userspace: run a QEMU built with the companion patch (or with -cpu kvm-pv-tlb-flush=on) so the guest actually observes KVM_FEATURE_PV_TLB_FLUSH.  Boot a 32-vCPU guest and run the selftest inside it with sleep-idle (PV helps) and busy-spin (PV cannot optimize) modes respectively:

    qemu-system-loongarch64 \
      -m 4G -smp 32 --cpu la464 --machine virt \
      -bios .../QEMU_EFI.fd \
      -kernel .../vmlinuz-...-pvtlb-v4+ \
      -initrd /tmp/ramdisk_test.gz \
      -serial mon:stdio \
      -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
      -device virtio-net-pci,netdev=net0 \
      -append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" -nographic

    # PV TLB flush enabled (idle threads sleep, vCPUs get preempted) guest# ./pv_tlb_flush_test 1 31 50000 0

    # Baseline (idle threads busy-spin, all vCPUs stay active) guest# ./pv_tlb_flush_test 1 31 50000 1

    With PV TLB flush (sleep idle):    ~152,285 ns/flush Without PV TLB flush (busy-spin):  ~481,045 ns/flush

    Improvement: ~68% latency reduction (~3.2x throughput increase)

Tao Cui (3):
    LoongArch: KVM: Add PV TLB flush support via steal-time shared memory
    LoongArch: KVM: Implement guest-side PV TLB flush
    KVM: selftests: loongarch: Add PV TLB flush performance test

   arch/loongarch/include/asm/kvm_host.h              |   1 +
   arch/loongarch/include/asm/kvm_para.h              |   9 +
   arch/loongarch/include/asm/paravirt.h              |  21 +++
   arch/loongarch/include/uapi/asm/kvm.h              |   1 +

   arch/loongarch/include/uapi/asm/kvm_para.h         |   1 + arch/loongarch/kernel/paravirt.c                   |  60 ++++++ arch/loongarch/kernel/smp.c                        |  30 +++- arch/loongarch/kvm/trace.h                         |  15 ++ arch/loongarch/kvm/vcpu.c                          |  34 +++- arch/loongarch/kvm/vm.c                            |   3 + .../selftests/kvm/loongarch/pv_tlb_flush_test.c    | 194 +++++++++++++++++++++

   11 files changed, 362 insertions(+), 7 deletions(-)

---
Changes in v4:
- Drop the "Preserve auto-enabled PV features on userspace override"
    patch: forcing auto-enabled features back on is migration-unsafe, and for PV TLB flush (which cannot degrade gracefully) a missed flush would corrupt the guest.  Enablement now follows the usual KVM model -- the kernel advertises KVM_LOONGARCH_VM_FEAT_PV_TLB_FLUSH and userspace (QEMU) explicitly sets KVM_FEATURE_PV_TLB_FLUSH after probing it; a companion QEMU patch is posted alongside, and the feature can be kept off for an older destination
    (-cpu kvm-pv-tlb-flush=off).

Changes in v3:
- Host side: replace amswap_db.w with amand_db.w to atomically read and clear only the preempted byte, preserving the pad bytes for future UAPI use.  Issue a normal load (unsafe_get_user) before the atomic amand_db.w to avoid operating on stale cache data.
- Host side: move the pv_auto_features OR operation before the compatibility check in kvm_loongarch_cpucfg_set_attr() so that userspace does not need updating for pure kernel-internal PV feature additions.
- Selftest: add input validation, error checking on pthread_create, and cleanup handling on failure.

Changes in v2:
- Host side: replace non-atomic unsafe_get_user + unsafe_put_user with atomic amswap_db.w inline assembly. This fixes two issues:
    1) unsafe_put_user failure could skip the TLB flush entirely
    2) non-atomic read+write race with guest-side try_cmpxchg could cause FLUSH_TLB requests to be lost
- Guest side: consolidate two separate READ_ONCE calls into a single READ_ONCE to eliminate a TOCTOU race where the host could clear preempted between the two reads. Also switch from byte-sized try_cmpxchg to 32-bit try_cmpxchg on the aligned word containing preempted.