KVM paravirt_ops backend (v3)

PATCH In review Anthony Liguori 3 participants 10 messages Jun 21, 2007 2 messages not in the archive

Subsystems (3)

KVM virtio core KVM paravirt

lore

Share & tools

lore

Anthony Liguori

author

2007-06-21 03:04 UTC #0 lore raw

cover KVM paravirt_ops backend (v3)

Hi,

This is an update to the paravirt_ops KVM backend series. I've made a number of changes and attempted to incorporate all the feedback from the last review. Some highlights:

1) Clean up the paravirt time source patch to use a more Xen-like model
2) Change the hypercall queueing to pass a PA on the flush hypercall
3) Add MMU support for release_{pt,pd} and TLB flush
4) Use KVM specific errno values
5) Switch from per_cpu to more appropriate functions

As for performance, I've got a few interesting results. kbuild with a guest using 2G of memory goes from 19 minutes to 12 minutes with the full series applied. Using 512mb, the build time goes from 10.75 minutes to 9 minutes. For 512mb, native is around 7 minutes so that's pretty close to what Avi had seen. The more dramatic improvement with large memory guests is probably because of the increased shadow page table activity due to high mem.

virtbench shows major improvements but I'm not 100% confident yet in the results as they are not very stable. I don't yet have a benchmark that shows the benefit of the CR caching so if I don't find one, I'll drop that from the queue.

As usual, the latest bits are available at
http://hg.codemonkey.ws/kvm-paravirt

Regards,

Anthony Liguori

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

A message that is not in the archive (sent off-list or not yet received): [email protected]

Anthony Liguori

author

2007-06-21 03:05 UTC #2 lore raw

PATCH 1/5 KVM paravirt_ops core infrastructure

Regards,

Anthony Liguori

Subject: [PATCH] KVM paravirt_ops core infrastructure
Author: Anthony Liguori <aliguori-r/[email protected]>

This patch implements paravirt_ops support for KVM and updates the current paravirtualization support in KVM to match. Some changes to the previous paravirtualization support in KVM:

1) Theoritical support for SMP guests
2) Use CPUID to discover paravirtualization
3) Use feature bitmap instead of versioning

Signed-off-by Anthony Liguori <aliguori-r/[email protected]>

arch/i386/Kconfig +7 -0

index 8770a5d..97ad1e1 100644
@@ -231,6 +231,13 @@ config VMI
 	  at the moment), by linking the kernel to a GPL-ed ROM module
 	  provided by the hypervisor.
 
+config KVM_GUEST
+	bool "KVM paravirt-ops support"
+	depends on PARAVIRT
+	help
+	  This option enables various optimizations for running under the KVM
+          hypervisor.
+
 config ACPI_SRAT
 	bool
 	default y

arch/i386/kernel/Makefile +1 -0

index 06da59f..12a4201 100644
@@ -42,6 +42,7 @@ obj-$(CONFIG_HPET_TIMER) 	+= hpet.o
 obj-$(CONFIG_K8_NB)		+= k8.o
 
 obj-$(CONFIG_VMI)		+= vmi.o vmiclock.o
+obj-$(CONFIG_KVM_GUEST)		+= kvm.o
 obj-$(CONFIG_PARAVIRT)		+= paravirt.o
 obj-y				+= pcspeaker.o

arch/i386/kernel/kvm.c new file +222 -0

222 lines not on this page · raw message

drivers/kvm/kvm_main.c +39 ~~-30~~

147 lines not on this page · raw message

include/linux/kvm_para.h +14 ~~-50~~

86 lines not on this page · raw message

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Anthony Liguori

author

2007-06-21 03:06 UTC #3 lore raw

PATCH 2/5 KVM: paravirt time source

In reply to a message not in the archive

This helps a lot on modern kernels where Linux is much more finicky about using the TSC as a time source.

Regards,

Anthony Liguori

Subject: [PATCH] KVM: paravirt time source
Author: Anthony Liguori <aliguori-r/[email protected]>

This patch implements a paravirtual time source based on a new field added to the VMCA. The host provides time in nanoseconds to the guest.

Signed-off-by Anthony Liguori <aliguori-r/[email protected]>

arch/i386/kernel/kvm.c +46 -0

index 22ea647..62c3b5b 100644
@@ -26,6 +26,18 @@
 #include <linux/cpu.h>
 #include <linux/mm.h>
 
+#include <linux/clocksource.h>
+#include <linux/workqueue.h>
+#include <linux/cpufreq.h>
+#include <linux/jiffies.h>
+#include <linux/init.h>
+#include <linux/dmi.h>
+#include <linux/acpi_pmtmr.h>
+
+#include "mach_timer.h"
+
+#define KVM_SCALE 22
+
 struct kvm_paravirt_state
 {
 	struct kvm_vmca *vmca;

@@ -38,6 +50,7 @@ struct kvm_paravirt_state
 static DEFINE_PER_CPU(struct kvm_paravirt_state *, paravirt_state);
 
 static int do_nop_io_delay;
+static int do_paravirt_clock;
 static u64 msr_set_vmca;
 
 static long kvm_hypercall(unsigned int nr, unsigned long p1,

@@ -62,6 +75,28 @@ static long kvm_hypercall(unsigned int nr, unsigned long p1,
 	return ret;
 }
 
+static cycle_t kvm_clocksource_read(void)
+{
+	struct kvm_paravirt_state *state = get_cpu_var(paravirt_state);
+	cycle_t ret;
+
+	kvm_hypercall(KVM_HYPERCALL_UPDATE_TIME, 0, 0, 0, 0);
+	ret = state->vmca->real_nsecs;
+	put_cpu_var(paravirt_state);
+
+	return ret;
+}
+
+static struct clocksource clocksource_kvm = {
+	.name			= "kvm",
+	.rating			= 200,
+	.read			= kvm_clocksource_read,
+	.mask			= CLOCKSOURCE_MASK(64),
+	.mult			= 1 << KVM_SCALE,
+	.shift			= KVM_SCALE,
+	.flags			= CLOCK_SOURCE_IS_CONTINUOUS,
+};
+
 /*
  * No need for any "IO delay" on KVM
  */

@@ -76,6 +111,14 @@ static void paravirt_ops_setup(void)
 	if (do_nop_io_delay)
 		paravirt_ops.io_delay = kvm_io_delay;
 
+	if (do_paravirt_clock) {
+		int err;
+
+		err = clocksource_register(&clocksource_kvm);
+		WARN_ON(err);
+		printk(KERN_INFO "KVM: using paravirt clock source\n");
+	}
+
 	paravirt_ops.paravirt_enabled = 1;
 
 	apply_paravirt(__parainstructions, __parainstructions_end);

@@ -114,6 +157,9 @@ static int paravirt_initialize(void)
 	if ((edx & KVM_FEATURE_NOP_IO_DELAY))
 		do_nop_io_delay = 1;
 
+	if ((edx & KVM_FEATURE_PARAVIRT_CLOCK))
+		do_paravirt_clock = 1;
+
 	on_each_cpu(paravirt_activate, NULL, 0, 1);
 
 	return 0;

drivers/kvm/kvm_main.c +27 -1

index 1369310..e455fca 100644
@@ -44,6 +44,7 @@
 #include <linux/cpumask.h>
 #include <linux/smp.h>
 #include <linux/kvm_para.h>
+#include <linux/delay.h>
 
 #include "x86_emulate.h"
 #include "segment_descriptor.h"

@@ -93,7 +94,8 @@ struct vfsmount *kvmfs_mnt;
 #define EFER_RESERVED_BITS 0xfffffffffffff2fe
 
 #define KVM_PARAVIRT_FEATURES \
-	(KVM_FEATURE_VMCA | KVM_FEATURE_NOP_IO_DELAY)
+	(KVM_FEATURE_VMCA | KVM_FEATURE_NOP_IO_DELAY | \
+	 KVM_FEATURE_PARAVIRT_CLOCK)
 
 #define KVM_MSR_SET_VMCA	0x87655678

@@ -1346,10 +1348,34 @@ int kvm_emulate_halt(struct kvm_vcpu *vcpu)
 }
 EXPORT_SYMBOL_GPL(kvm_emulate_halt);
 
+static int kvm_hypercall_update_time(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vmca *vmca;
+	struct timespec now;
+
+	if (unlikely(!vcpu->para_state_page))
+		return -KVM_EINVAL;
+
+	ktime_get_ts(&now);
+
+	vmca = kmap(vcpu->para_state_page);
+
+	vmca->real_nsecs = now.tv_nsec + now.tv_sec * (cycles_t)1e9;
+
+	mark_page_dirty(vcpu->kvm, vcpu->para_state_gpa >> PAGE_SHIFT);
+	kunmap(vcpu->para_state_page);
+
+	return 0;
+}
+
 static int dispatch_hypercall(struct kvm_vcpu *vcpu, unsigned long nr,
 			      unsigned long p1, unsigned long p2,
 			      unsigned long p3, unsigned long p4)
 {
+	switch (nr) {
+	case KVM_HYPERCALL_UPDATE_TIME:
+		return kvm_hypercall_update_time(vcpu);
+	}
 	return -KVM_ENOSYS;
 }

include/linux/kvm_para.h +5 -0

index 11ebad8..4f27fb1 100644
@@ -15,10 +15,12 @@
 
 #define KVM_FEATURE_VMCA		(1UL << 0)
 #define KVM_FEATURE_NOP_IO_DELAY	(1UL << 1)
+#define KVM_FEATURE_PARAVIRT_CLOCK	(1UL << 2)
 
 struct kvm_vmca
 {
 	u64 hypercall_gpa;
+	u64 real_nsecs;
 };
 
 /*

@@ -33,5 +35,8 @@ struct kvm_vmca
  */
 
 #define KVM_ENOSYS	ENOSYS
+#define KVM_EINVAL	EINVAL
+
+#define KVM_HYPERCALL_UPDATE_TIME	0
 
 #endif

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Anthony Liguori

author

2007-06-21 03:07 UTC #4 lore raw

PATCH 3/5 KVM: Implement CR read caching for KVM paravirt_ops

In reply to a message not in the archive

Regards,

Anthony Liguori

Subject: [PATCH] KVM: Implement CR read caching for KVM paravirt_ops
Author: Anthony Liguori <aliguori-r/[email protected]>

With hardware virtualization, CR reads often times require a VMEXIT which is rather expensive. Instead of reading CR and taking the VMEXIT, maintain a copy of each CR and return that on CR reads.

Signed-off-by Anthony Liguori <aliguori-r/[email protected]>

arch/i386/kernel/kvm.c +109 -0

index 62c3b5b..dcc45cd 100644
@@ -38,8 +38,13 @@
 
 #define KVM_SCALE 22
 
+#define CR0_TS_MASK (1ULL << 3)
+
 struct kvm_paravirt_state
 {
+	unsigned long cached_cr[5];
+	int cr_valid[5];
+
 	struct kvm_vmca *vmca;
 	struct kvm_hypercall_entry *queue;
 	void (*hypercall)(void);

@@ -49,6 +54,7 @@ struct kvm_paravirt_state
 
 static DEFINE_PER_CPU(struct kvm_paravirt_state *, paravirt_state);
 
+static int do_cr_read_caching;
 static int do_nop_io_delay;
 static int do_paravirt_clock;
 static u64 msr_set_vmca;

@@ -104,6 +110,93 @@ static void kvm_io_delay(void)
 {
 }
 
+/*
+ * Control register reads can be trapped.  Since trapping is relatively
+ * expensive, we can avoid paying the cost by caching logically.
+ */
+static __always_inline unsigned long kvm_read_cr(int reg)
+{
+	struct kvm_paravirt_state *state = x86_read_percpu(paravirt_state);
+
+	if (unlikely(!state->cr_valid[reg])) {
+		switch (reg) {
+		case 0:
+			state->cached_cr[reg] = native_read_cr0();
+			break;
+		case 3:
+			state->cached_cr[reg] = native_read_cr3();
+			break;
+		case 4:
+			state->cached_cr[reg] = native_read_cr4();
+			break;
+		default:
+			BUG();
+		}
+		state->cr_valid[reg] = 1;
+	}
+	return state->cached_cr[reg];
+}
+
+static __always_inline void kvm_write_cr(int reg, unsigned long value)
+{
+	struct kvm_paravirt_state *state = x86_read_percpu(paravirt_state);
+
+	state->cr_valid[reg] = 1;
+	state->cached_cr[reg] = value;
+
+	switch (reg) {
+	case 0:
+		native_write_cr0(value);
+		break;
+	case 3:
+		native_write_cr3(value);
+		break;
+	case 4:
+		native_write_cr4(value);
+		break;
+	default:
+		BUG();
+	}
+}
+
+static unsigned long kvm_read_cr0(void)
+{
+	return kvm_read_cr(0);
+}
+
+static void kvm_write_cr0(unsigned long value)
+{
+	kvm_write_cr(0, value);
+}
+
+/*
+ * We trap clts to ensure that our cached cr0 remains consistent.
+ */
+static void kvm_clts(void)
+{
+	write_cr0(read_cr0() & ~CR0_TS_MASK);
+}
+
+static unsigned long kvm_read_cr3(void)
+{
+	return kvm_read_cr(3);
+}
+
+static void kvm_write_cr3(unsigned long value)
+{
+	kvm_write_cr(3, value);
+}
+
+static unsigned long kvm_read_cr4(void)
+{
+	return kvm_read_cr(4);
+}
+
+static void kvm_write_cr4(unsigned long value)
+{
+	kvm_write_cr(4, value);
+}
+
 static void paravirt_ops_setup(void)
 {
 	paravirt_ops.name = "KVM";

@@ -119,6 +212,19 @@ static void paravirt_ops_setup(void)
 		printk(KERN_INFO "KVM: using paravirt clock source\n");
 	}
 
+	if (do_cr_read_caching) {
+		paravirt_ops.clts = kvm_clts;
+		paravirt_ops.read_cr0 = kvm_read_cr0;
+		paravirt_ops.write_cr0 = kvm_write_cr0;
+		paravirt_ops.read_cr3 = kvm_read_cr3;
+		paravirt_ops.write_cr3 = kvm_write_cr3;
+		paravirt_ops.read_cr4 = kvm_read_cr4;
+		paravirt_ops.write_cr4 = kvm_write_cr4;
+
+		/* CR4 always exists in a KVM guest */
+		paravirt_ops.read_cr4_safe = kvm_read_cr4;
+	}
+
 	paravirt_ops.paravirt_enabled = 1;
 
 	apply_paravirt(__parainstructions, __parainstructions_end);

@@ -160,6 +266,9 @@ static int paravirt_initialize(void)
 	if ((edx & KVM_FEATURE_PARAVIRT_CLOCK))
 		do_paravirt_clock = 1;
 
+	if ((edx & KVM_FEATURE_CR_READ_CACHE))
+		do_cr_read_caching = 1;
+
 	on_each_cpu(paravirt_activate, NULL, 0, 1);
 
 	return 0;

drivers/kvm/kvm_main.c +1 -1

index e455fca..35d73b8 100644
@@ -95,7 +95,7 @@ struct vfsmount *kvmfs_mnt;
 
 #define KVM_PARAVIRT_FEATURES \
 	(KVM_FEATURE_VMCA | KVM_FEATURE_NOP_IO_DELAY | \
-	 KVM_FEATURE_PARAVIRT_CLOCK)
+	 KVM_FEATURE_PARAVIRT_CLOCK | KVM_FEATURE_CR_READ_CACHE)
 
 #define KVM_MSR_SET_VMCA	0x87655678

include/linux/kvm_para.h +1 -0

index 4f27fb1..be185da 100644
@@ -16,6 +16,7 @@
 #define KVM_FEATURE_VMCA		(1UL << 0)
 #define KVM_FEATURE_NOP_IO_DELAY	(1UL << 1)
 #define KVM_FEATURE_PARAVIRT_CLOCK	(1UL << 2)
+#define KVM_FEATURE_CR_READ_CACHE	(1UL << 3)
 
 struct kvm_vmca
 {

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Anthony Liguori

author

2007-06-21 03:08 UTC #5 lore raw

PATCH 4/5 KVM: Add paravirt MMU write support

In reply to a message not in the archive

Regards,

Anthony Liguori

Subject: [PATCH] KVM: Add paravirt MMU write support
Author: Anthony Liguori <aliguori-r/[email protected]>

On at least AMD hardware, hypercall based manipulation of page table memory is significantly faster than taking a page fault. Additionally, using hypercalls to manipulation page table memory provides the infrastructure needed to do lazy MMU updates.

Signed-off-by Anthony Liguori <aliguori-r/[email protected]>

arch/i386/kernel/kvm.c +98 -0

index bf77159..21133e4 100644
@@ -25,6 +25,7 @@
 #include <linux/kvm_para.h>
 #include <linux/cpu.h>
 #include <linux/mm.h>
+#include <asm/tlbflush.h>
 
 #include <linux/clocksource.h>
 #include <linux/workqueue.h>

@@ -54,6 +55,7 @@ struct kvm_paravirt_state
 
 static DEFINE_PER_CPU(struct kvm_paravirt_state *, paravirt_state);
 
+static int do_mmu_write;
 static int do_cr_read_caching;
 static int do_nop_io_delay;
 static int do_paravirt_clock;

@@ -197,6 +199,84 @@ static void kvm_write_cr4(unsigned long value)
 	kvm_write_cr(4, value);
 }
 
+static void kvm_mmu_write(void *dest, const void *src, size_t size)
+{
+	const uint8_t *p = src;
+	u32 a1 = 0;
+
+	size >>= 2;
+	if (size == 2)
+		a1 = *(u32 *)&p[4];
+
+	kvm_hypercall(KVM_HYPERCALL_MMU_WRITE, (u32)dest, size, *(u32 *)p, a1);
+}
+
+/*
+ * We only need to hook operations that are MMU writes.  We hook these so that
+ * we can use lazy MMU mode to batch these operations.  We could probably
+ * improve the performance of the host code if we used some of the information
+ * here to simplify processing of batched writes.
+ */
+static void kvm_set_pte(pte_t *ptep, pte_t pte)
+{
+	kvm_mmu_write(ptep, &pte, sizeof(pte));
+}
+
+static void kvm_set_pte_at(struct mm_struct *mm, unsigned long addr,
+			   pte_t *ptep, pte_t pte)
+{
+	kvm_mmu_write(ptep, &pte, sizeof(pte));
+}
+
+static void kvm_set_pte_atomic(pte_t *ptep, pte_t pte)
+{
+	kvm_mmu_write(ptep, &pte, sizeof(pte));
+}
+
+static void kvm_set_pte_present(struct mm_struct *mm, unsigned long addr,
+				pte_t *ptep, pte_t pte)
+{
+	kvm_mmu_write(ptep, &pte, sizeof(pte));
+}
+
+static void kvm_pte_clear(struct mm_struct *mm,
+			  unsigned long addr, pte_t *ptep)
+{
+	pte_t pte = __pte(0);
+	kvm_mmu_write(ptep, &pte, sizeof(pte));
+}
+
+static void kvm_set_pmd(pmd_t *pmdp, pmd_t pmd)
+{
+	kvm_mmu_write(pmdp, &pmd, sizeof(pmd));
+}
+
+static void kvm_set_pud(pud_t *pudp, pud_t pud)
+{
+	kvm_mmu_write(pudp, &pud, sizeof(pud));
+}
+
+static void kvm_pmd_clear(pmd_t *pmdp)
+{
+	pmd_t pmd = __pmd(0);
+	kvm_mmu_write(pmdp, &pmd, sizeof(pmd));
+}
+
+static void kvm_flush_tlb(void)
+{
+	kvm_hypercall(KVM_HYPERCALL_FLUSH_TLB, 0, 0, 0, 0);
+}
+
+static void kvm_flush_tlb_single(unsigned long addr)
+{
+	kvm_hypercall(KVM_HYPERCALL_FLUSH_TLB_SINGLE, addr, 0, 0, 0);
+}
+
+static void kvm_release_pt(u32 pfn)
+{
+	kvm_hypercall(KVM_HYPERCALL_RELEASE_PT, pfn << PAGE_SHIFT, 0, 0, 0);
+}
+
 static void paravirt_ops_setup(void)
 {
 	paravirt_ops.name = "KVM";

@@ -225,6 +305,21 @@ static void paravirt_ops_setup(void)
 		paravirt_ops.read_cr4_safe = kvm_read_cr4;
 	}
 
+	if (do_mmu_write) {
+		paravirt_ops.set_pte = kvm_set_pte;
+		paravirt_ops.set_pte_at = kvm_set_pte_at;
+		paravirt_ops.set_pte_atomic = kvm_set_pte_atomic;
+		paravirt_ops.set_pte_present = kvm_set_pte_present;
+		paravirt_ops.pte_clear = kvm_pte_clear;
+		paravirt_ops.set_pmd = kvm_set_pmd;
+		paravirt_ops.pmd_clear = kvm_pmd_clear;
+		paravirt_ops.set_pud = kvm_set_pud;
+		paravirt_ops.flush_tlb_user = kvm_flush_tlb;
+		paravirt_ops.flush_tlb_single = kvm_flush_tlb_single;
+		paravirt_ops.release_pt = kvm_release_pt;
+		paravirt_ops.release_pd = kvm_release_pt;
+	}
+
 	paravirt_ops.paravirt_enabled = 1;
 }

@@ -267,6 +362,9 @@ static int paravirt_initialize(void)
 	if ((edx & KVM_FEATURE_CR_READ_CACHE))
 		do_cr_read_caching = 1;
 
+	if ((edx & KVM_FEATURE_MMU_WRITE))
+		do_mmu_write = 1;
+
 	on_each_cpu(paravirt_activate, NULL, 0, 1);
 
 	return 0;

drivers/kvm/kvm.h +1 -0

index b08272b..9a7462a 100644
@@ -538,6 +538,7 @@ void kvm_resched(struct kvm_vcpu *vcpu);
 void kvm_load_guest_fpu(struct kvm_vcpu *vcpu);
 void kvm_put_guest_fpu(struct kvm_vcpu *vcpu);
 void kvm_flush_remote_tlbs(struct kvm *kvm);
+void mmu_unshadow(struct kvm_vcpu *vcpu, gfn_t gfn);
 
 int kvm_read_guest(struct kvm_vcpu *vcpu,
 	       gva_t addr,

drivers/kvm/kvm_main.c +51 -1

77 lines not on this page · raw message

drivers/kvm/mmu.c +1 -1

index ad50cfd..1581286 100644
@@ -728,7 +728,7 @@ static int kvm_mmu_unprotect_page(struct kvm_vcpu *vcpu, gfn_t gfn)
 	return r;
 }
 
-static void mmu_unshadow(struct kvm_vcpu *vcpu, gfn_t gfn)
+void mmu_unshadow(struct kvm_vcpu *vcpu, gfn_t gfn)
 {
 	struct kvm_mmu_page *page;

include/linux/kvm_para.h +7 -0

index d4aa7e8..560de6a 100644
@@ -26,6 +26,7 @@ static int __init kvm_guest_init(void)
 #define KVM_FEATURE_NOP_IO_DELAY	(1UL << 1)
 #define KVM_FEATURE_PARAVIRT_CLOCK	(1UL << 2)
 #define KVM_FEATURE_CR_READ_CACHE	(1UL << 3)
+#define KVM_FEATURE_MMU_WRITE		(1UL << 4)
 
 struct kvm_vmca
 {

@@ -46,7 +47,13 @@ struct kvm_vmca
 
 #define KVM_ENOSYS	ENOSYS
 #define KVM_EINVAL	EINVAL
+#define KVM_EFAULT	EFAULT
+#define KVM_E2BIG	E2BIG
 
 #define KVM_HYPERCALL_UPDATE_TIME	0
+#define KVM_HYPERCALL_MMU_WRITE		1
+#define KVM_HYPERCALL_FLUSH_TLB		2
+#define KVM_HYPERCALL_FLUSH_TLB_SINGLE	3
+#define KVM_HYPERCALL_RELEASE_PT	4
 
 #endif

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Anthony Liguori

author

2007-06-21 03:09 UTC #6 lore raw

PATCH 5/5 KVM: Add hypercall queue for paravirt_ops implementation

In reply to a message not in the archive

Regards,

Anthony Liguori

Subject: [PATCH] KVM: Add hypercall queue for paravirt_ops implementation
Author: Anthony Liguori <aliguori-r/[email protected]>

Implemented a hypercall queue that can be used when paravirt_ops lazy mode is enabled. This patch enables queueing of MMU write operations and CR updates. This results in about a 50% bump in kernbench performance.

Signed-off-by Anthony Liguori <aliguori-r/[email protected]>

arch/i386/kernel/kvm.c +124 ~~-19~~

index 21133e4..52eb50d 100644
@@ -34,6 +34,7 @@
 #include <linux/init.h>
 #include <linux/dmi.h>
 #include <linux/acpi_pmtmr.h>
+#include <linux/hardirq.h>
 
 #include "mach_timer.h"

@@ -46,8 +47,12 @@ struct kvm_paravirt_state
 	unsigned long cached_cr[5];
 	int cr_valid[5];
 
-	struct kvm_vmca *vmca;
+	enum paravirt_lazy_mode mode;
 	struct kvm_hypercall_entry *queue;
+	int queue_index;
+	int max_queue_index;
+
+	struct kvm_vmca *vmca;
 	void (*hypercall)(void);
 
 	u64 vmca_gpa;

@@ -55,18 +60,18 @@ struct kvm_paravirt_state
 
 static DEFINE_PER_CPU(struct kvm_paravirt_state *, paravirt_state);
 
+static int do_hypercall_batching;
 static int do_mmu_write;
 static int do_cr_read_caching;
 static int do_nop_io_delay;
 static int do_paravirt_clock;
 static u64 msr_set_vmca;
 
-static long kvm_hypercall(unsigned int nr, unsigned long p1,
-			  unsigned long p2, unsigned long p3,
-			  unsigned long p4)
+static long _kvm_hypercall(struct kvm_paravirt_state *state,
+			   unsigned int nr, unsigned long p1,
+			   unsigned long p2, unsigned long p3,
+			   unsigned long p4)
 {
-	struct kvm_paravirt_state *state
-		= per_cpu(paravirt_state, smp_processor_id());
 	long ret;
 
 	asm volatile("call *(%6) \n\t"

@@ -83,6 +88,69 @@ static long kvm_hypercall(unsigned int nr, unsigned long p1,
 	return ret;
 }
 
+static int can_defer_hypercall(struct kvm_paravirt_state *state,
+			       unsigned int nr)
+{
+	if (state->mode == PARAVIRT_LAZY_MMU) {
+		switch (nr) {
+		case KVM_HYPERCALL_MMU_WRITE:
+		case KVM_HYPERCALL_FLUSH_TLB:
+		case KVM_HYPERCALL_FLUSH_TLB_SINGLE:
+			return 1;
+		}
+	} else if (state->mode == PARAVIRT_LAZY_CPU) {
+		if (nr == KVM_HYPERCALL_SET_CR)
+			return 1;
+	}
+
+	return 0;
+}
+
+static void hypercall_queue_flush(struct kvm_paravirt_state *state)
+{
+	if (state->queue_index) {
+		_kvm_hypercall(state, KVM_HYPERCALL_FLUSH, __pa(state->queue),
+			       state->queue_index, 0, 0);
+		state->queue_index = 0;
+	}
+}
+
+static void _kvm_hypercall_defer(struct kvm_paravirt_state *state,
+				 unsigned int nr,
+				 unsigned long p1, unsigned long p2,
+				 unsigned long p3, unsigned long p4)
+{
+	struct kvm_hypercall_entry *entry;
+
+	BUG_ON(preemptible());
+
+	if (state->queue_index == state->max_queue_index)
+		hypercall_queue_flush(state);
+
+	entry = &state->queue[state->queue_index++];
+	entry->nr = nr;
+	entry->p1 = p1;
+	entry->p2 = p2;
+	entry->p3 = p3;
+	entry->p4 = p4;
+}
+
+static long kvm_hypercall(unsigned int nr, unsigned long p1,
+			  unsigned long p2, unsigned long p3,
+			  unsigned long p4)
+{
+	struct kvm_paravirt_state *state = get_cpu_var(paravirt_state);
+	long ret = 0;
+
+	if (can_defer_hypercall(state, nr))
+		_kvm_hypercall_defer(state, nr, p1, p2, p3, p4);
+	else
+		ret = _kvm_hypercall(state, nr, p1, p2, p3, p4);
+
+	put_cpu_var(paravirt_state);
+	return ret;
+}
+
 static cycle_t kvm_clocksource_read(void)
 {
 	struct kvm_paravirt_state *state = get_cpu_var(paravirt_state);

@@ -146,18 +214,22 @@ static __always_inline void kvm_write_cr(int reg, unsigned long value)
 	state->cr_valid[reg] = 1;
 	state->cached_cr[reg] = value;
 
-	switch (reg) {
-	case 0:
-		native_write_cr0(value);
-		break;
-	case 3:
-		native_write_cr3(value);
-		break;
-	case 4:
-		native_write_cr4(value);
-		break;
-	default:
-		BUG();
+	if (state->mode == PARAVIRT_LAZY_CPU)
+		kvm_hypercall(KVM_HYPERCALL_SET_CR, reg, value, 0, 0);
+	else {
+		switch (reg) {
+		case 0:
+			native_write_cr0(value);
+			break;
+		case 3:
+			native_write_cr3(value);
+			break;
+		case 4:
+			native_write_cr4(value);
+			break;
+		default:
+			BUG();
+		}
 	}
 }

@@ -269,7 +341,24 @@ static void kvm_flush_tlb(void)
 
 static void kvm_flush_tlb_single(unsigned long addr)
 {
-	kvm_hypercall(KVM_HYPERCALL_FLUSH_TLB_SINGLE, addr, 0, 0, 0);
+	int mode = x86_read_percpu(paravirt_state)->mode;
+
+	if (mode == PARAVIRT_LAZY_MMU)
+		kvm_hypercall(KVM_HYPERCALL_FLUSH_TLB_SINGLE, addr, 0, 0, 0);
+	else
+		__native_flush_tlb_single(addr);
+}
+
+static void kvm_set_lazy_mode(enum paravirt_lazy_mode mode)
+{
+	struct kvm_paravirt_state *state
+		= per_cpu(paravirt_state, smp_processor_id());
+
+	if (mode == PARAVIRT_LAZY_FLUSH || mode == PARAVIRT_LAZY_NONE)
+		hypercall_queue_flush(state);
+
+	if (mode != PARAVIRT_LAZY_FLUSH)
+		state->mode = mode;
 }
 
 static void kvm_release_pt(u32 pfn)

@@ -320,6 +409,9 @@ static void paravirt_ops_setup(void)
 		paravirt_ops.release_pd = kvm_release_pt;
 	}
 
+	if (do_hypercall_batching)
+		paravirt_ops.set_lazy_mode = kvm_set_lazy_mode;
+
 	paravirt_ops.paravirt_enabled = 1;
 }

@@ -365,6 +457,9 @@ static int paravirt_initialize(void)
 	if ((edx & KVM_FEATURE_MMU_WRITE))
 		do_mmu_write = 1;
 
+	if ((edx & KVM_FEATURE_HYPERCALL_BATCHING))
+		do_hypercall_batching = 1;
+
 	on_each_cpu(paravirt_activate, NULL, 0, 1);
 
 	return 0;

@@ -375,6 +470,9 @@ static __init void paravirt_free_state(struct kvm_paravirt_state *state)
 	if (!state)
 		return;
 
+	if (state->queue)
+		__free_page(pfn_to_page(__pa(state->queue) >> PAGE_SHIFT));
+
 	if (state->hypercall)
 		__free_page(pfn_to_page(__pa(state->hypercall) >> PAGE_SHIFT));

@@ -401,8 +499,15 @@ static __init struct kvm_paravirt_state *paravirt_alloc_state(void)
 	if (!state->hypercall)
 		goto err;
 
+	state->queue = (void *)get_zeroed_page(GFP_KERNEL);
+	if (!state->queue)
+		goto err;
+
 	state->vmca_gpa = __pa(state->vmca);
 	state->vmca->hypercall_gpa = __pa(state->hypercall);
+	state->queue_index = 0;
+	state->max_queue_index
+		= (PAGE_SIZE / sizeof(struct kvm_hypercall_entry));
 
 	return state;

drivers/kvm/kvm.h +1 -0

index 9a7462a..7e53374 100644
@@ -291,6 +291,7 @@ struct kvm_vcpu {
 	gpa_t para_state_gpa;
 	struct page *para_state_page;
 	gpa_t hypercall_gpa;
+	struct page *queue_page;
 	unsigned long cr4;
 	unsigned long cr8;
 	u64 pdptrs[4]; /* pae */

drivers/kvm/kvm_main.c +59 -2

91 lines not on this page · raw message

include/linux/kvm_para.h +12 -0

index 560de6a..e220832 100644
@@ -27,6 +27,7 @@ static int __init kvm_guest_init(void)
 #define KVM_FEATURE_PARAVIRT_CLOCK	(1UL << 2)
 #define KVM_FEATURE_CR_READ_CACHE	(1UL << 3)
 #define KVM_FEATURE_MMU_WRITE		(1UL << 4)
+#define KVM_FEATURE_HYPERCALL_BATCHING	(1UL << 5)
 
 struct kvm_vmca
 {

@@ -34,6 +35,15 @@ struct kvm_vmca
 	u64 real_nsecs;
 };
 
+struct kvm_hypercall_entry
+{
+	unsigned long nr;
+	unsigned long p1;
+	unsigned long p2;
+	unsigned long p3;
+	unsigned long p4;
+};
+
 /*
  * Hypercall calling convention:
  *

@@ -55,5 +65,7 @@ struct kvm_vmca
 #define KVM_HYPERCALL_FLUSH_TLB		2
 #define KVM_HYPERCALL_FLUSH_TLB_SINGLE	3
 #define KVM_HYPERCALL_RELEASE_PT	4
+#define KVM_HYPERCALL_SET_CR		5
+#define KVM_HYPERCALL_FLUSH		6
 
 #endif

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Anthony Liguori

author

2007-06-21 03:16 UTC #7 lore raw

In reply to a message not in the archive

Quoted 2 lines · Anthony Liguori

As usual, the latest bits are available at
http://hg.codemonkey.ws/kvm-paravirt

I neglected to mention that the major missing piece right now is save/restore support. Since the VMCA is set with an MSR, I think I can just expose this MSR to userspace and add another save/restore device in QEMU. I'll need to get the real_time too to ensure that the paravirt clock remains monotonic. I figure I could make this an MSR too although that's a bit ugly.

Regards,

Anthony Liguori

Quoted 4 lines

Regards,
Anthony Liguori

Avi Kivity

2007-06-21 09:18 UTC #8 lore raw

In reply to a message not in the archive

Quoted 12 lines · Anthony Liguori

Hi,
This is an update to the paravirt_ops KVM backend series. I've made a number of changes and attempted to incorporate all the feedback from the last review. Some highlights:
1) Clean up the paravirt time source patch to use a more Xen-like model
2) Change the hypercall queueing to pass a PA on the flush hypercall
3) Add MMU support for release_{pt,pd} and TLB flush
4) Use KVM specific errno values
5) Switch from per_cpu to more appropriate functions

Quoted 7 lines

As for performance, I've got a few interesting results. kbuild with a guest using 2G of memory goes from 19 minutes to 12 minutes with the full series applied. Using 512mb, the build time goes from 10.75 minutes to 9 minutes. For 512mb, native is around 7 minutes so that's pretty close to what Avi had seen. The more dramatic improvement with large memory guests is probably because of the increased shadow page table activity due to high mem.

Ah, that explains why we were getting such different results. My tests were on x86-64.

Quoted 6 lines

virtbench shows major improvements but I'm not 100% confident yet in the results as they are not very stable. I don't yet have a benchmark that shows the benefit of the CR caching so if I don't find one, I'll drop that from the queue.

The only barrier to merging is that we're introducing yet another stable ABI. Since we can turn off things that turn out not so good later, and expect the guest to survive, I'm not to worried, so I'm inclined to merge this.

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

A message that is not in the archive (sent off-list or not yet received): 467A4280.9060503-atKUWr5tajBWk0Htik3J/[email protected]

Anthony Liguori

author

2007-06-21 13:19 UTC #10 lore raw

Quoted 31 lines · Avi Kivity

Quoted 12 lines · Anthony Liguori
Hi,
This is an update to the paravirt_ops KVM backend series. I've made a number of changes and attempted to incorporate all the feedback from the last review. Some highlights:
1) Clean up the paravirt time source patch to use a more Xen-like model
2) Change the hypercall queueing to pass a PA on the flush hypercall
3) Add MMU support for release_{pt,pd} and TLB flush
4) Use KVM specific errno values
5) Switch from per_cpu to more appropriate functions
Quoted 7 lines
As for performance, I've got a few interesting results. kbuild with a guest using 2G of memory goes from 19 minutes to 12 minutes with the full series applied. Using 512mb, the build time goes from 10.75 minutes to 9 minutes. For 512mb, native is around 7 minutes so that's pretty close to what Avi had seen. The more dramatic improvement with large memory guests is probably because of the increased shadow page table activity due to high mem.
Ah, that explains why we were getting such different results. My tests were on x86-64.
Quoted 6 lines
virtbench shows major improvements but I'm not 100% confident yet in the results as they are not very stable. I don't yet have a benchmark that shows the benefit of the CR caching so if I don't find one, I'll drop that from the queue.

Quoted 4 lines

The only barrier to merging is that we're introducing yet another stable ABI. Since we can turn off things that turn out not so good later, and expect the guest to survive, I'm not to worried, so I'm inclined to merge this.

Some unsigned longs are in kvm_para.h so x86_64 won't work ATM as a host. I'll fix that and do proper testing on a 64 bit host for the next posting (along with save/restore support).

Regards,

Anthony Liguori

Li, Xin B

2007-06-21 09:26 UTC #11 lore raw

In reply to a message not in the archive

Quoted 7 lines

As for performance, I've got a few interesting results. kbuild with a guest using 2G of memory goes from 19 minutes to 12 minutes with the full series applied. Using 512mb, the build time goes from 10.75 minutes to 9 minutes. For 512mb, native is around 7 minutes so that's pretty close to what Avi had seen. The more dramatic improvement with large memory guests is probably because of the increased shadow page table activity due to high mem.

Exactly, physical memory above 896M on x86_32 are mapped using temp mapping, which needs lots of invlpg instructions.
On xen, it needs to wake all the shadows to find the shadow of high mem, since Xen doesn't use reverse map. On KVM, should be much better. -Xin