Skip to contentSkip to conversation

kvmppc: paravirtualization interface

RFC PATCH In review [email protected] 3 participants 20 messages Jul 23, 2008 – Sep 16, 2008

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover kvmppc: paravirtualization interface

From: Christian Ehrhardt <[email protected]>

This patch series implements a paravirtualization interface using:
- the device tree mechanism to pass hypervisor informations to the guest
- hypercalls for guest->host calls
- an example exploiter of that interface (magic page)
This is work in progress, but working so far. I just start to really exploit the fuctionality behind the magic page mechanism therefor I can't provide any performance improvements so far, but it is evolved enough for RFC and to start the standardization discussion.

The used hypercall ABI was already discussed on the embedded-hypervisor mailing list and is available at http://kvm.qumranet.com/kvmwiki/PowerPC_Hypercall_ABI

The device tree format used here (ºse for the discussions on embedded-hypervisor) is the following.
- A node "hypervisor" to show the general availability of some hypervisor data
- flags for features like the example "feature,pv-magicpage" setting 1 = available, everything else = unavailable
- Some features might need to pass more data and can use an entry in the device tree like the example of "data,pv-magicpage-size"

Parties on cc:
[email protected]
The patches affect code in the generic powerpc boot&setup so I would be happy about comments if the hooks are ok that way.
[email protected]
This power.org TSC discusses about standardization of the virtualization interfaces. This patch series is perfectly suited due to it's simple changes to start the discussion about the device tree there.
[email protected]
The code is made for kvm on powerpc which lives on this list.

[patches in series]
Subject: [PATCH 1/6] kvmppc: read device tree hypervisor node infrastructure
Providing the guest functionality to read hypervisor features from the device tree and adding the basic hook to the powerpc boot6setup code
Subject: [PATCH 2/6] kvmppc: add hypercall infrastructure - host part
Subject: [PATCH 3/6] kvmppc: add hypercall infrastructure - guest part
patch 2&3 add the hypercall infrastruture as mentioned above
Subject: [PATCH 4/6] kvmppc: magic page hypercall - host part
Subject: [PATCH 5/6] kvmppc: magic page paravirtualization - guest part
patch 4&5 add the magic page mechanism which will later on be used for binary rewriting the guest.
Subject: [PATCH 6/6] kvmppc: kvm-userspace: device tree modification for magicpage
This connects host and guest reading host capabilities and modifying the device tree passed to the guest accordingly

---
[diffstat]

 arch/powerpc/kernel/kvm.c            |   48 +++++++++++++++++++++++++++++++++++
 arch/powerpc/kvm/emulate.c           |    5 +++
 b/arch/powerpc/kernel/Makefile       |    2 +
 b/arch/powerpc/kernel/kvm.c          |   30 +++++++++++++++++++++
 b/arch/powerpc/kernel/setup_32.c     |    3 ++
 b/arch/powerpc/kvm/emulate.c         |   27 +++++++++++++++++++
 b/arch/powerpc/kvm/powerpc.c         |   18 ++++++++++++-
 b/arch/powerpc/platforms/44x/Kconfig |    7 +++++
 b/include/asm-powerpc/kvm_para.h     |   37 ++++++++++++++++++++++++--
 b/include/linux/kvm.h                |    6 ++++
 b/libkvm/libkvm-powerpc.c            |    6 ++++
 b/libkvm/libkvm.h                    |    6 ++++
 b/qemu/hw/device_tree.c              |   10 +++++++
 b/qemu/hw/device_tree.h              |    1
 b/qemu/hw/ppc440_bamboo.c            |   15 ++++++++++
 b/qemu/qemu-kvm-powerpc.c            |    5 +++
 b/qemu/qemu-kvm.h                    |    1
 include/asm-powerpc/kvm_para.h       |   47 +++++++++++++++++++++++++++++++++-
 18 files changed, 269 insertions(+), 5 deletions(-)
#1 lore raw

PATCH 1/6 kvmppc: read device tree hypervisor node infrastructure

From: Christian Ehrhardt <[email protected]>

This patch adds the guest portion of the device tree based host->guest communication. Using the device tree infrastructure this patch implements kvm_para_available and kvm_arch_para_features (in this patch just the infrastructure, no specific feature registered).

---

[diffstat]

 arch/powerpc/kernel/Makefile       |    2 ++
 arch/powerpc/kernel/kvm.c          |   30 ++++++++++++++++++++++++++++++
 arch/powerpc/kernel/setup_32.c     |    3 +++
 arch/powerpc/platforms/44x/Kconfig |    7 +++++++
 include/asm-powerpc/kvm_para.h     |   37 ++++++++++++++++++++++++++++++++++---
 5 files changed, 76 insertions(+), 3 deletions(-)

[diff]

arch/powerpc/kernel/Makefile +2 -0
@@ -80,6 +80,8 @@
 
 obj-$(CONFIG_8XX_MINIMAL_FPEMU) += softemu8xx.o
 
+obj-$(CONFIG_KVM_GUEST)		+= kvm.o
+
 ifneq ($(CONFIG_PPC_INDIRECT_IO),y)
 obj-y				+= iomap.o
 endif
arch/powerpc/kernel/kvm.c new file +30 -0
new file mode 100644
@@ -0,0 +1,30 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+ *
+ * Copyright IBM Corp. 2008
+ *
+ * Authors:
+ * 	Hollis Blanchard <[email protected]>
+ * 	Christian Ehrhardt <[email protected]>
+ */
+
+#include <linux/percpu.h>
+#include <linux/mm.h>
+#include <linux/kvm_para.h>
+
+void __init kvm_guest_init(void)
+{
+	if (!kvm_para_available())
+		return;
+}
arch/powerpc/kernel/setup_32.c +3 -0
@@ -17,6 +17,7 @@
 #include <linux/cpu.h>
 #include <linux/console.h>
 #include <linux/lmb.h>
+#include <linux/kvm_para.h>
 
 #include <asm/io.h>
 #include <asm/prom.h>
@@ -319,5 +320,7 @@
 		ppc_md.setup_arch();
 	if ( ppc_md.progress ) ppc_md.progress("arch: exit", 0x3eab);
 
+	kvm_guest_init();
+
 	paging_init();
 }
arch/powerpc/platforms/44x/Kconfig +7 -0
@@ -152,3 +152,10 @@
 # 44x errata/workaround config symbols, selected by the CPU models above
 config IBM440EP_ERR42
 	bool
+
+config KVM_GUEST
+	bool "KVM Guest support"
+	depends on EXPERIMENTAL
+	help
+	This option enables various optimizations for running under the KVM
+	hypervisor.
include/asm-powerpc/kvm_para.h +34 -3
@@ -14,7 +14,9 @@
  *
  * Copyright IBM Corp. 2008
  *
- * Authors: Hollis Blanchard <[email protected]>
+ * Authors:
+ * 	Hollis Blanchard <[email protected]>
+ * 	Christian Ehrhardt <[email protected]>
  */
 
 #ifndef __POWERPC_KVM_PARA_H__
@@ -22,15 +24,44 @@
 
 #ifdef __KERNEL__
 
+#include <linux/of.h>
+
+static struct kvmppc_para_features {
+	char *dtcell;
+	int feature;
+} para_features[] = {
+};
+
 static inline int kvm_para_available(void)
 {
-	return 0;
+	struct device_node *dn;
+
+	dn = of_find_node_by_path("/hypervisor");
+
+	return !!dn;
 }
 
 static inline unsigned int kvm_arch_para_features(void)
 {
-	return 0;
+	struct device_node *dn;
+	const int *dtval;
+	unsigned int features = 0;
+	int i;
+
+	dn = of_find_node_by_path("/hypervisor");
+	if (!dn)
+		return 0;
+
+	for (i = 0; i < ARRAY_SIZE(para_features)-1; i++) {
+		dtval = of_get_property(dn, para_features[i].dtcell, NULL);
+		if (dtval && *dtval = 1)
+			features |= (1 << para_features[i].feature);
+	}
+
+	return features;
 }
+
+void kvm_guest_init(void);
 
 #endif /* __KERNEL__ */
 
#2 lore raw

Re: [PATCH 1/6] kvmppc: read device tree hypervisor node infrastructure

On Wed, Jul 23, 2008 at 10:36:42AM +0200, [email protected] wrote:

Hi Christian,
A few comments inlined ...

Quoted 22 lines
include/asm-powerpc/kvm_para.h +7 -1
@@ -14,7 +14,9 @@
  *
  * Copyright IBM Corp. 2008
  *
- * Authors: Hollis Blanchard <[email protected]>
+ * Authors:
+ * 	Hollis Blanchard <[email protected]>
+ * 	Christian Ehrhardt <[email protected]>
  */
 
 #ifndef __POWERPC_KVM_PARA_H__
@@ -22,15 +24,44 @@
 
 #ifdef __KERNEL__
 
+#include <linux/of.h>
+
+static struct kvmppc_para_features {
+	char *dtcell;

include/asm-powerpc/kvm_para.h line 39

Quoted 10 lines
include/asm-powerpc/kvm_para.h +7 -1
+	int feature;
+} para_features[] = {
+};
+
 static inline int kvm_para_available(void)
 {
-	return 0;
+	struct device_node *dn;
+
+	dn = of_find_node_by_path("/hypervisor");

You need an of_node_put(dn);

Quoted 7 lines
include/asm-powerpc/kvm_para.h +2 -1
+
+	return !!dn;
 }
 
 static inline unsigned int kvm_arch_para_features(void)
 {
-	return 0;

include/asm-powerpc/kvm_para.h line 55

Quoted 10 lines
include/asm-powerpc/kvm_para.h +10 -0
+	struct device_node *dn;
+	const int *dtval;
+	unsigned int features = 0;
+	int i;
+
+	dn = of_find_node_by_path("/hypervisor");
+	if (!dn)
+		return 0;
+
+	for (i = 0; i < ARRAY_SIZE(para_features)-1; i++) {

Why -1? Isn't ARRAY_SIZE(para_features) adequate?

include/asm-powerpc/kvm_para.h line 60

Quoted 5 lines
include/asm-powerpc/kvm_para.h +5 -0
+		dtval = of_get_property(dn, para_features[i].dtcell, NULL);
+		if (dtval && *dtval = 1)
+			features |= (1 << para_features[i].feature);
+	}
+

You need an of_node_put(dn);

include/asm-powerpc/kvm_para.h line 62

Quoted 2 lines
include/asm-powerpc/kvm_para.h +1 -0
+	return features;
 }

Yours Tony

linux.conf.au http://www.marchsouth.org/
Jan 19 - 24 2009 The Australian Linux Technical Conference!

#3 lore raw

Re: [PATCH 1/6] kvmppc: read device tree hypervisor node infrastructure

Quoted 6 lines · Tony Breeds

On Wed, Jul 23, 2008 at 10:36:42AM +0200, [email protected] wrote:

Hi Christian,
A few comments inlined ...

[...]

Quoted 12 lines

include/asm-powerpc/kvm_para.h line 39

Quoted 8 lines
include/asm-powerpc/kvm_para.h +4 -1
+
 static inline int kvm_para_available(void)
 {
-	return 0;
+	struct device_node *dn;
+
+	dn = of_find_node_by_path("/hypervisor");

You need an of_node_put(dn);

I just looked at the linux/of.h and did not see that I have to free it again.
Thanks for the hint, I inserted both calls.

Quoted 4 lines
Quoted 4 lines
include/asm-powerpc/kvm_para.h +2 -0
+
+	return !!dn;
 }
Quoted 17 lines
Quoted 4 lines

static inline unsigned int kvm_arch_para_features(void) {

include/asm-powerpc/kvm_para.h +0 -1
-	return 0;

include/asm-powerpc/kvm_para.h line 55

Quoted 10 lines
include/asm-powerpc/kvm_para.h +10 -0
+	struct device_node *dn;
+	const int *dtval;
+	unsigned int features = 0;
+	int i;
+
+	dn = of_find_node_by_path("/hypervisor");
+	if (!dn)
+		return 0;
+
+	for (i = 0; i < ARRAY_SIZE(para_features)-1; i++) {

Why -1? Isn't ARRAY_SIZE(para_features) adequate?

yeah I already had this, bit the change was folded into the wrong patch, fixed now

[...]

Quoted 6 lines

Yours Tony

linux.conf.au http://www.marchsouth.org/
Jan 19 - 24 2009 The Australian Linux Technical Conference!

Signature · 8 lines
Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#4 lore raw

PATCH 2/6 kvmppc: add hypercall infrastructure - host part

In reply to [email protected]

From: Christian Ehrhardt <[email protected]>

This adds the host portion of the hypercall infrastructure which receives the guest calls - no specific hcall function is implemented in this patch.

---

[diffstat]

 arch/powerpc/kvm/emulate.c     |   27 +++++++++++++++++++++++++++
 include/asm-powerpc/kvm_para.h |    2 ++
 2 files changed, 29 insertions(+)

[diff]

arch/powerpc/kvm/emulate.c +27 -0
@@ -203,6 +203,24 @@
 	kvmppc_set_msr(vcpu, vcpu->arch.srr1);
 }
 
+static int kvmppc_do_hypercall(struct kvm_vcpu *vcpu)
+{
+	int ret = 0;
+
+	switch (vcpu->arch.gpr[0]) {
+	default:
+		printk(KERN_ERR"unknown hypercall %d\n", vcpu->arch.gpr[0]);
+		kvmppc_dump_vcpu(vcpu);
+		ret = -ENOSYS;
+	}
+
+	vcpu->arch.gpr[11] = ret;
+	vcpu->arch.pc += 4; /* Advance past hypercall instruction. */
+
+	return ret;
+}
+
+
 /* XXX to do:
  * lhax
  * lhaux
@@ -232,6 +250,15 @@
 	int advance = 1;
 
 	switch (get_op(inst)) {
+	case 0:
+		if (inst = KVM_HYPERCALL_BIN) {
+			kvmppc_do_hypercall(vcpu);
+			advance = 0; /* kvmppc_do_hypercall handles the PC. */
+		} else {
+			printk(KERN_ERR"unknown op %d\n", get_op(inst));
+			emulated = EMULATE_FAIL;
+		}
+		break;
 	case 3:                                                 /* trap */
 		printk("trap!\n");
 		kvmppc_queue_exception(vcpu, BOOKE_INTERRUPT_PROGRAM);
include/asm-powerpc/kvm_para.h +2 -0
@@ -22,6 +22,8 @@
 
 #ifdef __KERNEL__
 
+#define KVM_HYPERCALL_BIN 0x03ffffff
+
 static inline int kvm_para_available(void)
 {
 	return 0;
#5 lore raw

Re: [PATCH 2/6] kvmppc: add hypercall infrastructure - host part

Quoted 1 line · On Wed, Jul 23, 2008 at 10:36:43AM +0200, [email protected] wrote:

From: Christian Ehrhardt <[email protected]>

Hi Christian,
A few comments

Quoted 4 lines

arch/powerpc/kvm/emulate.c line 212

Quoted 10 lines
arch/powerpc/kvm/emulate.c +7 -0
@@ -203,6 +203,24 @@
 	kvmppc_set_msr(vcpu, vcpu->arch.srr1);
 }
 
+static int kvmppc_do_hypercall(struct kvm_vcpu *vcpu)
+{
+	int ret = 0;
+
+	switch (vcpu->arch.gpr[0]) {
+	default:
+		printk(KERN_ERR"unknown hypercall %d\n", vcpu->arch.gpr[0]);

I think the preffered style is printk(KERN_ERR "...) You've made the same style mistake in most of you printk()'s in your other patches aswell.

Yours Tony

linux.conf.au http://www.marchsouth.org/
Jan 19 - 24 2009 The Australian Linux Technical Conference!

#6 lore raw

Re: [PATCH 2/6] kvmppc: add hypercall infrastructure - host part

Quoted 9 lines · Tony Breeds
Quoted 8 lines · On Wed, Jul 23, 2008 at 10:36:43AM +0200, [email protected] wrote:

From: Christian Ehrhardt <[email protected]>

arch/powerpc/kvm/emulate.c +0 -0
@@ -203,6 +203,24 @@
 	kvmppc_set_msr(vcpu, vcpu->arch.srr1);
 }
 
Quoted 11 lines

arch/powerpc/kvm/emulate.c line 212

Quoted 7 lines
arch/powerpc/kvm/emulate.c +7 -0
+static int kvmppc_do_hypercall(struct kvm_vcpu *vcpu)
+{
+	int ret = 0;
+
+	switch (vcpu->arch.gpr[0]) {
+	default:
+		printk(KERN_ERR"unknown hypercall %d\n", vcpu->arch.gpr[0]);

I think the preffered style is printk(KERN_ERR "...) You've made the same style mistake in most of you printk()'s in your other patches aswell.

Note that these days people use pr_err() instead.

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   [email protected]
Internet: http://www.sony-europe.com/

A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis 293-0376800-10 GEBA-BE-BB

#7 lore raw

PATCH 3/6 kvmppc: add hypercall infrastructure - guest part

In reply to [email protected]

From: Christian Ehrhardt <[email protected]>

This adds the guest portion of the hypercall infrastructure, basically an illegal instruction with a defined layout.
See http://kvm.qumranet.com/kvmwiki/PowerPC_Hypercall_ABI for more detail on the hypercall ABI for powerpc.

---

[diffstat]

 kvm_para.h |   16 ++++++++++++++++
 1 file changed, 16 insertions(+)

[diff]

include/asm-powerpc/kvm_para.h +16 -0
@@ -25,6 +25,8 @@
 #ifdef __KERNEL__
 
 #include <linux/of.h>
+
+#define KVM_HYPERCALL_BIN 0x03ffffff
 
 static struct kvmppc_para_features {
 	char *dtcell;
@@ -63,6 +65,20 @@
 
 void kvm_guest_init(void);
 
+static inline long kvm_hypercall1(unsigned int nr, unsigned long p1)
+{
+	register unsigned long hcall asm ("r0") = nr;
+	register unsigned long arg1 asm ("r3") = p1;
+	register long ret asm ("r11");
+
+	asm volatile(".long %1"
+			: "=r"(ret)
+			: "i"(KVM_HYPERCALL_BIN), "r"(hcall), "r"(arg1)
+			: "r4", "r5", "r6", "r7", "r8",
+			  "r9", "r10", "r12", "cc");
+	return ret;
+}
+
 #endif /* __KERNEL__ */
 
 #endif /* __POWERPC_KVM_PARA_H__ */
#8 lore raw

Re: [PATCH 3/6] kvmppc: add hypercall infrastructure - guest part

Quoted 1 line · On Wed, Jul 23, 2008 at 10:36:44AM +0200, [email protected] wrote:

From: Christian Ehrhardt <[email protected]>

Hi Christian,

Quoted 18 lines

This adds the guest portion of the hypercall infrastructure, basically an illegal instruction with a defined layout.
See http://kvm.qumranet.com/kvmwiki/PowerPC_Hypercall_ABI for more detail on the hypercall ABI for powerpc.

---

[diffstat]

 kvm_para.h |   16 ++++++++++++++++
 1 file changed, 16 insertions(+)

[diff]

include/asm-powerpc/kvm_para.h line 29

Quoted 5 lines
include/asm-powerpc/kvm_para.h +2 -0
@@ -25,6 +25,8 @@
 #ifdef __KERNEL__
 
 #include <linux/of.h>
+
+#define KVM_HYPERCALL_BIN 0x03ffffff

Ummm didn't you add this in patch 2 of 6?

Yours Tony

linux.conf.au http://www.marchsouth.org/
Jan 19 - 24 2009 The Australian Linux Technical Conference!

#9 lore raw

Re: [PATCH 3/6] kvmppc: add hypercall infrastructure - guest part

Quoted 26 lines · Tony Breeds
Quoted 2 lines · On Wed, Jul 23, 2008 at 10:36:44AM +0200, [email protected] wrote:

From: Christian Ehrhardt <[email protected]>

Hi Christian,

Quoted 19 lines

This adds the guest portion of the hypercall infrastructure, basically an illegal instruction with a defined layout.
See http://kvm.qumranet.com/kvmwiki/PowerPC_Hypercall_ABI for more detail on the hypercall ABI for powerpc.

---

[diffstat]

 kvm_para.h |   16 ++++++++++++++++
 1 file changed, 16 insertions(+)

[diff]

include/asm-powerpc/kvm_para.h +0 -0
@@ -25,6 +25,8 @@
 #ifdef __KERNEL__
 
Quoted 7 lines

include/asm-powerpc/kvm_para.h line 30

Quoted 4 lines
include/asm-powerpc/kvm_para.h +2 -0
 #include <linux/of.h>
+
+#define KVM_HYPERCALL_BIN 0x03ffffff
 

Ummm didn't you add this in patch 2 of 6?

This is just because I initially wanted to split Host & Guest patch series. I need to separate my patches a bit more anyway for the next submission thanks for pointing out this duplication.

Quoted 6 lines

Yours Tony

linux.conf.au http://www.marchsouth.org/
Jan 19 - 24 2009 The Australian Linux Technical Conference!

Signature · 8 lines
Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#10 lore raw

PATCH 4/6 kvmppc: magic page hypercall - host part

In reply to [email protected]

From: Christian Ehrhardt <[email protected]>

This adds the host part of the magic page registration. This is a memory area of the guest granted to the host.
The patch just introduces the infrastruture to receive the guest paddr. This is work in progress and it is intended to later on use this memory as storage area a guest can read unprivileged (using binary rewriting to change privileges instructions).

---

[diffstat]

 arch/powerpc/kvm/emulate.c     |    5 +++++
 arch/powerpc/kvm/powerpc.c     |   18 +++++++++++++++++-
 include/asm-powerpc/kvm_para.h |    2 ++
 include/linux/kvm.h            |    6 ++++++
 4 files changed, 30 insertions(+), 1 deletion(-)

[diff]

arch/powerpc/kvm/emulate.c +5 -0
@@ -208,6 +208,11 @@
 	int ret = 0;
 
 	switch (vcpu->arch.gpr[0]) {
+	case KVM_HCALL_RESERVE_MAGICPAGE:
+		/* FIXME TODO implement the real fuctionality using that */
+		printk(KERN_ERR"%s - receive magicpage address 0x%x\n",
+			__func__, vcpu->arch.gpr[3]);
+		break;
 	default:
 		printk(KERN_ERR"unknown hypercall %d\n", vcpu->arch.gpr[0]);
 		kvmppc_dump_vcpu(vcpu);
arch/powerpc/kvm/powerpc.c +17 -1
@@ -148,6 +148,9 @@
 	case KVM_CAP_COALESCED_MMIO:
 		r = KVM_COALESCED_MMIO_PAGE_OFFSET;
 		break;
+	case KVM_CAP_PPCPV_MAGICPAGE:
+		r = 1;
+		break;
 	default:
 		r = 0;
 		break;
@@ -159,7 +162,20 @@
 long kvm_arch_dev_ioctl(struct file *filp,
                         unsigned int ioctl, unsigned long arg)
 {
-	return -EINVAL;
+	long r = -EINVAL;
+
+	switch (ioctl) {
+	case KVM_GET_PPCPV_MAGICPAGE_SIZE:
+		r = -EINVAL;
+		if (arg)
+			goto out;
+		r = 1024;
+		break;
+	default:
+		r = -EINVAL;
+	}
+out:
+	return r;
 }
 
 int kvm_arch_set_memory_region(struct kvm *kvm,
include/asm-powerpc/kvm_para.h +2 -0
@@ -24,6 +24,8 @@
 
 #define KVM_HYPERCALL_BIN 0x03ffffff
 
+#define KVM_HCALL_RESERVE_MAGICPAGE	0
+
 static inline int kvm_para_available(void)
 {
 	return 0;
include/linux/kvm.h +6 -0
@@ -365,6 +365,11 @@
 #define KVM_TRACE_PAUSE           _IO(KVMIO,  0x07)
 #define KVM_TRACE_DISABLE         _IO(KVMIO,  0x08)
 /*
+ * ioctls for powerpc paravirtualization extensions
+ */
+#define KVM_GET_PPCPV_MAGICPAGE_SIZE       _IO(KVMIO,   0x09)
+
+/*
  * Extension capability list.
  */
 #define KVM_CAP_IRQCHIP	  0
@@ -382,6 +387,7 @@
 #define KVM_CAP_PV_MMU 13
 #define KVM_CAP_MP_STATE 14
 #define KVM_CAP_COALESCED_MMIO 15
+#define KVM_CAP_PPCPV_MAGICPAGE 16
 
 /*
  * ioctls for VM fds
#11 lore raw

Re: [PATCH 4/6] kvmppc: magic page hypercall - host part

On Wed, Jul 23, 2008 at 10:36:45AM +0200, [email protected] wrote:

Hi Christian,

arch/powerpc/kvm/powerpc.c line 169

Quoted 9 lines

long kvm_arch_dev_ioctl(struct file *filp,
unsigned int ioctl, unsigned long arg)
{

arch/powerpc/kvm/powerpc.c +5 -1
-	return -EINVAL;
+	long r = -EINVAL;
+
+	switch (ioctl) {
+	case KVM_GET_PPCPV_MAGICPAGE_SIZE:
+		r = -EINVAL;

Not needed you set it on the declaration.

arch/powerpc/kvm/powerpc.c line 172

Quoted 3 lines
arch/powerpc/kvm/powerpc.c +3 -0
+		if (arg)
+			goto out;
+		r = 1024;

Ummm what does 1024 represent? can it me #defined? or at least add a comment.

arch/powerpc/kvm/powerpc.c line 175

Quoted 3 lines
arch/powerpc/kvm/powerpc.c +3 -0
+		break;
+	default:
+		r = -EINVAL;

Not needed you set it on the declaration.

arch/powerpc/kvm/powerpc.c line 179

Quoted 4 lines
arch/powerpc/kvm/powerpc.c +3 -0
+	}
+out:
+	return r;
 }

Yours Tony

linux.conf.au http://www.marchsouth.org/
Jan 19 - 24 2009 The Australian Linux Technical Conference!

#12 lore raw

PATCH 5/6 kvmppc: magic page paravirtualization - guest part

In reply to [email protected]

From: Christian Ehrhardt <[email protected]>

This patch adds the guest handling for the magic page mechanism. A Hypervisor can modify the device tree passed to the guest. Using that already existing interface a guest can simply detect available hypervisor features and agree on the supported ones using hypercalls.
In this example it is checked for the feature switch "feature,pv-magicpage" in the hypervisor node and additional data which represents the size the hypervisor requests in "data,pv-magicpage-size".
When the guest read that data and wants to support it the memory is allocated and passed to the hypervisor using the KVM_HCALL_RESERVE_MAGICPAGE hypercall.

---

[diffstat]

 arch/powerpc/kernel/kvm.c      |   48 +++++++++++++++++++++++++++++++++++++++++
 include/asm-powerpc/kvm_para.h |   27 ++++++++++++++++++++++-
 2 files changed, 74 insertions(+), 1 deletion(-)

[diff]

arch/powerpc/kernel/kvm.c +48 -0
@@ -22,9 +22,57 @@
 #include <linux/percpu.h>
 #include <linux/mm.h>
 #include <linux/kvm_para.h>
+#include <linux/bootmem.h>
+
+/*
+ * this is guest memory granted to the hypervisor;
+ * the hypervisor can place data in this area and rewrite
+ * privileged instructions to read from this area without
+ * trapping.
+ * Only the Hypervisor needs to be aware of the structure layout
+ * which makes the guest more felxible - the guest only guarantees
+ * the size which is requested by the hypervisor and read from a
+ * device tree entry.
+ */
+void *kvm_magicpage;
+
+static void __init kvmppc_register_magic_page(void)
+{
+	unsigned long paddr;
+	int size;
+	long err;
+
+	size = kvmppc_pv_read_data(KVM_PVDATA_MAGICPAGE_SIZE);
+	if (size < 0) {
+		printk(KERN_ERR"%s: couldn't read size for kvmppc style "
+			"paravirtualization support (got %d)\n",
+			__func__, size);
+		return;
+	}
+
+	/* FIXME Guest SMP needs that percpu
+	 * On SMP we might also need a free implementation */
+	kvm_magicpage = alloc_bootmem(size);
+	if (!kvm_magicpage) {
+		printk(KERN_ERR"%s - failed to allocate %d bytes\n",
+			 __func__, size);
+		return;
+	}
+
+	paddr = (unsigned long)__pa(kvm_magicpage);
+	err = kvm_hypercall1(KVM_HCALL_RESERVE_MAGICPAGE, paddr);
+	if (err)
+		printk(KERN_ERR"%s: couldn't register magic page\n", __func__);
+	else
+		printk(KERN_NOTICE"%s: registered %d bytes for "
+			"virtualization support\n", __func__, size);
+}
 
 void __init kvm_guest_init(void)
 {
 	if (!kvm_para_available())
 		return;
+
+	if (kvm_para_has_feature(KVM_FEATURE_PPCPV_MAGICPAGE))
+		kvmppc_register_magic_page();
 }
include/asm-powerpc/kvm_para.h +26 -1
@@ -28,10 +28,18 @@
 
 #define KVM_HYPERCALL_BIN 0x03ffffff
 
+#define KVM_HCALL_RESERVE_MAGICPAGE	0
+
+#define KVM_PVDATA_MAGICPAGE_SIZE	"data,pv-magicpage-size"
+
+/* List of PV features supported, returned as a bitfield */
+#define KVM_FEATURE_PPCPV_MAGICPAGE	0
+
 static struct kvmppc_para_features {
 	char *dtcell;
 	int feature;
 } para_features[] = {
+	{ "feature,pv-magicpage", KVM_FEATURE_PPCPV_MAGICPAGE }
 };
 
 static inline int kvm_para_available(void)
@@ -54,13 +62,30 @@
 	if (!dn)
 		return 0;
 
-	for (i = 0; i < ARRAY_SIZE(para_features)-1; i++) {
+	for (i = 0; i < ARRAY_SIZE(para_features); i++) {
 		dtval = of_get_property(dn, para_features[i].dtcell, NULL);
 		if (dtval && *dtval = 1)
 			features |= (1 << para_features[i].feature);
 	}
 
 	return features;
+}
+
+/* reads the specified data field out of the hypervisor node */
+static inline int kvmppc_pv_read_data(char *dtcell)
+{
+	struct device_node *dn;
+	const int *dtval;
+
+	dn = of_find_node_by_path("/hypervisor");
+	if (!dn)
+		return -EINVAL;
+
+	dtval = of_get_property(dn, dtcell, NULL);
+	if (dtval)
+		return *dtval;
+	else
+		return -EINVAL;
 }
 
 void kvm_guest_init(void);
#13 lore raw

Re: [PATCH 5/6] kvmppc: magic page paravirtualization - guest part

On Wed, Jul 23, 2008 at 10:36:46AM +0200, [email protected] wrote:

Hi Christian,

<snip>

arch/powerpc/kernel/kvm.c line 37

Quoted 11 lines
arch/powerpc/kernel/kvm.c +11 -0
+/*
+ * this is guest memory granted to the hypervisor;
+ * the hypervisor can place data in this area and rewrite
+ * privileged instructions to read from this area without
+ * trapping.
+ * Only the Hypervisor needs to be aware of the structure layout
+ * which makes the guest more felxible - the guest only guarantees
+ * the size which is requested by the hypervisor and read from a
+ * device tree entry.
+ */
+void *kvm_magicpage;

static?

<snip>

Quoted 5 lines
include/asm-powerpc/kvm_para.h +5 -0
+/* reads the specified data field out of the hypervisor node */
+static inline int kvmppc_pv_read_data(char *dtcell)
+{
+	struct device_node *dn;
+	const int *dtval;

include/asm-powerpc/kvm_para.h line 88

Quoted 10 lines
include/asm-powerpc/kvm_para.h +10 -0
+
+	dn = of_find_node_by_path("/hypervisor");
+	if (!dn)
+		return -EINVAL;
+
+	dtval = of_get_property(dn, dtcell, NULL);
+	if (dtval)
+		return *dtval;
+	else
+		return -EINVAL;

You need an of_node_put(dn) in this function somewhere.

Yours Tony

linux.conf.au http://www.marchsouth.org/
Jan 19 - 24 2009 The Australian Linux Technical Conference!

#14 lore raw

PATCH 6/6 kvmppc: kvm-userspace: device tree modification for magicpage

In reply to [email protected]

From: Christian Ehrhardt <[email protected]>

This patch to kvm-userspace connects the other host & guest patches in this series. On guest initialization it checks the hosts capabilities for the magicpage mechanism. If available the device tree passed to the guest gets the "hypervisor" node added and in that node the feature flag and the requested magic page size (read from the host kernel via an ioctl) is stored.

---

[diffstat]

 libkvm/libkvm-powerpc.c |    6 ++++++
 libkvm/libkvm.h         |    6 ++++++
 qemu/hw/device_tree.c   |   10 ++++++++++
 qemu/hw/device_tree.h   |    1 +
 qemu/hw/ppc440_bamboo.c |   15 +++++++++++++++
 qemu/qemu-kvm-powerpc.c |    5 +++++
 qemu/qemu-kvm.h         |    1 +
 7 files changed, 44 insertions(+)

[diff]

libkvm/libkvm-powerpc.c +6 -0
@@ -19,6 +19,7 @@
 
 #include "libkvm.h"
 #include "kvm-powerpc.h"
+#include <sys/ioctl.h>
 #include <errno.h>
 #include <stdio.h>
 #include <inttypes.h>
@@ -105,6 +106,11 @@
 	return 0;
 }
 
+int kvm_get_magicpage_size(kvm_context_t kvm)
+{
+	return ioctl(kvm->fd, KVM_GET_PPCPV_MAGICPAGE_SIZE, 0);
+}
+
 int kvm_arch_run(struct kvm_run *run, kvm_context_t kvm, int vcpu)
 {
 	int ret = 0;
libkvm/libkvm.h +6 -0
@@ -639,6 +639,12 @@
 
 #endif
 
+#ifdef KVM_CAP_PPCPV_MAGICPAGE
+
+int kvm_get_magicpage_size(kvm_context_t kvm);
+
+#endif
+
 int kvm_translate(kvm_context_t kvm, int vcpu, struct kvm_translation *tr);
 
 #endif
qemu/hw/device_tree.c +10 -0
@@ -190,4 +190,14 @@
 		exit(1);
 	}
 }
+
+void dt_add_subnode(void *fdt, const char *name, char *node_path)
+{
+	int offset;
+	offset = get_offset_of_node(fdt, node_path);
+	if (fdt_add_subnode(fdt, offset, name) < 0) {
+		printf("Unable to create device tree node '%s'\n", name);
+		exit(1);
+	}
+}
 #endif
qemu/hw/device_tree.h +1 -0
@@ -23,4 +23,5 @@
 		uint32_t *val_array, int size);
 void dt_string(void *fdt, char *node_path, char *property,
 		char *string);
+void dt_add_subnode(void *fdt, const char *name, char *node_path);
 #endif
qemu/hw/ppc440_bamboo.c +15 -0
@@ -51,6 +51,7 @@
 	uint32_t cpu_freq;
 	uint32_t timebase_freq;
 	uint32_t mem_reg_property[]={0, 0, ram_size};
+	int pv_magicpage_size;
 
 	printf("%s: START\n", __func__);
 
@@ -167,6 +168,20 @@
 	dt_cell(fdt, "/chosen", "linux,initrd-end",
 				(initrd_base + initrd_size));
 	dt_string(fdt, "/chosen", "bootargs", (char *)kernel_cmdline);
+
+	if (kvm_enabled()
+	    && kvm_qemu_check_extension(KVM_CAP_PPCPV_MAGICPAGE)) {
+		pv_magicpage_size = kvmppc_pv_get_magicpage_size();
+		if (pv_magicpage_size < 0) {
+			fprintf(stderr, "%s: error reading magic page size\n",
+				 __func__);
+			exit(1);
+		}
+		dt_add_subnode(fdt, "hypervisor", "/");
+		dt_cell(fdt, "/hypervisor", "feature,pv-magicpage", 1);
+		dt_cell(fdt, "/hypervisor", "data,pv-magicpage-size",
+			pv_magicpage_size);
+	}
 #endif
 
 	if (kvm_enabled()) {
qemu/qemu-kvm-powerpc.c +5 -0
@@ -214,6 +214,11 @@
     return 0; /* XXX ignore failed DCR ops */
 }
 
+int kvmppc_pv_get_magicpage_size(void)
+{
+	return kvm_get_magicpage_size(kvm_context);
+}
+
 int mmukvm_get_physical_address(CPUState *env, mmu_ctx_t *ctx,
                                 target_ulong eaddr, int rw, int access_type)
 {
qemu/qemu-kvm.h +1 -0
@@ -86,6 +86,7 @@
 #ifdef TARGET_PPC
 int handle_powerpc_dcr_read(int vcpu, uint32_t dcrn, uint32_t *data);
 int handle_powerpc_dcr_write(int vcpu,uint32_t dcrn, uint32_t data);
+int kvmppc_pv_get_magicpage_size();
 #endif
 
 #if !defined(SYS_signalfd)
#15 lore raw
In reply to [email protected]
Quoted 10 lines · On Wed, Jul 23, 2008 at 10:36:41AM +0200, [email protected] wrote:

From: Christian Ehrhardt <[email protected]>

This patch series implements a paravirtualization interface using:
- the device tree mechanism to pass hypervisor informations to the guest
- hypercalls for guest->host calls
- an example exploiter of that interface (magic page)
This is work in progress, but working so far. I just start to really exploit the fuctionality behind the magic page mechanism therefor I can't provide any performance improvements so far, but it is evolved enough for RFC and to start the standardization discussion.

Are you aiming this for the current merge window, ie for 2.6.27?

Yours Tony

linux.conf.au http://www.marchsouth.org/
Jan 19 - 24 2009 The Australian Linux Technical Conference!

#16 lore raw
Quoted 14 lines · Tony Breeds
Quoted 11 lines · On Wed, Jul 23, 2008 at 10:36:41AM +0200, [email protected] wrote:

From: Christian Ehrhardt <[email protected]>

This patch series implements a paravirtualization interface using:
- the device tree mechanism to pass hypervisor informations to the guest
- hypercalls for guest->host calls
- an example exploiter of that interface (magic page)
This is work in progress, but working so far. I just start to really exploit the fuctionality behind the magic page mechanism therefor I can't provide any performance improvements so far, but it is evolved enough for RFC and to start the standardization discussion.

Quoted 2 lines

Are you aiming this for the current merge window, ie for 2.6.27?

The aim is not really fixed. It would be nice to get into 2.6.27, but since I can't yet expect how long it takes ...

Actually the guest patches would already go through reviews and upstream, due to the fact that the guest code changes are not that (the major part of the implementation will go over kvmppc -> kvm upstream). But since I want to discuss about the standardization on the embedded hypervisor list first, the naming of the device tree entries are not fixed yet.
Therefor I can't yet define which kernel version merge window I'll target/reach.

btw - embedded hypervisor - I got advised that this is a closed list which I forgot.
Sorry for all who got bounces on a replay-all action. The next version of the patch series will go to the involved open source lists only and a separate more standardization than patch style mail series to embedded hypervisor.

Quoted 5 lines

Yours Tony

linux.conf.au http://www.marchsouth.org/
Jan 19 - 24 2009 The Australian Linux Technical Conference!

Signature · 8 lines
Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#17 lore raw
Quoted 2 lines · Christian Ehrhardt

The aim is not really fixed. It would be nice to get into 2.6.27, but since I can't yet expect how long it takes ...

Ahh okay, Id say given that we're 2/3rds through the merge window then 2.8.28 is a safer target.

Quoted 6 lines

Actually the guest patches would already go through reviews and upstream, due to the fact that the guest code changes are not that (the major part of the implementation will go over kvmppc -> kvm upstream). But since I want to discuss about the standardization on the embedded hypervisor list first, the naming of the device tree entries are not fixed yet.

Okay. When that's done we can take another look at them.

Quoted 5 lines

Therefor I can't yet define which kernel version merge window I'll target/reach.

btw - embedded hypervisor - I got advised that this is a closed list which I forgot.

... snipped from this reply

Yours Tony

linux.conf.au http://www.marchsouth.org/
Jan 19 - 24 2009 The Australian Linux Technical Conference!

#18 lore raw

cover kvmppc: paravirtualization interface - host part v2

In reply to [email protected]

From: Christian Ehrhardt <[email protected]>

This patch series implements the host part of an paravirtualization interface using:
- the device tree mechanism to pass hypervisor informations to the guest (kvm-userspace)
- hypercalls backend for guest->host calls
- an example exploiter of that interface (magic page) that uses that for binary rewriting saving guest exits by avoiding privileged instructions

Version 2 contains the feedback to my last submission and implemented an exploitation of that interface using the magic page mechanism to rewrite guest code.

The used hypercall ABI was already discussed on the embedded-hypervisor mailing list and is available at http://kvm.qumranet.com/kvmwiki/PowerPC_Hypercall_ABI

The device tree format used here (ºse for the discussions on embedded-hypervisor) is the following.
- A node "hypervisor" to show the general availability of some hypervisor data
- flags for features like the example "feature,pv-magicpage" setting 1 = available, everything else = unavailable
- Some features might need to pass more data and can use an entry in the device tree like the example of "data,pv-magicpage-size"

If the guest wants that pv support it has to allocate the requested size of memory (aligned to the tlb entry size rounded up => e.g. 4000 bytes 4k aligned, 4096 bytes 4k aligned, 4097 bytes 16k aligned and so on).

The speedup with both patch queues applied is around 40% dependent on what the guest is doing (measured with boot times and some simple tasks). The raw saving of exits about ~50-60% as you can see in the kvm stat exit counters.

[patches in series]
[PATCH 1/6] kvmppc: add hypercall infrastructure - host part [PATCH 2/6] kvmppc: magic page hypercall - host part
[PATCH 3/6] kvmppc: rewrite guest code - sprg0-3
[PATCH 4/6] kvmppc: rewrite guest code - dear, esr, srr0, srr1 [PATCH 5/6] kvmppc: rewrite guest code - wrtee
[PATCH 6/6] kvmppc: kvm-userspace: device tree modification for magic page

---
[diffstat]

 arch/powerpc/kvm/booke_guest.c   |   78 ++++++++++---
 arch/powerpc/kvm/emulate.c       |  233 +++++++++++++++++++++++++++++++++++++--
 arch/powerpc/kvm/powerpc.c       |    1
 b/arch/powerpc/kvm/booke_guest.c |   14 ++
 b/arch/powerpc/kvm/emulate.c     |   28 ++++
 b/arch/powerpc/kvm/powerpc.c     |   19 +++
 b/include/asm-powerpc/kvm_host.h |    6 +
 b/include/asm-powerpc/kvm_para.h |    2
 b/include/asm-powerpc/kvm_ppc.h  |    5
 b/include/linux/kvm.h            |    5
 b/libkvm/libkvm-powerpc.c        |    6 +
 b/libkvm/libkvm.h                |    4
 b/qemu/hw/device_tree.c          |   10 +
 b/qemu/hw/device_tree.h          |    1
 b/qemu/hw/ppc440_bamboo.c        |   18 +++
 b/qemu/qemu-kvm-powerpc.c        |    5
 b/qemu/qemu-kvm.h                |    1
 include/asm-powerpc/kvm_para.h   |   56 +++++++++
 18 files changed, 466 insertions(+), 26 deletions(-)
#19 lore raw

cover kvmppc: paravirtualization interface - host part v3

In reply to [email protected]

From: Christian Ehrhardt <[email protected]>

Version 3 updates:
- fixed adress release at a kunmap_atomic call
- removed style issues (whitespace/indent)
- all checks to test if pvmem is currently available now use the wrappers kvmppc_has_pvmem or kvmppc_is_pvmem
- folded some changes in a better order to prevent multiple trivial patches for the same code sections
- the hypercall ABI is now implemented in beat style
- the rewrite of wrtee is dropped, instead mfmsr is rewritten. This saves more exits on non cooperating guests and for now rmeoving the need for guest coop patches (less invasive).
- separated and removed some changes unrelated to paravirtualization to streamline this series
- changed mapping of guest memory to call gfn_to_page without mm locks held (upstream code just changed)
- instruction rewriting is now done after emulation guarded by an unlikely() which should save a lot of runtime complexity (adding some extra code though):
a) in non pv case this is now just an if (unlikely(x)) that is never true
b) in pv case the rewrite function does not need to be called for every emulation - only for those that are rewritable (which additionally don't occur anymore after being rewritten once).

This patch series implements the host part of an paravirtualization interface using:
- the device tree mechanism to pass hypervisor informations to the guest (kvm-userspace)
- hypercalls backend for guest->host calls
- an example exploiter of that interface (magic page) that uses that for binary rewriting saving guest exits by avoiding privileged instructions

The device tree format used here (ºse for the discussions on embedded-hypervisor) is the following.
- A node "hypervisor" to show the general availability of some hypervisor data
- flags for features like the example "feature,pv-magicpage" setting 1 = available, everything else = unavailable
- Some features might need to pass more data and can use an entry in the device tree like the example of "data,pv-magicpage-size"

If the guest wants that pv support it has to allocate the requested size of memory (aligned to the tlb entry size rounded up => e.g. 4000 bytes 4k aligned, 4096 bytes 4k aligned, 4097 bytes 16k aligned and so on).

The speedup with both patch queues applied is around 40% dependent on what the guest is doing (measured with boot times and some simple tasks). The raw saving of exits about ~50-60% as you can see in the kvm stat exit counters.

[patches in series]
[PATCH 1/6] kvmppc: add hypercall infrastructure - host part [PATCH 2/6] kvmppc: magic page hypercall - host part
[PATCH 3/6] kvmppc: rewrite guest code - sprg0-3
[PATCH 4/6] kvmppc: rewrite guest code - dear, esr, srr0, srr1 [PATCH 5/6] kvmppc: rewrite guest code - mfmsr
[PATCH 6/6] kvmppc: kvm-userspace: device tree modification for magic page

---
[diffstat]
kernel:

 arch/powerpc/kvm/booke_guest.c        |   70 ++++-
 arch/powerpc/kvm/emulate.c            |  415 ++++++++++++++++++++++++++++++++--
 b/arch/powerpc/kvm/booke_guest.c      |   11
 b/arch/powerpc/kvm/booke_interrupts.S |    9
 b/arch/powerpc/kvm/emulate.c          |   16 +
 b/arch/powerpc/kvm/powerpc.c          |   20 +
 b/include/asm-powerpc/kvm_host.h      |    1
 b/include/asm-powerpc/kvm_para.h      |    4
 b/include/asm-powerpc/kvm_ppc.h       |    3
 b/include/linux/kvm.h                 |    5
 include/asm-powerpc/kvm_host.h        |    5
 include/asm-powerpc/kvm_para.h        |   54 ++++
kvm-userspace:
 b/libkvm/libkvm-powerpc.c             |    6
 b/libkvm/libkvm.h                     |    4
 b/qemu/hw/device_tree.c               |   10
 b/qemu/hw/device_tree.h               |    1
 b/qemu/hw/ppc440_bamboo.c             |   18 +
 b/qemu/qemu-kvm-powerpc.c             |    5
 b/qemu/qemu-kvm.h                     |    1
 19 files changed, 625 insertions(+), 33 deletions(-)