KVM: prepare user interface for smp guests

While the current implementation of KVM only supports UP guests, it should be quite easy to add SMP guest support in the future. However, I'd like to get the userspace interface right today.

The current interface is specified as a bunch of ioctls on a /dev/kvm chardev. Some of the ioctls operate on the entire virtual machine (creating memory, for example), while others operate on a specific virtual cpu (vcpu) (running a vcpu or accessing its registers).

A potential problem with the this is that the cacheline containing filp->f_count will ping-pong when different vcpus are accessed (assuming each vcpu runs on a distinct cpu). While with today's hardware entering and exiting guest mode will likely dominate over this, the hardware will improve, core counts will rise, and a 64-way guest running on a 4-socket VT-9 implementation will likely have a lot of cacheline bouncing.

To address this, we need a filp for every vcpu. Two approches have been suggested:

1. Have the ioctl() that creates a vcpu return a new fd

2. Have a pseudo filesystem to represent the virtual machine, a syscall to return an inode in that fs, and a directory or file for each vcpu.

(There is also a third approach -- do nothing -- but since this is a holy userspace interface, and because the changes clean up the interface somewhat, I'd like to do them)

I've tried to summarize the differences between the two approaches:

- fs requires a new syscall, and is therefore more intrusive, especially for such a specialized driver.
- ioctl messes up too much in vfs internals for its own good (it wants to export cdev_get(). in the patch below I duplicated it because I was lazy).
- fs is similar to spufs, while ioctl returning an fd is probably a (bad) precedent.
- ioctl is less code.

Patch implementing the ioctl approach below. Please do comment.

Index: linux-2.6/drivers/kvm/kvm.h
===================================================================
@@ -6,6 +6,7 @@
 #include <linux/mutex.h>
 #include <linux/spinlock.h>
 #include <linux/mm.h>
+#include <asm/atomic.h>
 
 #include "vmx.h"
 

@@ -208,6 +209,7 @@ struct kvm_memory_slot {
 
 struct kvm {
 	spinlock_t lock; /* protects everything except vcpus */
+	atomic_t count;
 	int nmemslots;
 	struct kvm_memory_slot memslots[KVM_MEMORY_SLOTS];
 	struct list_head active_mmu_pages;

Index: linux-2.6/include/linux/kvm.h
===================================================================
@@ -1,13 +1,6 @@
 #ifndef __LINUX_KVM_H
 #define __LINUX_KVM_H
 
-/*
- * Userspace interface for /dev/kvm - kernel based virtual machine
- *
- * Note: this interface is considered experimental and may change without
- *       notice.
- */
-
 #include <asm/types.h>
 #include <linux/ioctl.h>
 

@@ -88,10 +81,6 @@ struct kvm_run {
 
 /* for KVM_GET_REGS and KVM_SET_REGS */
 struct kvm_regs {
-	/* in */
-	__u32 vcpu;
-	__u32 padding;
-
 	/* out (KVM_GET_REGS) / in (KVM_SET_REGS) */
 	__u64 rax, rbx, rcx, rdx;
 	__u64 rsi, rdi, rsp, rbp;

@@ -118,10 +107,6 @@ struct kvm_dtable {
 
 /* for KVM_GET_SREGS and KVM_SET_SREGS */
 struct kvm_sregs {
-	/* in */
-	__u32 vcpu;
-	__u32 padding;
-
 	/* out (KVM_GET_SREGS) / in (KVM_SET_SREGS) */
 	struct kvm_segment cs, ds, es, fs, gs, ss;
 	struct kvm_segment tr, ldt;

@@ -152,7 +137,6 @@ struct kvm_translation {
 /* for KVM_INTERRUPT */
 struct kvm_interrupt {
 	/* in */
-	__u32 vcpu;
 	__u32 irq;
 };
 

@@ -165,8 +149,8 @@ struct kvm_breakpoint {
 /* for KVM_DEBUG_GUEST */
 struct kvm_debug_guest {
 	/* int */
-	__u32 vcpu;
 	__u32 enabled;
+	__u32 padding;
 	struct kvm_breakpoint breakpoints[4];
 	__u32 singlestep;
 };

@@ -183,6 +167,11 @@ struct kvm_dirty_log {
 
 #define KVMIO 0xAE
 
+/*
+ * Vcpu-specific operations
+ *
+ * Use with file descriptors returned from ioctl(KVM_CREATE_VCPU)
+ */
 #define KVM_RUN                   _IOWR(KVMIO, 2, struct kvm_run)
 #define KVM_GET_REGS              _IOWR(KVMIO, 3, struct kvm_regs)
 #define KVM_SET_REGS              _IOW(KVMIO, 4, struct kvm_regs)

@@ -191,6 +180,12 @@ struct kvm_dirty_log {
 #define KVM_TRANSLATE             _IOWR(KVMIO, 7, struct kvm_translation)
 #define KVM_INTERRUPT             _IOW(KVMIO, 8, struct kvm_interrupt)
 #define KVM_DEBUG_GUEST           _IOW(KVMIO, 9, struct kvm_debug_guest)
+
+/*
+ * Guest-wide operations
+ *
+ * Use with file descriptors returned from open("/dev/kvm")
+ */
 #define KVM_SET_MEMORY_REGION     _IOW(KVMIO, 10, struct kvm_memory_region)
 #define KVM_CREATE_VCPU           _IOW(KVMIO, 11, int /* vcpu_slot */)
 #define KVM_GET_DIRTY_LOG         _IOW(KVMIO, 12, struct kvm_dirty_log)

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Separating the objects into different file descriptors sounds like a good idea, but reusing an open dentry/inode with a new file and different file operations is a rather unusual way to do it. Your concept of allocating a new context on each open is already weird, but there have been other examples of that before.

I'd suggest going to a syscall-based model with your own file system right away, even if you don't use the spufs approach but something in the middle:

* You do a trivial nonmountable new file system with anonymous objects, similar to eventpollfs, and hand out file descriptors to inodes in it, for both the kvm and the vcpu objects.
* You replace the syscall you'd normally use to hand out a new kvm instance with an ioctl on /dev/kvm, and don't allow any other operations on that device.

This would be a much more consistant object model, compared with other generic kernel functionality that is not bound to an actual device. You still have all the flexibility of a loadable module without core kernel changes for the development phase, and can easily switch to real syscalls when merging it into mainline.

I really think that a small number of syscalls is where you should be heading, whether you use a file system or not, but I understand that ioctls are convenient for development.

	Arnd <><

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Yes, it doesn't feel right.

Actually that seemed to me quite natural.

I agree, that sounds like a good plan. I'll look into it.

BTW, what does lsof show for spufs users? I thought lsof /dev/kvm would be a good way to look for virtual machines.

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

It's described in LDD2 and other books, but the traditional view is still that one device node in /dev refers to an actual device or at least something that acts like a device (e.g. /dev/null, dev/tty).

It does what you expect. Since spufs is mounted, 'ls /spu/' shows you the existing spu contexts, 'lsof /spu/*' shows you the tasks using those.

	Arnd <><

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642