nVMX: Nested VMX, v7

Hi,

This is the seventh iteration of the nested VMX patch set. It fixes a bunch of bugs in the previous iteration, and in particular it now works correctly with EPT in the L0 hypervisor, so "ept=0" no longer needs to be specified.

This new set of patches should apply to the current KVM trunk (I checked with 66fc6be8d2b04153b753182610f919faf9c705bc). In particular it uses the recently added is_guest_mode() function (common to both nested svm and vmx) instead of inventing our own flag.

About nested VMX:
-----------------

The following 28 patches implement nested VMX support. This feature enables a guest to use the VMX APIs in order to run its own nested guests. In other words, it allows running hypervisors (that use VMX) under KVM. Multiple guest hypervisors can be run concurrently, and each of those can in turn host multiple guests.

The theory behind this work, our implementation, and its performance characteristics were presented in OSDI 2010 (the USENIX Symposium on Operating Systems Design and Implementation). Our paper was titled "The Turtles Project: Design and Implementation of Nested Virtualization", and was awarded "Jay Lepreau Best Paper". The paper is available online, at:

http://www.usenix.org/events/osdi10/tech/full_papers/Ben-Yehuda.pdf

This patch set does not include all the features described in the paper. In particular, this patch set is missing nested EPT (shadow page tables are used in L1, while L0 can use shadow page tables or EPT). It is also missing some features required to run VMWare Server as a guest. These missing features will be sent as follow-on patchs.

Running nested VMX:
------------------

The current patches have a number of requirements, which will be relaxed in follow-on patches:

1. This version was only tested with KVM (64-bit) as a guest hypervisor, and Linux as a nested guest.

2. SMP is supported in the code, but is unfortunately buggy in this version and often leads to hangs. Use the "nosmp" option in the L0 (topmost) kernel to avoid this bug (and to reduce your performance ;-))..

3. No modifications are required to user space (qemu). However, qemu does not currently list "VMX" as a CPU feature in its emulated CPUs (even when they are named after CPUs that do normally have VMX). Therefore, the "-cpu host" option should be given to qemu, to tell it to support CPU features which exist in the host - and in particular VMX.
This requirement can be made unnecessary by a trivial patch to qemu (which I will submit in the future).

4. The nested VMX feature is currently disabled by default. It must be explicitly enabled with the "nested=1" option to the kvm-intel module.

5. Nested VPID is not properly supported in this version. You must give the "vpid=0" module options to kvm-intel to turn this feature off.

Patch statistics:
-----------------

 Documentation/kvm/nested-vmx.txt |  237 ++
 arch/x86/include/asm/kvm_host.h  |    2 
 arch/x86/include/asm/vmx.h       |   31 
 arch/x86/kvm/svm.c               |    6 
 arch/x86/kvm/vmx.c               | 2416 ++++++++++++++++++++++++++++-
 arch/x86/kvm/x86.c               |   16 
 arch/x86/kvm/x86.h               |    6 
 7 files changed, 2676 insertions(+), 38 deletions(-)

--
Nadav Har'El
IBM Haifa Research Lab