nVMX: Nested VMX, v6

Hi,

About three months have passed since my previous (v5) nested VMX patch set, and it can no longer be applied cleanly to the current KVM trunk.

This version of the patches can be applied to the current trunk, and addresses dozens of concerns that have been raised by Avi Kivity, Marcelo Tosatti, Gleb Natapov, and Eddie Dong while reviewing v5.
There are still several outstanding issues (e.g., that IDT handling code that we've been discussing) that are not addressed in this version, but rest assured that I have not forgotten them - I simply want a newer version, and one that works with a current KVM, to be available to potential reviewers or testers.

About nested VMX:
-----------------

The following 27 patches implement nested VMX support. This feature enables a guest to use the VMX APIs in order to run its own nested guests. In other words, it allows running hypervisors (that use VMX) under KVM. Multiple guest hypervisors can be run concurrently, and each of those can in turn host multiple guests.

The theory behind this work, our implementation, and its performance characteristics were presented this month in OSDI (the USENIX Symposium on Operating Systems Design and Implementation). Our paper was titled "The Turtles Project: Design and Implementation of Nested Virtualization", and was awarded "Jay Lepreau Best Paper". The paper is available online, at:

http://www.usenix.org/events/osdi10/tech/full_papers/Ben-Yehuda.pdf

This patch set does not include all the features described in the paper. In particular, this patch set is missing nested EPT (shadow page tables are used) and missing some features required to run VMWare Server as a guest. These missing features will be sent as follow-on patchs.

Running nested VMX:
------------------

The current patches have a number of requirements, which will be relaxed in follow-on patches:

1. This version was only tested with KVM (64-bit) as a guest hypervisor, and Linux as a nested guest.

2. SMP is supported in the code, but is unfortunately buggy in this version and often leads to hangs. Use the "nosmp" option in the L0 (topmost) kernel to avoid this bug (and to reduce your performance ;-))..

3. No modifications are required to user space (qemu). However, qemu does not currently list "VMX" as a CPU feature in its emulated CPUs (even when they are named after CPUs that do normally have VMX). Therefore, the "-cpu host" option should be given to qemu, to tell it to support CPU features which exist in the host - and in particular VMX.
This requirement can be made unnecessary by a trivial patch to qemu (which I will submit in the future).

4. The nested VMX feature is currently disabled by default. It must be explicitly enabled with the "nested=1" option to the kvm-intel module.

5. Nested EPT and VPID are not properly supported in this version. You must give the "ept=0 vpid=0" module options to kvm-intel to turn both features off.

Patch statistics:
-----------------

 Documentation/kvm/nested-vmx.txt |  237 ++
 arch/x86/include/asm/kvm_host.h  |    2 
 arch/x86/include/asm/vmx.h       |   31 
 arch/x86/kvm/svm.c               |    6 
 arch/x86/kvm/vmx.c               | 2396 ++++++++++++++++++++++++++++-
 arch/x86/kvm/x86.c               |   16 
 arch/x86/kvm/x86.h               |    6 
 7 files changed, 2657 insertions(+), 37 deletions(-)

--
Nadav Har'El
IBM Haifa Research Lab