Nested Virtualization: Overview
cover Nested Virtualization: Overview
Hi!
This patch series brings Nested Virtualization to Xen. This is the third patch series. Improvements to the previous patch submission:
- implement HVM-on-HVM (instead of SVM-on-HVM)
- move cpuid handling into tools (per Keir's request)
There might still be some nuances to fiddle with to make it fit for VMX. Feedback from Intel is appreciated, therefore.
The patch series:
patch 01: add nestedhvm guest config option to the tools.
This is the only one patch touching the tools
patch 02: change local_event_delivery_* to take vcpu argument.
This prevents spurios xen crashes on guest shutdown/destroy with nestedhvm enabled.
patch 03: Add data structures for nested virtualization.
patch 04: add nestedhvm function hooks, described in XenNestedHVM.pdf patch 05: The heart of nested virtualization.
patch 06: Allow switch to paged real mode during vmrun emulation.
Emulate cr0 and cr4 when guest does not intercept them (i.e. Hyper-V/Windows7, KVM)
patch 07: When injecting an exception into nested guest, inject
#VMEXIT into the guest if intercepted
patch 08: Allow guest to enable SVM in EFER
patch 09: Handle interrupts (generic part)
patch 10: SVM specific implementation for nested virtualization patch 11: Handle interrupts (SVM specific)
patch 12: The piece of code that effectively turns nested virtualization on.
Use HVM_PARAM_* this time.
patch 13: Change p2m infrastructure to operate with per-p2m instead
of per-domain. Combined with patch 15, this allows to run nested guest with hap-on-hap.
patch 14: Handle nested pagefault to enable hap-on-hap and handle
nested guest page-table-walks to emulate instructions the guest does not intercept (i.e. WBINVD with Windows 7).
Signature · 6 lines
---to satisfy European Law for business letters: Advanced Micro Devices GmbH Einsteinring 24, 85609 Dornach b. Muenchen Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632
Hi, Chris:
Quoted 7 lines · Christoph Egger
Hi!
This patch series brings Nested Virtualization to Xen. This is the third patch series. Improvements to the previous patch submission:
- implement HVM-on-HVM (instead of SVM-on-HVM)
Given that we don't have consensus on cross architecture nested virtualization support, I am doubting why this is urgent for now. I would prefer we make SVM-on-SVM and VMX-on-VMX work first. After that, if you prove SVM-on-VMX has real performance gain (which I doubt), we can see how to make a much generic effort to accomodate both natively nested virtualization and cross architecture nested virtualization.
Drawing a picture which doesn't have a real usage with massive common code change is a kind of too much load for us now. Xen hvm_function table is a good example. Intel enabled VMX at very beginning of Xen HVM support, and SVM comes later on with a lot of code reuse from VMX side. Then the community and both side work together to make an API wrapper to reuse common code better and accomodate both architecture. I think we have to go similar path to make it work first.
Quoted 5 lines
- move cpuid handling into tools (per Keir's request)
There might still be some nuances to fiddle with to make it fit for VMX. Feedback from Intel is appreciated, therefore.
Thx, Eddie
Quoted 5 lines · Eddie
Hi, Chris:
Quoted 2 lines · Christoph Egger
Hi!
Quoted 8 lines
Quoted 5 lines
This patch series brings Nested Virtualization to Xen. This is the third patch series. Improvements to the previous patch submission:
- implement HVM-on-HVM (instead of SVM-on-HVM)
Given that we don't have consensus on cross architecture nested virtualization support, I am doubting why this is urgent for now.
The reason to be "urgent" is not the time. This is the best way from the software engineering side.
Quoted 4 lines
I would prefer we make SVM-on-SVM and VMX-on-VMX work first. After that, if you prove SVM-on-VMX has real performance gain (which I doubt), we can see how to make a much generic effort to accomodate both natively nested virtualization and cross architecture nested virtualization.
Tim and Keir made clear they don't want to have two implementations after I submitted my patch series the *first* time.
Quoted 7 lines
Drawing a picture which doesn't have a real usage with massive common code change is a kind of too much load for us now. Xen
hvm_functiontable is a good example. Intel enabled VMX at very beginning of Xen HVM support, and SVM comes later on with a lot of code reuse from VMX side. Then the community and both side work together to make an API wrapper to reuse common code better and accomodate both architecture. I think we have to go similar path to make it work first.
Tim remembers on this and said in this mail
http://lists.xensource.com/archives/html/xen-devel/2010-04/msg00812.html
"With HVM it has turned out to be quite a lot, but it's taken years of reshuffling to pull code out into common HVM routines (and we're not there yet)."
If I understand Tim correctly, the way you suggest is a "no". (Added Tim on CC)
Christoph
Quoted 7 lines
Quoted 5 lines
- move cpuid handling into tools (per Keir's request)
There might still be some nuances to fiddle with to make it fit for VMX. Feedback from Intel is appreciated, therefore.
Thx, Eddie
Signature · 6 lines
---to satisfy European Law for business letters: Advanced Micro Devices GmbH Einsteinring 24, 85609 Dornach b. Muenchen Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632
Quoted 8 lines · Christoph Egger
Quoted 4 lines
Quoted 1 line
- implement HVM-on-HVM (instead of SVM-on-HVM)
Given that we don't have consensus on cross architecture nested virtualization support, I am doubting why this is urgent for now.
The reason to be "urgent" is not the time. This is the best way from the software engineering side.
Quoted 7 lines
Quoted 4 lines
I would prefer we make SVM-on-SVM and VMX-on-VMX work first. After that, if you prove SVM-on-VMX has real performance gain (which I doubt), we can see how to make a much generic effort to accomodate both natively nested virtualization and cross architecture nested virtualization.
Tim and Keir made clear they don't want to have two implementations after I submitted my patch series the *first* time.
I think maybe this is an argument over two different things. To be clear we want to support VMX-on-VMX and SVM-on-SVM. I assume this is what Christoph means by HVM-on-HVM: any-like-on-like. In which case there is no disagreement here.
Now, separately there is a debate to be had on how much code can be shared in HVM-on-HVM, given the big differences between VMX and SVM. I would guess that there will be at least things in the area of nested shadow and nested HAP that can be shared, for example. Probably there is other stuff too. The question is to what degree do we pursue that now rather than get divergent stuff in tree and then go after it later. My mind isn't totally made up on that; I don't know about Tim's.
-- Keir
Quoted 9 lines · Keir Fraser
Quoted 2 lines · Christoph Egger
Tim and Keir made clear they don't want to have two implementations after I submitted my patch series the *first* time.
I think maybe this is an argument over two different things. To be clear we want to support VMX-on-VMX and SVM-on-SVM. I assume this is what Christoph means by HVM-on-HVM: any-like-on-like. In which case there is no disagreement here.
Quoted 7 lines
Now, separately there is a debate to be had on how much code can be shared in HVM-on-HVM, given the big differences between VMX and SVM. I would guess that there will be at least things in the area of nested shadow and nested HAP that can be shared, for example. Probably there is other stuff too. The question is to what degree do we pursue that now rather than get divergent stuff in tree and then go after it later. My mind isn't totally made up on that; I don't know about Tim's.
The general tone of Christoph's latest patchset seems about right to me:
the concept of a VCPU being in nested HVM mode or not, the control interface, and the bulk of the interrupt/exception injection logic seem like they should be common from day one, unless there's a particular reason not to. The details of exactly how the nested VMC[BS] is accessed and updated are of course arch-specific.
The two HAP-on-HAP designs are quite different but since all the EPT code is already separate from the other p2m code that's OK. Shadow-on-shadow and shadow-on-HAP ought to just work[tm] without any extra moving parts.
I've no strong feelings about the details of the interface between the common and the arch-specific code, but it seems like with a bit of flexibility on both sides it could be made suitable for everyone.
Cheers,
Tim.
Signature · 3 lines
Tim Deegan <[email protected]> Principal Software Engineer, XenServer Engineering Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)