Nested Virtualization: trap

xen/arch/x86/hvm/hvm.c +40 -0

+int hvm_inject_exception(unsigned int trapnr, int errcode, unsigned long cr2)
+{
+    uint64_t exitcode;
+    bool_t is_intercepted;
+    struct vcpu *v = current;
+    struct nestedhvm *hvm = &VCPU_NESTEDHVM(v);
+
+    if ( !nestedhvm_enabled(v->domain) ) {
+        hvm_funcs.inject_exception(trapnr, errcode, cr2);
+        return 0;
+    }
+
+    if ( nestedhvm_vmentry_emulate(v) ) {
+        hvm_funcs.inject_exception(trapnr, errcode, cr2);
+        return 0;
+    }
+
+    if ( !nestedhvm_vcpu_in_guestmode(v) ) {
+        hvm_funcs.inject_exception(trapnr, errcode, cr2);
+        return 0;
+    }
+
+    exitcode = nestedhvm_exception2exitcode(trapnr);
+    hvm->nh_hostflags.fields.forcevmexit = 1;
+    hvm->nh_forcevmexit.exitcode = exitcode;
+    is_intercepted = hvm_nestedhvm_vm_intercepted_by_guest(v, exitcode);
+    hvm->nh_hostflags.fields.forcevmexit = 0;
+
+    if ( is_intercepted )
+    {
+        enum nestedhvm_vmexits nsret;
+
+        hvm->nh_forcevmexit.exitcode = exitcode;
+        hvm->nh_forcevmexit.exitinfo1 = errcode;
+        hvm->nh_forcevmexit.exitinfo2 = cr2;
+
+        hvm->nh_hostflags.fields.forcevmexit = 1;
+        nsret = nestedhvm_vcpu_vmexit(v, guest_cpu_user_regs(), 0 /* dummy */);
+        hvm->nh_hostflags.fields.forcevmexit = 0;
+

Quoted 46 lines

+int hvm_inject_exception(unsigned int trapnr, int errcode, unsigned long cr2) +{

xen/arch/x86/hvm/hvm.c +36 -0

+    uint64_t exitcode;
+    bool_t is_intercepted;
+    struct vcpu *v = current;
+    struct nestedhvm *hvm = &VCPU_NESTEDHVM(v);
+
+    if ( !nestedhvm_enabled(v->domain) ) {
+        hvm_funcs.inject_exception(trapnr, errcode, cr2);
+        return 0;
+    }
+
+    if ( nestedhvm_vmentry_emulate(v) ) {
+        hvm_funcs.inject_exception(trapnr, errcode, cr2);
+        return 0;
+    }
+
+    if ( !nestedhvm_vcpu_in_guestmode(v) ) {
+        hvm_funcs.inject_exception(trapnr, errcode, cr2);
+        return 0;
+    }
+
+    exitcode = nestedhvm_exception2exitcode(trapnr);
+    hvm->nh_hostflags.fields.forcevmexit = 1;
+    hvm->nh_forcevmexit.exitcode = exitcode;
+    is_intercepted = hvm_nestedhvm_vm_intercepted_by_guest(v, exitcode);
+    hvm->nh_hostflags.fields.forcevmexit = 0;
+
+    if ( is_intercepted )
+    {
+        enum nestedhvm_vmexits nsret;
+
+        hvm->nh_forcevmexit.exitcode = exitcode;
+        hvm->nh_forcevmexit.exitinfo1 = errcode;
+        hvm->nh_forcevmexit.exitinfo2 = cr2;
+
+        hvm->nh_hostflags.fields.forcevmexit = 1;
+        nsret = nestedhvm_vcpu_vmexit(v, guest_cpu_user_regs(), 0 /*

dummy */); + hvm->nh_hostflags.fields.forcevmexit = 0;

xen/arch/x86/hvm/hvm.c +7 -0

+
+        switch (nsret) {
+        case NESTEDHVM_VMEXIT_DONE:
+        case NESTEDHVM_VMEXIT_ERROR: /* L1 guest will crash L2 guest */
+            return 1;
+        case NESTEDHVM_VMEXIT_HOST:
+        case NESTEDHVM_VMEXIT_CONTINUE:

Quoted 4 lines

xen/arch/x86/hvm/hvm.c +3 -0

+        case NESTEDHVM_VMEXIT_FATALERROR:
+        default:
+            gdprintk(XENLOG_ERR, "unexpected nestedhvm error %i\n",

nsret); + return -1;

This new [-1,0,1] return value is ignored by almost all callers. Would it be possible to get rid of the special case for #PF, and go back to returning void?

There is exactly one reason to have them: Intel seems to want "shadow-on-shadow". In this case the page fault handler walks the guests shadow page table. If that fails the page fault handler wants to inject a VMEXIT(#PF) into the guest to let the guest fix its shadow page table. If the guest page walk is successfull the page fault intercept handler wants to inject the page fault exception into the nested guest.

The page fault intercept handler in
SVM (see [PATCH 10/14] Nested Virtualization: svm specific implementation) assumes that the guest intercepts a page fault.
It uses the return value to check if hvm_inject_exception() did what is
expected: Injecting a VMEXIT(#PF), which is the case when the assumption is correct.
The page fault intercept handler calls svm_inject_exception() to inject a page fault into the nested guest.

If you can invalidate this error check reason then yes, I can go back to make hvm_inject_exception() return void.

Christoph

Signature · 6 lines

---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Quoted 7 lines · Christoph Egger

There is exactly one reason to have them: Intel seems to want "shadow-on-shadow". In this case the page fault handler walks the guests shadow page table. If that fails the page fault handler wants to inject a VMEXIT(#PF) into the guest to let the guest fix its shadow page table. If the guest page walk is successfull the page fault intercept handler wants to inject the page fault exception into the nested guest.

OK, I think I'm getting tied up in the naming scheme. Let me try to lay out what I think is going on and you can tell me where I'm going wrong.

If the L0 Xen is using shadow pagetables, then it must be intercepting #PF. We expect the guest to be using shadows (and intercepting #PF) too.

When an actual #PF occurs when L2 is running, we VMEXIT to L0, which walks the pagetables provided by L1. In the interesting case, L0 sees that the L1 pagetables justify the fault. It injects #PF into the VM.

If the L1 guest is using shadows, it intercepts #PF and does its own shadow pagetable tricks (which might involve it injecting #PF into the L2 guest). If it's not, the injected #PF goes straight to the L2.

Quoted 8 lines

The page fault intercept handler in
SVM (see [PATCH 10/14] Nested Virtualization: svm specific implementation) assumes that the guest intercepts a page fault. It uses the return value to check if hvm_inject_exception() did what is
expected: Injecting a VMEXIT(#PF), which is the case when the assumption is correct.
The page fault intercept handler calls svm_inject_exception() to inject a page fault into the nested guest.

The new return code from hvm_inject_exception() is
0: Normal injection into the running L1 or the running L2. 1: VMEXIT from the running L2 to the L1, caused by injecting an
intercepted exception. (This is the expected case in the scenario above).
-1: Any other case.

The logic in svm_vmexit_handler() when the L0 shadow code decides to inject #PF is now:

if ( L2 is running and L1 is not using HAP (i.e. sh-on-hap or sh-on-sh) ) {
Inject #PF (allowing the L1 to intercept);
If the L1 didn't intercept (or injection failed)
then crash the L1.
} else (i.e L1 running directly or hap-on-hap) {
Inject directly, ignoring the L1 intercepts.
}

I don't see why this change is needed. AFAICS, all cases are covered by unconditionally calling hvm_inject_exception() here. *-on-hap never takes this path at all because L0 doesn't intercept #PF when it uses HAP, so the only difference this makes is to catch the case where L1 didn't intercept #PF and it was injected directly into L2.

But this is an acceptable thing for the L1 to do (even though Xen doesn't ever behave that way), so I think it's wrong to crash the guest.

What was the reason for calling svm_inject_exception() directly in some cases?

Cheers,

Tim.

Quoted 4 lines

If you can invalidate this error check reason then yes, I can go back to make hvm_inject_exception() return void.

Christoph

Quoted 5 lines

I don't see why this change is needed. AFAICS, all cases are covered by unconditionally calling hvm_inject_exception() here. *-on-hap never takes this path at all because L0 doesn't intercept #PF when it uses HAP, so the only difference this makes is to catch the case where L1 didn't intercept #PF and it was injected directly into L2.

Correct. How should L1 fix its shadow page table in this case? I expect L2 to go wild because of that.

Quoted 2 lines

But this is an acceptable thing for the L1 to do (even though Xen doesn't ever behave that way), so I think it's wrong to crash the guest.

Ok, that is the point where our opinions differ. Can you explain me why this is acceptable? As I said above, I expect the L2 guest to go wild in this case.

Quoted 2 lines

What was the reason for calling svm_inject_exception() directly in some cases?

svm_inject_exception() always injects a #PF into running L1 or L2. It never injects a VMEXIT into L1.

Hi,

Quoted 6 lines · Christoph Egger

Quoted 5 lines

I don't see why this change is needed. AFAICS, all cases are covered by unconditionally calling hvm_inject_exception() here. *-on-hap never takes this path at all because L0 doesn't intercept #PF when it uses HAP, so the only difference this makes is to catch the case where L1 didn't intercept #PF and it was injected directly into L2.

Quoted 2 lines

Correct. How should L1 fix its shadow page table in this case? I expect L2 to go wild because of that.

L1 might not be using shadow pagetables. It might be some sort of cooperative microkernel allowing the L2 to do its own paging. Even in Xen, if we started using HVM containers for performance of 64-bit direct-paged guests, we might consider having guest pagefaults delivered straight to the guest (well, except that it would break ptwr_foo).

Quoted 7 lines

Quoted 3 lines

But this is an acceptable thing for the L1 to do (even though Xen doesn't ever behave that way), so I think it's wrong to crash the guest.

Ok, that is the point where our opinions differ. Can you explain me why this is acceptable? As I said above, I expect the L2 guest to go wild in this case.

Quite possibly, but it's (a) exactly what would happen on real hardware, and (b) not L0's problem if L1 misbehaves (except to protect other L1s from it).

Quoted 5 lines

Quoted 2 lines

What was the reason for calling svm_inject_exception() directly in some cases?

svm_inject_exception() always injects a #PF into running L1 or L2. It never injects a VMEXIT into L1.

Yes; but _why_ do we need to inject directly into L2?