Skip to contentSkip to conversation

Fix 32bit Windows guest VMs save/restore

PATCH NAK Wei Huang 2 participants 10 messages Jan 31, 2011 – Feb 1, 2011

lore
Share & tools
lore

Wei Huang

author
#0 lore raw

PATCH 1/1 Fix 32bit Windows guest VMs save/restore

The attached patch fixes the save/restore issue seen with 32bit Windows guest VMs. The root cause is that current Xen doesn't intercept SYSENTER-related MSRs for 32bit guest VMs. As a result, the guest_sysenter_xxx fields contain incorrect values and shouldn't be used for save/restore. This patch checks the LMA bit of EFER register in the save/restore code path.

Please apply it to both Xen-4.0 and Xen-unstable trees.

xen/arch/x86/hvm/svm/svm.c +32 -6
diff -r 5ce41defa1fa xen/arch/x86/hvm/svm/svm.c
@@ -157,6 +157,7 @@
 
 static int svm_vmcb_save(struct vcpu *v, struct hvm_hw_cpu *c)
 {
+    bool_t lma = !!(v->arch.hvm_vcpu.guest_efer & EFER_LMA);
     struct vmcb_struct *vmcb = v->arch.hvm_svm.vmcb;
 
     c->cr0 = v->arch.hvm_vcpu.guest_cr[0];
@@ -164,9 +165,20 @@
     c->cr3 = v->arch.hvm_vcpu.guest_cr[3];
     c->cr4 = v->arch.hvm_vcpu.guest_cr[4];
 
-    c->sysenter_cs = v->arch.hvm_svm.guest_sysenter_cs;
-    c->sysenter_esp = v->arch.hvm_svm.guest_sysenter_esp;
-    c->sysenter_eip = v->arch.hvm_svm.guest_sysenter_eip;
+    /* SYSENTER MSRs aren't intercepted under 32bit mode. So the data should 
+     * be copied from VMCB save area under 32bit mode. */
+    if ( lma ) 
+    {
+        c->sysenter_cs = v->arch.hvm_svm.guest_sysenter_cs;
+        c->sysenter_esp = v->arch.hvm_svm.guest_sysenter_esp;
+        c->sysenter_eip = v->arch.hvm_svm.guest_sysenter_eip;
+    } 
+    else
+    {
+        c->sysenter_cs = vmcb->sysenter_cs;
+        c->sysenter_esp = vmcb->sysenter_esp;
+        c->sysenter_eip = vmcb->sysenter_eip;
+    }
 
     c->pending_event = 0;
     c->error_code = 0;
@@ -185,8 +197,12 @@
 {
     unsigned long mfn = 0;
     p2m_type_t p2mt;
+    bool_t lma;
     struct vmcb_struct *vmcb = v->arch.hvm_svm.vmcb;
     struct p2m_domain *p2m = p2m_get_hostp2m(v->domain);
+    
+    /* Note: Please make sure guest_efer been restored at this point. */
+    lma = !!(v->arch.hvm_vcpu.guest_efer & EFER_LMA);
 
     if ( c->pending_valid &&
          ((c->pending_type == 1) || (c->pending_type > 6) ||
@@ -224,9 +240,19 @@
     hvm_update_guest_cr(v, 2);
     hvm_update_guest_cr(v, 4);
 
-    v->arch.hvm_svm.guest_sysenter_cs = c->sysenter_cs;
-    v->arch.hvm_svm.guest_sysenter_esp = c->sysenter_esp;
-    v->arch.hvm_svm.guest_sysenter_eip = c->sysenter_eip;
+    /* Copy data into VMCB save area under 32bit mode */
+    if ( lma )
+    {
+        v->arch.hvm_svm.guest_sysenter_cs = c->sysenter_cs;
+        v->arch.hvm_svm.guest_sysenter_esp = c->sysenter_esp;
+        v->arch.hvm_svm.guest_sysenter_eip = c->sysenter_eip;
+    }
+    else
+    {
+        vmcb->sysenter_cs = c->sysenter_cs;
+        vmcb->sysenter_esp = c->sysenter_esp;
+        vmcb->sysenter_eip = c->sysenter_eip;
+    }
 
     if ( paging_mode_hap(v->domain) )
     {

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Nacked-by on v1 1/1 NAK
#1 lore raw
Quoted 7 lines · Wei Huang

The attached patch fixes the save/restore issue seen with 32bit Windows guest VMs. The root cause is that current Xen doesn't intercept SYSENTER-related MSRs for 32bit guest VMs. As a result, the guest_sysenter_xxx fields contain incorrect values and shouldn't be used for save/restore. This patch checks the LMA bit of EFER register in the save/restore code path.

Quoted 5 lines

Please apply it to both Xen-4.0 and Xen-unstable trees.

This handling of the SYSENTER MSRs is overly complicated. I suggest reverting a bunch of the original handling of cross-vendor migration as follows:
* Never intercept the SYSENTER MSRs.
* Remove the vcpu->arch.hvm_svm.guest_sysenter_* fields.
* Always hvm save/restore from/to the values in the vmcb.
* Modify svm_msr_read_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb() and
then read the sysenter msr value from vmcb
* Modify svm_msr_write_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb(),
then modify the sysenter msr in the vmcb, and then svm_vmload().

Result is that we get rid of some redundant fields from the vcpu structure and have one canonical place we always keep the sysenter msr values, in the vmcb. The extra cost in the msr read/write functions is totally inconsequential, and only used after guest migration from an Intel CPU anyway. Hardly something to optimise for.

-- Keir

Quoted 6 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#2 lore raw
Quoted 1 line · Keir Fraser

This handling of the SYSENTER MSRs is overly complicated.

By the way, apart from the complexity, I suspect the current approach has further bugs because I don't believe the canonical sysenter msr values are correctly shuffled between the vmcb and vcpu structures when a guest moves in and out of long mode.

Just another example why having two different canonical places for one data item is a bad idea.

-- Keir

Quoted 27 lines

I suggest
reverting a bunch of the original handling of cross-vendor migration as follows:
* Never intercept the SYSENTER MSRs.
* Remove the vcpu->arch.hvm_svm.guest_sysenter_* fields.
* Always hvm save/restore from/to the values in the vmcb.
* Modify svm_msr_read_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb() and
then read the sysenter msr value from vmcb
* Modify svm_msr_write_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb(),
then modify the sysenter msr in the vmcb, and then svm_vmload().

Result is that we get rid of some redundant fields from the vcpu structure and have one canonical place we always keep the sysenter msr values, in the vmcb. The extra cost in the msr read/write functions is totally inconsequential, and only used after guest migration from an Intel CPU anyway. Hardly something to optimise for.

-- Keir

Quoted 6 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

author
#4 lore raw
In reply to Keir Fraser

Hi Keir,

My comments inline below.

-Wei

Quoted 14 lines · Keir Fraser
Quoted 7 lines · Wei Huang

The attached patch fixes the save/restore issue seen with 32bit Windows guest VMs. The root cause is that current Xen doesn't intercept SYSENTER-related MSRs for 32bit guest VMs. As a result, the guest_sysenter_xxx fields contain incorrect values and shouldn't be used for save/restore. This patch checks the LMA bit of EFER register in the save/restore code path.

Quoted 5 lines

Please apply it to both Xen-4.0 and Xen-unstable trees.

Quoted 6 lines

This handling of the SYSENTER MSRs is overly complicated. I suggest reverting a bunch of the original handling of cross-vendor migration as follows:
* Never intercept the SYSENTER MSRs.

The reason for Christoph to create this patch is AMD doesn't support SYSENTER in long mode. If we don't intercept MSRs under long mode, we will get stuck with #UD after migration from Intel platform. Did you actually mean "* Always intercept the SYSENTER MSRs" here?

Quoted 27 lines

* Remove the vcpu->arch.hvm_svm.guest_sysenter_* fields.
* Always hvm save/restore from/to the values in the vmcb.
* Modify svm_msr_read_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb() and
then read the sysenter msr value from vmcb
* Modify svm_msr_write_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb(),
then modify the sysenter msr in the vmcb, and then svm_vmload().

Result is that we get rid of some redundant fields from the vcpu structure and have one canonical place we always keep the sysenter msr values, in the vmcb. The extra cost in the msr read/write functions is totally inconsequential, and only used after guest migration from an Intel CPU anyway. Hardly something to optimise for.

-- Keir

Quoted 5 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

author
#3 lore raw
In reply to Keir Fraser

This is true. I thought about it. Some problematic code path can be "enter 32bit mode => execute sysenter => turn on long mode" (or vice versa). In this case, SYSENTER MSRs should be copied from VMCB to vcpu fields. But current Xen doesn't do so.

-Wei

Quoted 3 lines · Keir Fraser

suspect the current approach has
further bugs because I don't believe the canonical sysenter msr values are correctly shuffled between the vmcb and vcpu structures

#5 lore raw
In reply to Wei Huang
Quoted 6 lines · Wei Huang
Quoted 4 lines

This handling of the SYSENTER MSRs is overly complicated. I suggest reverting a bunch of the original handling of cross-vendor migration as follows:
* Never intercept the SYSENTER MSRs.

The reason for Christoph to create this patch is AMD doesn't support SYSENTER in long mode.

Yes.

Quoted 2 lines

If we don't intercept MSRs under long mode, we will get stuck with #UD after migration from Intel platform.

It's the SYSENTER instruction that causes the UD, right, not the WRMSR writes to the SYSENTER MSRs? Then my described approach will work -- the SYSENTER instruction will be handled by Xen's x86_emulate(), calling out to svm_msr_read_intercept() to grab the SYSENTER MSR values (from the VMCB, as I described). In fact x86_emulate() handles WRMSR too, so even if WRMSR caused UD we'd still handle it.

Quoted 2 lines

Did you
actually mean "* Always intercept the SYSENTER MSRs" here?

No, I think my approach works as I described it.

-- Keir

Quoted 29 lines
Quoted 27 lines

* Remove the vcpu->arch.hvm_svm.guest_sysenter_* fields.
* Always hvm save/restore from/to the values in the vmcb.
* Modify svm_msr_read_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb() and
then read the sysenter msr value from vmcb
* Modify svm_msr_write_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb(),
then modify the sysenter msr in the vmcb, and then svm_vmload().

Result is that we get rid of some redundant fields from the vcpu structure and have one canonical place we always keep the sysenter msr values, in the vmcb. The extra cost in the msr read/write functions is totally inconsequential, and only used after guest migration from an Intel CPU anyway. Hardly something to optimise for.

-- Keir

Quoted 5 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

author
#6 lore raw

But there is another issue: upper 32bit of sysenter MSRs in VMCB save area will be truncated with VMSARE/VMEXIT (see comments in vmcb.h). Could we use these VMCB fields as a storage for 64bit MSRs?

Thanks,
-Wei

Quoted 65 lines · Keir Fraser
Quoted 6 lines · Wei Huang
Quoted 4 lines

This handling of the SYSENTER MSRs is overly complicated. I suggest reverting a bunch of the original handling of cross-vendor migration as follows:
* Never intercept the SYSENTER MSRs.

The reason for Christoph to create this patch is AMD doesn't support SYSENTER in long mode.

Yes.

Quoted 2 lines

If we don't intercept MSRs under long mode, we will get stuck with #UD after migration from Intel platform.

It's the SYSENTER instruction that causes the UD, right, not the WRMSR writes to the SYSENTER MSRs? Then my described approach will work -- the SYSENTER instruction will be handled by Xen's x86_emulate(), calling out to svm_msr_read_intercept() to grab the SYSENTER MSR values (from the VMCB, as I described). In fact x86_emulate() handles WRMSR too, so even if WRMSR caused UD we'd still handle it.

Quoted 2 lines

Did you
actually mean "* Always intercept the SYSENTER MSRs" here?

No, I think my approach works as I described it.

-- Keir

Quoted 22 lines
Quoted 22 lines

* Remove the vcpu->arch.hvm_svm.guest_sysenter_* fields.
* Always hvm save/restore from/to the values in the vmcb.
* Modify svm_msr_read_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb() and
then read the sysenter msr value from vmcb
* Modify svm_msr_write_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb(),
then modify the sysenter msr in the vmcb, and then svm_vmload().

Result is that we get rid of some redundant fields from the vcpu structure and have one canonical place we always keep the sysenter msr values, in the vmcb. The extra cost in the msr read/write functions is totally inconsequential, and only used after guest migration from an Intel CPU anyway. Hardly something to optimise for.

-- Keir

Quoted 5 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Quoted 7 lines
Quoted 5 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#7 lore raw
Quoted 3 lines · Wei Huang

But there is another issue: upper 32bit of sysenter MSRs in VMCB save area will be truncated with VMSARE/VMEXIT (see comments in vmcb.h). Could we use these VMCB fields as a storage for 64bit MSRs?

Is that a bug? Seems unfortunate and unnecessary.

Well, in that case I would suggest to *always* intercept the MSRs, keep the vcpu-structure fields, and in svm_msr_write_intercept(MSR_IA32_SYSENTER_*) update vcpu structure *and* do svm_sync_vmcb(); update vmcb; svm_vmload().

This keeps the canonical version of the msrs always in the vcpu structure, with a (possibly 32-bit truncated) copy in the vmcb. The truncation is safe since the MSRs will only directly be used in guest mode by legacy-mode execution of SYSENTER.

How's that?

-- Keir

Quoted 73 lines

Thanks,
-Wei

Quoted 66 lines · Keir Fraser
Quoted 6 lines · Wei Huang
Quoted 4 lines

This handling of the SYSENTER MSRs is overly complicated. I suggest reverting a bunch of the original handling of cross-vendor migration as follows:
* Never intercept the SYSENTER MSRs.

The reason for Christoph to create this patch is AMD doesn't support SYSENTER in long mode.

Yes.

Quoted 2 lines

If we don't intercept MSRs under long mode, we will get stuck with #UD after migration from Intel platform.

It's the SYSENTER instruction that causes the UD, right, not the WRMSR writes to the SYSENTER MSRs? Then my described approach will work -- the SYSENTER instruction will be handled by Xen's x86_emulate(), calling out to svm_msr_read_intercept() to grab the SYSENTER MSR values (from the VMCB, as I described). In fact x86_emulate() handles WRMSR too, so even if WRMSR caused UD we'd still handle it.

Quoted 2 lines

Did you
actually mean "* Always intercept the SYSENTER MSRs" here?

No, I think my approach works as I described it.

-- Keir

Quoted 23 lines
Quoted 23 lines

* Remove the vcpu->arch.hvm_svm.guest_sysenter_* fields.
* Always hvm save/restore from/to the values in the vmcb.
* Modify svm_msr_read_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb()
and
then read the sysenter msr value from vmcb
* Modify svm_msr_write_intercept(MSR_IA32_SYSENTER_*) to svm_sync_vmcb(),
then modify the sysenter msr in the vmcb, and then svm_vmload().

Result is that we get rid of some redundant fields from the vcpu structure and have one canonical place we always keep the sysenter msr values, in the vmcb. The extra cost in the msr read/write functions is totally inconsequential, and only used after guest migration from an Intel CPU anyway. Hardly something to optimise for.

-- Keir

Quoted 5 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Quoted 7 lines
Quoted 5 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

author
#8 lore raw

How does this one look? It should address your concern about duplicating data in both vcpu and vmcb. I have tested it with both 32bit and 64bit Windows successfully.

-Wei

=================
Fix 32bit guest VM save/restore issues associated with SYSENTER MSRs on AMD platforms.

This patch turn-on SYSENTER MSRs interception for 32bit guest VMs on AMD CPUs. With it, hvm_svm.guest_sysenter_xx fields always contain the canonical version of SYSENTER MSRs and are used in guest save/restore. The data fields in VMCB save area are updated as necessary.

=================

xen/arch/x86/hvm/svm/svm.c +27 -15
diff -r d972e89797f1 xen/arch/x86/hvm/svm/svm.c
@@ -224,10 +224,11 @@
     hvm_update_guest_cr(v, 2);
     hvm_update_guest_cr(v, 4);
 
-    v->arch.hvm_svm.guest_sysenter_cs = c->sysenter_cs;
-    v->arch.hvm_svm.guest_sysenter_esp = c->sysenter_esp;
-    v->arch.hvm_svm.guest_sysenter_eip = c->sysenter_eip;
-
+    /* load sysenter MSRs into VMCB save area and VCPU fields */
+    vmcb->sysenter_cs = v->arch.hvm_svm.guest_sysenter_cs = c->sysenter_cs;
+    vmcb->sysenter_esp = v->arch.hvm_svm.guest_sysenter_esp = c->sysenter_esp;
+    vmcb->sysenter_eip = v->arch.hvm_svm.guest_sysenter_eip = c->sysenter_eip;
+    
     if ( paging_mode_hap(v->domain) )
     {
         vmcb_set_np_enable(vmcb, 1);
@@ -433,14 +434,6 @@
     if ( lma )
         new_efer |= EFER_LME;
     vmcb_set_efer(vmcb, new_efer);
-
-    /*
-     * In legacy mode (EFER.LMA=0) we natively support SYSENTER/SYSEXIT with
-     * no need for MSR intercepts. When EFER.LMA=1 we must trap and emulate.
-     */
-    svm_intercept_msr(v, MSR_IA32_SYSENTER_CS, lma);
-    svm_intercept_msr(v, MSR_IA32_SYSENTER_ESP, lma);
-    svm_intercept_msr(v, MSR_IA32_SYSENTER_EIP, lma);
 }
 
 static void svm_sync_vmcb(struct vcpu *v)
@@ -1142,6 +1135,21 @@
 {
     struct vcpu *v = current;
     struct vmcb_struct *vmcb = v->arch.hvm_svm.vmcb;
+    int sync = 0;
+
+    switch ( msr )
+    {
+    case MSR_IA32_SYSENTER_CS:
+    case MSR_IA32_SYSENTER_ESP:
+    case MSR_IA32_SYSENTER_EIP:
+        sync = 1;
+        break;
+    default:
+        break;
+    }
+
+    if ( sync )
+        svm_sync_vmcb(v);    
 
     switch ( msr )
     {
@@ -1149,13 +1157,13 @@
         goto gpf;
 
     case MSR_IA32_SYSENTER_CS:
-        v->arch.hvm_svm.guest_sysenter_cs = msr_content;
+        vmcb->sysenter_cs = v->arch.hvm_svm.guest_sysenter_cs = msr_content;
         break;
     case MSR_IA32_SYSENTER_ESP:
-        v->arch.hvm_svm.guest_sysenter_esp = msr_content;
+        vmcb->sysenter_esp = v->arch.hvm_svm.guest_sysenter_esp = msr_content;
         break;
     case MSR_IA32_SYSENTER_EIP:
-        v->arch.hvm_svm.guest_sysenter_eip = msr_content;
+        vmcb->sysenter_eip = v->arch.hvm_svm.guest_sysenter_eip = msr_content;
         break;
 
     case MSR_IA32_DEBUGCTLMSR:
@@ -1213,6 +1221,10 @@
         wrmsr_hypervisor_regs(msr, msr_content);
         break;
     }
+
+    if ( sync )
+        svm_vmload(vmcb);
+
     return X86EMUL_OKAY;
 
  gpf:

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#9 lore raw

Yeah, good. I'll double check it next week and apply it then for 4.1.x and 4.0.x.

Thanks,
Keir

Quoted 19 lines · Wei Huang

How does this one look? It should address your concern about duplicating data in both vcpu and vmcb. I have tested it with both 32bit and 64bit Windows successfully.

-Wei

=================
Fix 32bit guest VM save/restore issues associated with SYSENTER MSRs on AMD platforms.

This patch turn-on SYSENTER MSRs interception for 32bit guest VMs on AMD CPUs. With it, hvm_svm.guest_sysenter_xx fields always contain the canonical version of SYSENTER MSRs and are used in guest save/restore. The data fields in VMCB save area are updated as necessary.

=================